o ۾i @s~UddlZddlmZddlmZmZmZmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZddlmZddlmZmZe rWdd lmZdd lmZddlm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,e$e-Z.e dZ/da0e/dBe1d<dZ2de3de4fddZ5ej6ddZ7de/fddZ8eGdddZ9dede:e3de;d de dad&e?d'e+d(e?de@e+eejAgejAfffd)d*ZBd'e+de+fd+d,ZC -dbd'e+d.e?de@e?e?e?e?e?e?ffd/d0ZD - 1dcd'e+d.e?d2e4de@e?e?e?e?ffd3d4ZE dad5ejAd6e?d7e?de:e@e?e?ffd8d9ZF -dbd:d;dd?ZGd@ejAdAe?dejAfdBdCZHdDejAdejAfdEdFZIdGe3dHe dIe:e@e3e e fde fdJdKZJeGdLdMdMe ZKdNe3dOe;e(de;e(fdPdQZLdRejAdSejMfdTdUZN -  -dddVejAdWe?dXe?dBdYe?dejAf dZd[ZOd\ejAdVejAd]ed^e3dejAf d_d`ZPdS)eN)Callable) dataclassfieldfieldsmake_dataclass) TYPE_CHECKINGAnyLiteralProtocolget_args)runtime_checkable) VllmConfigget_layers_from_vllm_config)cdiv) KVCacheSpec MambaSpec)SchedulerOutput) InputBatch)get_kv_connector_cache_layout) init_logger)AttentionLayerBase)AttentionBackend AttentionImplAttentionMetadataCommonAttentionMetadatasubclass_attention_backend)NHDHND_KV_CACHE_LAYOUT_OVERRIDEvaluereturncCs |ttvSN)r KVCacheLayoutType)r r$T/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/attention/backends/utils.pyis_valid_kv_cache_layout/s r&cCsTd}tdurt}td||Stj}|durt}|St|s"Jtd||S)NzM`_KV_CACHE_LAYOUT_OVERRIDE` variable detected. Setting KV cache layout to %s.zT`VLLM_KV_CACHE_LAYOUT` environment variable detected. Setting KV cache layout to %s.)rlogger info_onceenvsVLLM_KV_CACHE_LAYOUTrr& cache_layoutr$r$r%get_kv_cache_layout3s$ r-r,cCs|adSr")rr+r$r$r%set_kv_cache_layoutQsr.c@sjeZdZUdZeed<edBed<eed<dZeed<e dddZ edBed <e dddZ edBed <dS) PerLayerParametersa Currently, FlashInfer backend only support models in which all layers share the same values for the following hyperparameters. Should not be used for trtllm-gen backend since it supports different values for the following hyperparameters. window_leftNlogits_soft_capsm_scaleF has_sinks)defaultcomparehas_same_window_leftshas_same_all_params) __name__ __module__ __qualname____doc__int__annotations__floatr3boolrr6r7r$r$r$r%r/Vs   r/ vllm_config layer_namescls_rc Cst|t|}i}|D]8\}}|j}t||sJt|dd}|dur(|dnd} t|dd} |j} t|dddu} t| | | | ||<q |S)zc Scan layers in `layer_names` and determine some hyperparameters to use during `plan`. sliding_windowNrrr1sinks)rritemsimpl isinstancegetattrscaler/) r@rArBlayersper_layer_paramskeylayerrF window_sizer0r1r2r3r$r$r%get_per_layer_parametershs$   rOrKcs\t|dks Jdt|}|dtfdd|D_tfdd|D_S)ad Currently, FlashInfer backend other than trtllm-gen only support models in which all layers share the same values for the following hyperparameters: - `window_left` - `logits_soft_cap` - `sm_scale` So this function asserts that all layers share the same values for these hyperparameters and returns the global values. rz'No attention layers found in the model.c3s|] }|jjkVqdSr")r0.0params global_paramsr$r% s  z/infer_global_hyperparameters..c3s|]}|kVqdSr"r$rPrSr$r%rUs )lenlistvaluesallr6r7)rK param_setsr$rSr%infer_global_hyperparameterss    r[attn_chunk_sizecommon_attn_metadata block_sizecs|j}|j}|j}|jj}|dd|dd}|jd}t||||| tj } ||| } dt || |} t | } | dt | | | } tjtj d| }t | | |d}t || | }| ||dk<t|||d||dk||dk<tjdtj d}tj ||dddd|d<tj| d|tj d}| || d<||}t || ||t | | }||}||dksJd|d|||}|dddftj|tj d}|dj|jddd}t tj|tj d| |}t|t|fd d }||}t|}t|}t|}t||j|d d |j|d d t||j||||jd |t|d |fS)Nrrdtype)outzattn_chunk_size z is not divisible by block_size )maxcs|fdS)Nr)view) block_tablebatch_indices_torchblock_indices_torchvirtual_batchesr$r%Qs z6make_local_attention_virtual_batches..T)device non_blocking) query_start_loc_cpuquery_start_locseq_lensnum_reqsnum_actual_tokens max_query_len max_seq_lenblock_table_tensor slot_mappingcausal _seq_lens_cpu_num_computed_tokens_cpu)rmnumpy seq_lens_cpurtrnrkshapenpminimumastypeint32rcumsumrepeatarangeemptyfullreshapecliptorch from_numpyr<rcrtorVrqru)r\r]r^query_start_loc_np seq_lens_nprerk q_seqlensactual_batch_sizeq_tokens_in_first_blocktokens_in_last_block local_blocks cu_num_blocks block_offsetsrrarangeseqlens_q_localcu_seqlens_q_localseqlens_k_localnum_computed_tokens_localk_seqstarts_absolute block_startspages_per_local_batch block_indices batch_indicesmake_block_tableblock_table_localrmrzrsr$rfr%$make_local_attention_virtual_batchess                rc Cs|jdkr|S|jdusJ|jdusJ|j}|j}|d|}|j}|j}tj||dddd}tj||d}tj|d|j |j d}d|d<tj |dd|dd<t | } t |} t||jddd |j|| | |j|j|jd|j|jd }|S) Nr_T)right) minlength)rkrardimcpu)rl) rnrmrorprqrrrsrtrurvrwrx)rrlogits_indices_paddednum_logits_indicesrprnr bucketizebincountrrkrarr<rcitemsumrrrorsrtrurwrx) r]rrlogits_indicesrprn request_idsnum_decode_tokensdecode_query_start_locdecode_max_query_lentotal_num_decode_tokensr$r$r%1make_kv_sharing_fast_prefill_common_attn_metadatajsB    rr_decode_thresholdcCs|j}|j}|j}|j}|j}||kr|dd|ddfS|dd|dd}||k}||k|@} |jdd} | jdd} | } || } t |sZ| dd| ddfS|| }|| }t | so| |d| |dfS| | }|| }||| }||}| ||| ||fS)a Assuming a reordered batch, finds the boundary between prefill and decode requests. Args: common_attn_metadata: CommonAttentionMetadata object containing the batch metadata. decode_threshold: The maximum query length to be considered a decode. Returns: num_decodes: The number of decode requests. num_extends: The number of extend requests. num_prefills: The number of prefill requests. num_decode_tokens: The number of tokens in the decode requests. num_extend_tokens: The number of tokens in the extend requests. num_prefill_tokens: The number of tokens in the prefill requests. rr_Nrr) rrrprqrmrzr<argmaxrrany)r]rrrrp num_tokensrnro query_lensis_prefill_or_extend is_prefill first_extend first_prefill num_decodesrnum_prefills_or_extendsnum_prefill_or_extend_tokens num_extends num_prefillsnum_prefill_tokensnum_extend_tokensr$r$r%"split_decodes_prefills_and_extendssJ      rFrequire_uniformcCs4|j}|j}|j}|j}||kr|r|dkr|d|dfS|dd|dd}|d|kr6d|d|fS|r^t||dk|dkBrW||d|ksQJd|d|dfS||dk}n||k}t|sm|d|dfS|j dd} t|d| |ksJ| } || } || } || } | | | | fS)a Assuming a reordered batch, finds the boundary between prefill and decode requests. Args: common_attn_metadata: CommonAttentionMetadata object containing the batch metadata. decode_threshold: The maximum query length to be considered a decode. require_uniform: If True, requires that all decode requests have the same query length. When set, some queries may be considered prefills even if they are <= decode_threshold, in order to ensure uniformity. Returns: num_decodes: The number of decode requests. num_prefills: The number of prefill requests. num_decode_tokens: The number of tokens in the decode requests. num_prefill_tokens: The number of tokens in the prefill requests. r_rNrztokens not padded correctlyr) rrrprqrmrrrYrr<r)r]rrrrrprrnrrrrrrrr$r$r%split_decodes_and_prefillss6       rrzworkspace_sizerequest_offsetc Csg}dt|}}t||ksJ||krT|d}}||krE|||}|krE||7}|d7}||krE|||}|ks-|||||f||ks|S)a Split the prefill requests into chunks such that the total sequence length of each chunk is less than or equal to the workspace size. Args: seq_lens_cpu: The sequence lengths of the prefill requests on CPU. workspace_size: The maximum workspace size (in tokens) per chunk. request_offset: The offset to add to the request indices. Returns: A list of tuples of (reqs_start, reqs_end) representing chunk boundaries. rr_)rVrrYrappend) rzrr chunk_boundsinstart chunk_totalsr$r$r%split_prefill_chunks%s   r input_batchrscheduler_outputrcs^t|j}fdd|jD}t|}|jd|}|dk}||k|@}||k|@} tj|jtjd} d| | <d| |<t| } t| } tj|tjd} d| | | | <d| | | d<| | k}| sldSt |d}tj | |d d }||}d d t ||D}|D] }||}||kr||||||}|||<|}||ksqd S)z Reorders the batch to split into prefill and decode requests; places all requests with <= decode_threshold tokens at the front of the batch. Returns: True if the batch was modified, False otherwise. csg|]}j|qSr$)num_scheduled_tokens)rQidrr$r% Us z?reorder_batch_to_split_decodes_and_prefills..Nrr`r_Fstable)kindcSsi|] \}}t|t|qSr$)r<)rQsrcdstr$r$r% usz?reorder_batch_to_split_decodes_and_prefills..T)rVreq_idsr|arraynum_computed_tokens_cpuzerosr{rr<rrwhereargsortzip swap_statesget)rrrrprnum_scheduled_tokens_npnum_computed_tokens_npr is_decode is_extend req_regionsrrtarget_regions needs_swap orig_indices sorted_order src_indices src_dest_maprrnext_dstr$rr%+reorder_batch_to_split_decodes_and_prefills@sD       rquery batch_sizecCsv|dksJd|d|jd}|jd}|jd}||dks/Jd|d|||}|||||S) z Reshapes the query tensor for the specified batch size, so that it has shape (batch_size, seq_len, num_heads, head_dim). zquery must be 3D, got Drr_rz total_tokens=z is not divisible by batch_size=rr{rd)rr total_tokens num_headshead_dimseq_lenr$r$r%reshape_query_for_spec_decodes    r attn_outputcCs^|dkr|S|dksJd|d|jd|jd}|||jd|jdS)zo Reshapes the attention output tensor, so that the batch_size and seq_len dimensions are combined. rzattn_output must be 4D, got rrr_rr)rrr$r$r%#reshape_attn_output_for_spec_decodes  r name_prefix metadata_clsrcCs||j}t|||fd}|S)zH Return a new subclass of `metadata_cls` with additional fields )bases)r8r)rrrnameWrappedr$r$r%subclass_attention_metadatas rc@s0eZdZUdZejdBed<dZedBed<dS)KVSharingFastPrefillMetadataNrr) r8r9r:rrTensorr=rr<r$r$r$r%rs rprefixunderlying_attn_backendcCs*|}Gddd|}t|||d}|S)Nc s2eZdZ ddedededeffdd ZZS) zGcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilderFcommon_prefix_lenr] fast_buildr!cs6t|}t|||}Gddd|jt}|||S)Nc@seZdZddZdS)z|create_fast_prefill_custom_backend..FastPrefillAttentionBuilder.build..KVSharingFastPrefillAttentionMetadatacSs:t|jD] }t||jt||jq|j|_|j|_dSr")r __class__setattrrrHrr)selfmetadatar]_fieldr$r$r%__init__s  zcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilder.build..KVSharingFastPrefillAttentionMetadata.__init__N)r8r9r:rr$r$r$r%%KVSharingFastPrefillAttentionMetadatas r)rsuperbuildr r)r rr]r new_common_attn_metadatar rr r$r%rs  zMcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilder.build)F) r8r9r:r<rr?rr __classcell__r$r$rr%FastPrefillAttentionBuildersr)rattention_backend_cls builder_cls)get_builder_clsr)rrunderlying_builderr attn_backendr$r$r%"create_fast_prefill_custom_backendsrquery_start_loc_p_cpurkc Cs|jjdksJ|}i}d}d}dD]}| | }i||<|||d<|||d<ttt t ||}|||d<t ||d} | ||d<t d| d} g} t |D] \} } | t| q_tj| tjd } | ||d <|durtj| fttj|d }tj| fttj|d }n|| kr|| t|| t|d | ||d | | |||d <|||d<q|||fS)Nr)numstotmlist mlist_lenirr` offsetlistrarkr batch_ptrtoken_chunk_offset_ptr)rktypediffrrrrr|rrrVrc enumerateextendrangetensorrr PAD_SLOT_IDnelementresize_fill_copy_)rrkseqlens nums_dictr%r&BLOCK_Mrr!r"MAX_NUM_PROGRAMSr#idxnumr$r$r%compute_causal_conv1d_metadatasR          r8rodcp_sizedcp_rankcp_kv_cache_interleave_sizec Cs|d}|durtj|tj|jdd|d}n tj|ggtj|jd}|tjdd|j d}||||}|||}t |||d|}||} | dS)zWhile using dcp, kv_cache size stored on each rank may be different, use this function to calculate split decode seq_lens of each dcp rank. Only consider dcp now, we can extend the case of cp based on this. rNr$r_r) sizerrrrk unsqueezerr,rr{rsqueeze) ror9r:r; num_requests rank_offsetsseq_lens_tiledbase remainderdcp_local_seq_lensr$r$r%get_dcp_local_seq_lenss6    rEre kv_cache_specmamba_cache_modecCsd|dvr|St|ts Jtj|d|jdd}tjd|j|jd}|d|}t |d|S)an Get the block table tensor for mamba kernels from the input common_attn_metadata.block_table_tensor given different mamba cache modes. - "all": input (#requests, cdiv(max_model_len, block_size)); output (#requests, cdiv(max_model_len, block_size)). - "none": input (#requests, 1 + num_speculative_blocks); output (#requests, 1 + num_speculative_blocks). - "align": input (#requests, cdiv(max_model_len, block_size)); output (#requests, 1 + num_speculative_blocks), which are the last 1 + num_speculative_blocks of each request. )rYnoner_r)min)rk) rGrrclampr^rnum_speculative_blocksrkr=gather)rerorFrG start_indicesoffsetsindices_to_gatherr$r$r%mamba_get_block_table_tensor<s  rP)r)r_)r_F)r_Nr_)Q functoolscollections.abcr dataclassesrrrrtypingrrr r r ryr|rtyping_extensionsr vllm.configr rvllm.utils.math_utilsrvllm.v1.kv_cache_interfacerrvllm.v1.core.sched.outputrvllm.v1.worker.gpu_input_batchr vllm.envsr)/vllm.distributed.kv_transfer.kv_connector.utilsr vllm.loggerr/vllm.model_executor.layers.attention_layer_baservllm.v1.attention.backendrrrrrr8r'r#rr=r-strr?r& lru_cacher-r.r/rWr'dictrOr[r<tuplerrrrrrrrrrrrrkr8rErPr$r$r$r%s&             !  U  = E >  C    - 6 (