o iNs @s,ddlmZmZddlmZmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlZddlmZe rbddlmZddlmZdd lmZdd lmZdd lmZdd lm Z dd l!m"Z"Gddde#eZ$GdddZ%GdddeZ&GdddZ'ede'dZ(eGdddZ)edZ*GdddeZ+Gdddee e*Z,Gdd d e Z-Gd!d"d"ee e(Z.Gd#d$d$e.e(e e(Z/Gd%d&d&e.e(e e(Z0Gd'd(d(e.e(e e(Z1d)e#d*e2fd+d,Z3d-e#d.e4e&d/e4e,e*d*e4e&fd0d1Z5d-e#d.e4e&d2e6e#e fd*e4e&fd3d4Z7dS)5)ABCabstractmethod) dataclassreplace)Enum) TYPE_CHECKINGAnyClassVarGenericProtocolTypeVarget_argsN) deprecated) VllmConfig) CacheDType)ColumnParallelLinear)QuantKey)DeviceCapability)KVCacheLayoutType) AttentionSpecc@&eZdZdZdZ dZ dZ dZdS) AttentionTypezO Attention type. Use string to be compatible with `torch.compile`. decoderencoder encoder_onlyencoder_decoderN)__name__ __module__ __qualname____doc__DECODERENCODER ENCODER_ONLYENCODER_DECODERr$r$O/home/ubuntu/vllm_env/lib/python3.10/site-packages/vllm/v1/attention/backend.pyrsrc@s$eZdZUeed<defddZdS) MultipleOfbasecCs ||_dSN)r')selfr'r$r$r%__init__* zMultipleOf.__init__N)rrrint__annotations__r*r$r$r$r%r&'s r&c@seZdZUdZdZeed<ejej gZ e e ej ed<ddgZe e ded<d Zeed <ed e eeBfd d Zeed efddZeed edfddZeeddZee dMdededededed eedff ddZe dNded eedffddZed eeeffd d!Zed e efd"d#Zeded efd$d%Z ed&ej d efd'd(Z!ed)d*d efd+d,Z"eded-Bd efd.d/Z#ed efd0d1Z$ed efd2d3Z%ed efd4d5Z&ed efd6d7Z'ed efd8d9Z(ed:ed efd;d<Z)ed=d>d efd?d@Z*eded&ej d)d*dedAedBedCedDd>d ed-BfdEdFZ+eded&ej d)d*dedAedBedCedGedDd>d:ed e efdHdIZ,edOdKdLZ-d-S)PAttentionBackendz&Abstract class for attention backends.Faccept_output_buffersupported_dtypesautobfloat16rsupported_kv_cache_dtypesT forward_includes_kv_cache_updatereturncCs tdgS)N)r&r$r$r$r% get_supported_kernel_block_sizes; z1AttentionBackend.get_supported_kernel_block_sizescCtr(NotImplementedErrorr$r$r$r%get_name?zAttentionBackend.get_nameAttentionImplBasecCr9r(r:r$r$r$r% get_impl_clsDr=zAttentionBackend.get_impl_clscCr9r(r:r$r$r$r%get_builder_clsIr=z AttentionBackend.get_builder_cls num_blocks block_size num_kv_heads head_sizecache_dtype_str.cCr9r(r:)rArBrCrDrEr$r$r%get_kv_cache_shapeN z#AttentionBackend.get_kv_cache_shapeinclude_num_layers_dimensioncCr9)av Get the physical (memory layout) ordering of the kv cache dimensions. e.g. if the KV cache shape is [2, num_blocks, block_size, num_heads, head_size], and get_kv_cache_stride_order returns (1, 3, 0, 2, 4) then the physical ordering of dimensions is [num_blocks, num_heads, 2, block_size, head_size]. If this function is unimplemented / raises NotImplementedError, the physical layout of the KV cache will match the logical shape. Args: include_num_layers_dimension: if True, includes an additional num_layers dimension, which is assumed to be prepended to the logical KV cache shape. With the above example, a return value (2, 4, 0, 1, 3, 5) corresponds to [num_blocks, num_heads, num_layers, 2, block_size, head_size]. If an additional dimension is NOT included in the returned tuple, the physical layout will not include a layers dimension. Returns: A tuple of ints which is a permutation of range(len(shape)). r:)rHr$r$r%get_kv_cache_stride_orderYsz*AttentionBackend.get_kv_cache_stride_ordercCs |j|jfSr()rrclsr$r$r% full_cls_namexs zAttentionBackend.full_cls_namecCsgSr(r$rJr$r$r%get_supported_head_sizes|z)AttentionBackend.get_supported_head_sizescCs|}| p ||vSr()rM)rKrDsupported_head_sizesr$r$r%supports_head_sizesz#AttentionBackend.supports_head_sizedtypecCs ||jvSr()r0)rKrQr$r$r%supports_dtyper8zAttentionBackend.supports_dtypekv_cache_dtypezCacheDType | NonecCs|durdS|j p||jvSNT)r3)rKrSr$r$r%supports_kv_cache_dtypes z(AttentionBackend.supports_kv_cache_dtypeNcCslddlm}|dur dSt|}||vrdS|}|sdS|D]}t|tr*|j}||dkr3dSq dS)Nr) BlockSizeTF)vllm.config.cacherVr r7 isinstancer&r')rKrBrV valid_sizessupported_kernel_block_sizessupported_sizer$r$r%supports_block_sizes   z$AttentionBackend.supports_block_sizecCdSNFr$rJr$r$r%is_mlarNzAttentionBackend.is_mlacCr]r^r$rJr$r$r% supports_sinkrNzAttentionBackend.supports_sinkcCr]r^r$rJr$r$r%supports_alibi_sqrtrNz$AttentionBackend.supports_alibi_sqrtcCr]r^r$rJr$r$r%supports_mm_prefixrNz#AttentionBackend.supports_mm_prefixcCr]r^r$rJr$r$r% is_sparserNzAttentionBackend.is_sparse attn_typecCs |tjkS)zCheck if backend supports a given attention type. By default, only supports decoder attention. Backends should override this to support other attention types. )rr )rKrdr$r$r%supports_attn_types z#AttentionBackend.supports_attn_type capabilityrcCr]rTr$)rKrfr$r$r%supports_compute_capabilityrNz,AttentionBackend.supports_compute_capabilityuse_mlahas_sink use_sparsedevice_capabilityc CdSr(r$) rKrDrQrSrBrhrirjrkr$r$r%supports_combination z%AttentionBackend.supports_combination use_mm_prefixc Cs*g} ||s | d||s| d||s | d||s*| d|r5|s5| d||krH|rC| dn| d|rS|sS| d||krf|ra| d n| d | | sp| d | | s~| d | d | |||||||| } | dur| | | S)Nzhead_size not supportedzdtype not supportedzkv_cache_dtype not supportedzblock_size not supportedz5partial multimodal token full attention not supportedzMLA not supportedznon-MLA not supportedzsink setting not supportedzsparse not supportedznon-sparse not supportedz compute capability not supportedzattention type z not supported) rPappendrRrUr\rbr_r`rcrgrerm) rKrDrQrSrBrhrirjrorkrdinvalid_reasonscombination_reasonr$r$r%validate_configurationsP                     z'AttentionBackend.validate_configurationKVCacheLayoutType | NonecCrlr(r$rJr$r$r%get_required_kv_cache_layoutrNz-AttentionBackend.get_required_kv_cache_layout)r1F)r5rt).rrrrr/boolr-torchfloat16r2r0r listrQr3r4 staticmethodr,r&r7rstrr<typer?r@tuplerFrI classmethodrLrMrPrRrUr\r_r`rarbrcrergrmrsrur$r$r$r%r..s               8r.c@s eZdZdS)AttentionMetadataN)rrrr$r$r$r%rsrT)boundc@seZdZUdZejed<ejed< ejed< eed< eed< eed< eed< ejed <ejed <d Ze ed <d Z ejd Bed<d Z ed Bed<d Z ejd Bed<d Z ejd Bed<d Zejd Bed<d Zejd Bed< d Zejd Bed<d Zejd Bed<d Zejd Bed<defddZdejfddZd)ddZeeddejfdd Zeed!dejfd"d#Zdejfd$d%Zded&eddfd'd(Zd S)*CommonAttentionMetadataz Per-batch attention metadata, shared across layers and backends. AttentionMetadataBuilder instances use it to construct per-layer metadata. For many of the tensors we keep both GPU and CPU versions. query_start_locquery_start_loc_cpuseq_lensnum_reqsnum_actual_tokens max_query_len max_seq_lenblock_table_tensor slot_mappingTcausalNlogits_indices_paddednum_logits_indicesencoder_seq_lensencoder_seq_lens_cpudcp_local_seq_lensdcp_local_seq_lens_cpu _seq_lens_cpu_num_computed_tokens_cpu_num_computed_tokens_cacher5cCs |jjdS)Nr)rshaper)r$r$r% batch_sizeOs z"CommonAttentionMetadata.batch_sizecCs|jdd|jddS)zENaive because it assumes that query ends where the next query starts.r6N)rrr$r$r%naive_query_lensRsz(CommonAttentionMetadata.naive_query_lenscKst|fi|Sr()r)r)kwargsr$r$r%rVszCommonAttentionMetadata.replacez Prefer using device seq_lens directly to avoid implicit H<>D sync. If a CPU copy is needed, use `seq_lens.cpu()` instead. Will be removed in a future release, please migrate as soon as possible. cCs|jdur |jd|_|jS)Ncpu)rrtorr$r$r% seq_lens_cpuYs z$CommonAttentionMetadata.seq_lens_cpua Prefer using device seq_lens directly to avoid implicit H<>D sync which breaks full async scheduling. If a CPU copy is needed, it can be derived from query_start_loc_cpu and seq_lens. Will be removed in a future release, please migrate as soon as possible. cCs8|jdur|jdd|jdd}|j||_|jS)Nr6r)rrr)r)query_seq_lensr$r$r%num_computed_tokens_cpufs  z/CommonAttentionMetadata.num_computed_tokens_cpucCs8|jdur|jdd|jdd}|j||_|jS)z>Compute num_computed_tokens on device (seq_lens - query_lens).Nr6r)rrr)r) query_lensr$r$r%compute_num_computed_tokensws  z3CommonAttentionMetadata.compute_num_computed_tokensnum_actual_reqscsfdd}tdid|jddd|jddd|jdd|jdur2|jdndd|jdurA|jdnFdd d |d |jd |jd |jdd|j d|d|j d|j d|j d||j d||jd||jd||jSd d |d |jd |jd |jdd|j d|d|j d|j d|j d||j d||jd||jd||jS)Ncs|dur |dSdSr(r$)xrr$r%sz2CommonAttentionMetadata.unpadded..rr6rrrrrrrrrrrrrrrrrr$)rrrrrrrrrrrrrrrrr)r)rrmaybe_slice_reqsr$rr%unpaddeds                   z CommonAttentionMetadata.unpadded)r5r)rrrrrxTensorr-r,rrwrrrrnpndarrayrrrrrrrrpropertyrrrrrr$r$r$r%rsb        rMc@r)AttentionCGSupportzConstants for the cudagraph support of the attention backend Here we do not consider the cascade attention, as currently it is never cudagraph supported.r6rN)rrrrALWAYS UNIFORM_BATCHUNIFORM_SINGLE_TOKEN_DECODENEVERr$r$r$r%rsrc@sDeZdZUejZeeed<dZe dBed<dZ e ed<e ddde ed d d ejfd d Zededd d dddefddZ   d0de dBde de ddfddZe d1de dede defddZdedejdejdefdd Zdedefd!d"Zded#e defd$d%Zde d&ejd'e d(e d)e d*e d+e d,e d-e de fd.d/ZdS)2AttentionMetadataBuilder_cudagraph_supportNreorder_batch_thresholdFsupports_update_block_table kv_cache_specr layer_names vllm_configrdevicecCs||_||_||_||_dSr()rrrr)r)rrrrr$r$r%r*s z!AttentionMetadataBuilder.__init__rKr5cCs|jS)z6Get the cudagraph support level of this builder class.)r)rKrrr$r$r%get_cudagraph_supportsz.AttentionMetadataBuilder.get_cudagraph_supportr6supports_spec_as_decodesupports_dcp_with_varlencCsx||_|jdur*|r*|jj}|dur*|jdur*d|jrdnd|j}t|j||_|jjjdkr8|s:d|_dSdSdS)Nr6r)rrspeculative_confignum_speculative_tokensparallel_draftingmaxparallel_configdecode_context_parallel_size)r)rrrrmax_num_queries_for_specr$r$r%_init_reorder_batch_thresholds(   z6AttentionMetadataBuilder._init_reorder_batch_thresholdcommon_prefix_lencommon_attn_metadata fast_buildcCr9)a Central method that builds attention metadata. Some builders (MLA) require reorder_batch to be called prior to build. Args: common_prefix_len: The length of the common prefix of the batch. common_attn_metadata: The common attention metadata. fast_build: The meta-data will prioritize speed of building over then speed at execution. Can be used for spec-decode where the result of a build call may only be used for few layers/iters. r:)r)rrrr$r$r%buildszAttentionMetadataBuilder.buildmetadata blk_tablercCr9)a Update the block table for the attention metadata. Faster when theres multiple kv-cache groups that create virtually the same metadata but just with different block tables. Only needs to be implemented if supports_update_block_table is True. r:)r)rrrr$r$r%update_block_table z+AttentionMetadataBuilder.update_block_tablecCs|jd|dS)z Build attention metadata for CUDA graph capture. Uses build by default. Subclasses that override this method should call self.build or super().build_for_cudagraph_capture. r)rrr)r)rr$r$r%build_for_cudagraph_capturesz4AttentionMetadataBuilder.build_for_cudagraph_capture draft_indexcCs|jd|ddS)a Build attention metadata for draft model. Uses build by default. Args: common_attn_metadata: The common attention metadata. draft_index: The index of the current draft operation. When speculating a chain of tokens, this index refers to the draft attempt for the i-th token. For tree-based attention, this index instead refers to the draft attempt for the i-th level in the tree of tokens. rT)rrrr)r)rrr$r$r%build_for_drafting"s z+AttentionMetadataBuilder.build_for_draftingrnum_query_headsrC use_alibiuse_sliding_windowuse_local_attentionnum_smsdcp_world_sizec Cr]r^r$) r)rrrrCrrrrrr$r$r%use_cascade_attention8rnz.AttentionMetadataBuilder.use_cascade_attention)r6FFrv) rrrrrrr r-rr,rrwrrzr|rxrr*rr}rrrrrrrrrrrrr$r$r$r%rs                rc @szeZdZUejed<ejed<ejed<eed<eed<eed<ejed<dejd ejd ejd ejd ed ejf ddZdS)AttentionLayer_q_scale_k_scale_v_scale_q_scale_float_k_scale_float_v_scale_float _prob_scalequerykeyvaluekv_cache attn_metadatar5cCrlr(r$)r)rrrrrr$r$r%forwardPszAttentionLayer.forwardN) rrrrxrr-floatrrr$r$r$r%rGs*    rcseZdZUdZeed<eed<eed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <dZ eed <eed <eed <eed<eed<eed<eed<fddZdejfddZZS)r>zBase class for attention implementations. Contains common attributes and initialization logic shared by both standard AttentionImpl and MLAAttentionImpl. Does not define a forward method - subclasses define their own forward interfaces. num_headsrDscaleFcan_return_lse_for_decode supports_pcp0supports_mtp_with_cp_non_trivial_interleave_sizeneed_to_return_lse_for_decodesupports_quant_query_inputsupports_per_head_quant_scalesrdcp_rankpcp_world_sizepcp_ranktotal_cp_world_size total_cp_rankcst|}zddlm}|j|_|j|_Wnty'd|_d|_Ynwzddlm }|j|_ |j|_ WntyId|_ d|_ Ynw|j |j|_ |j |j|j|_ |jdkob|j|_|S)Nr) get_dcp_groupr6) get_pcp_group)super__new__vllm.distributed.parallel_stater world_sizer rank_in_grouprAssertionErrorrrrrrrr)rKargsrr)rr __class__r$r%rs,         zAttentionImplBase.__new__ act_dtypecCrlr(r$)r)rr$r$r%process_weights_after_loadingsz/AttentionImplBase.process_weights_after_loading)rrrrr,r-rrrwrrrrrrrxrQr __classcell__r$r$rr%r>Zs&       r>c@seZdZdZedddddejdfdededededBde edBd edBd e d edBd e d e dBddfddZ e   d de de jde jde jde jdede jdBde jdBde jdBde jfddZd!ddZdS)" AttentionImplz6Standard attention implementation with forward method.Nr1rrDrrC alibi_slopessliding_windowrSlogits_soft_caprdkv_sharing_target_layer_namer5c Cr9r(r:) r)rrDrrCrrrSr rdr r$r$r%r*szAttentionImpl.__init__layerrrrrroutput output_scaleoutput_block_scalec Cr9r(r:) r)r rrrrrr r rr$r$r%rrzAttentionImpl.forward quant_keyrcCr])ab Does this attention implementation support fused output quantization. This is used by the AttnFusionPass to only fuse output quantization onto implementations that support it. :param quant_key: QuantKey object that describes the quantization op :return: is fusion supported for this type of quantization Fr$)r)rr$r$r%fused_output_quant_supportedrGz*AttentionImpl.fused_output_quant_supported)NNN)rr)rrrrrrr r,rrzr|r*rrxrrrrr$r$r$r%rsv         rc*@seZdZdZe  d&dededededeedBdedBd ed edBd ed edBd edBdedededededdde dBdedBddf(ddZ ede j de j de j de j de de j d e j ddfd!d"Zede j ee j e j fBde j de d#edee j e j dBff d$d%ZdS)'MLAAttentionImplzFMLA attention implementation with forward_mqa and forward_mha methods.NrrDrrCrrrSr rdr  q_lora_rank kv_lora_rankqk_nope_head_dimqk_rope_head_dim qk_head_dim v_head_dim kv_b_projrindexerq_pad_num_headsr5cCr9r(r:r)rrDrrCrrrSr rdr rrrrrrrrrr$r$r%r*zMLAAttentionImpl.__init__q kv_c_normedk_pekv_c_and_k_pe_cacherk_scaler cCr9)zMHA-style prefill forward pass.r:)r)rrrr rr!r r$r$r% forward_mharnzMLAAttentionImpl.forward_mhar cCr9zMQA-style decode forward pass.r:r)rr rr r$r$r% forward_mqarGzMLAAttentionImpl.forward_mqaNN)rrrrrr,rrzr|objectr*rxrrr"r~rr%r$r$r$r%rs        rc*@seZdZdZe  d dededededeedBdedBd ed edBd ed edBd edBdedededededdde dBdedBddf(ddZ ede j e e j e j fBde j dedede e j e j dBff ddZdS)!SparseMLAAttentionImplzSparse MLA attention implementation with only forward_mqa method. Sparse MLA implementations only support decode (MQA-style) attention. They do not support prefill (MHA-style) attention. NrrDrrCrrrSr rdr rrrrrrrrrrr5cCr9r(r:rr$r$r%r*rzSparseMLAAttentionImpl.__init__rr rr cCr9r#r:r$r$r$r%r%/rGz"SparseMLAAttentionImpl.forward_mqar&)rrrrrr,rrzr|r'r*rxrr~rrr%r$r$r$r%r(st      r(rSr5cCs |dS)Nfp8) startswith)rSr$r$r%is_quantized_kv_cache;r+r+ name_prefixattention_backend_cls builder_clscs$||j}t||fdfddiS)zN Return a new subclass where `get_builder_cls` returns `builder_cls`. r@csSr(r$r$r.r$r%rJsz,subclass_attention_backend..rr})r,r-r.namer$r/r%subclass_attention_backend?s r2 overridescCs||j}t||f|Sr(r0)r,r-r3r1r$r$r%)subclass_attention_backend_with_overridesNs r4)8abcrr dataclassesrrenumrtypingrrr r r r r numpyrrxtyping_extensionsr vllm.configrrWr!vllm.model_executor.layers.linearr9vllm.model_executor.layers.quantization.utils.quant_utilsrvllm.platforms.interfacer vllm.v1.attention.backends.utilsrvllm.v1.kv_cache_interfacerr|rr&r.rrrrrrrr>rrr(rwr+r}r2dictr4r$r$r$r%sf $        j ~O.7-