o ۾iG@s~ddlmZmZmZmZddlZddlmZz ddlm Z dZ dZ Wne y:Z z ee Z dZ WYdZ [ ndZ [ wwz ddlmZdZdZWne y_Z z ee ZdZWYdZ [ ndZ [ wwz ddlmZdZdZWne yZ z ee ZdZWYdZ [ ndZ [ wwd Zd*d eeeeffd d Zd*d eeeeffd dZd*d eeeeffddZd*ded efddZd*ded eefddZddZejddddddddddddf dej deej deej deej deef ddZ!dddd dddd dddddddddddeddddfd!eeed"edefd#d$Z"     d+ddd%d&d'Z#     d+ddd%d(d)Z$dS),)OptionalUnionTupleListN) _vllm_fa2_CTF) _vllm_fa3_C)_flash_attn_fwdreturncCs.ts ddtfStj|ddkrdSdS)NFzFA2 is unavaible due to: r)Fz=FA2 is only supported on devices with compute capability >= 8TN) FA2_AVAILABLEFA2_UNAVAILABLE_REASONtorchcudaget_device_capabilitydevicer]/home/ubuntu/.local/lib/python3.10/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py_is_fa2_supported( rcCsBts ddtfStj|ddkstj|ddkrdSdS)NFzFA3 is unavaible due to: r )FzFA4 is only supported on devices with compute capability == 10r ) FA4_AVAILABLEFA4_UNAVAILABLE_REASONrrrrrrr_is_fa4_supported9rr fa_versioncCV|dvs Jd||dkrt|dS|dkrt|dS|dkr)t|dSdS)Nr Unsupported FA version: r rr$r%rrr r!rrrris_fa_version_supportedAs   r)cCr")Nr#r&r rr$r%r'r(rrrfa_version_unsupported_reasonJs   r*cCs"|dur|ddkr|S|S)Nr)stride contiguous)xrrrmaybe_contiguousZs"r/r+r+ cache_seqlens cu_seqlens_qcu_seqlens_k_new cache_leftpad page_sizecCsZt|}|dur |}tjj|||||||||| d| d| | | ||d|d||||}|S)Nrr)r/ropsrget_scheduler_metadata) batch_size max_seqlen_q max_seqlen_k num_heads_q num_heads_kvheaddimr1 qkv_dtype headdim_vr2r3r4r5max_seqlen_k_newcausal window_size has_softcap num_splitspack_gqa sm_marginscheduler_metadatarrrr7^s, r7rBrDc"CsJ|dus |dus Jd|dus|dusJd|dus$|dus$Jd| dur/|jdd} | dur6d}nt| dks>J| d | d f}d d |||fD\}}}t|}|dkr|durp|durp|durp|durptd |durxtd|d krtdtjj||||||dur|n||d||||| | d| |d |d | |o| d k|d\}} no|dkr|dusJdtjjj g|||dd||||dd||||ddddd|||| | |d |d | d||dd ||||R\}} }!}!nt d||r#|| fS|S)aU dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in K, V with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nz*cu_seqlens_k or seqused_k must be providedz>cu_seqlens_k and seqused_k cannot be provided at the same timez5seqused_k must be provided if block_table is providedr+r0r rrcSg|]}t|qSrr/.0r.rrr z*flash_attn_varlen_func..zHFA2 does not support scheduler_metadata, q_descale, k_descale, v_descalezFA2 does not support s_auxz#FA2 does not support num_splits > 1Fr$zAlibi is not supported in FA3Tr&) shapelenr empty_likeNotImplementedErrorr6r varlen_fwdrfwd ValueError)"qkvr9r2r: cu_seqlens_k seqused_kq_v dropout_p softmax_scalerArBsoftcap alibi_slopes deterministicreturn_attn_probs block_tablereturn_softmax_lseoutrG q_descale k_descale v_descalerDr!s_aux cp_world_sizecp_rankcp_tot_seqused_kreal_window_sizedummy_cu_seqlens_k softmax_lse_rrrflash_attn_varlen_funcsV            rq)rdrecCsv|dur |jdd}dd|||fD\}}}tjj||||||||| ||| | | o.|dkd\}}|r9||fS|S)aCompute attention with vertical and slash sparsity patterns. Most Arguments are the same with the flash_attn_func interface, except for 4 extra args: block_count and block_offset for slash sparsity patterns, and column_count and column_index for vertical sparsity patterns. For more details please refer to Appendix C.4.2 of paper https://arxiv.org/abs/2407.02490. Arguments: q: (batch_size, seqlen, nheads, headdim) k: (batch_size, seqlen, nheads_k, headdim) v: (batch_size, seqlen, nheads_k, headdim) block_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) block_offset: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_S) column_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) column_index: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_V) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nr+rIcSrJrrKrLrrrrNorOz$sparse_attn_func..r)rPrr6r fwd_sparse)rWrXrY block_count block_offset column_count column_indexr]r^rAr_r`rarbrdrerorrrsparse_attn_func9s*3 rwcCs| dur |jdd} dd|||fD\}}}tjj||||||||||d|| | | | d| ||o4| dkd\}}|r?||fS|S)al Compute attention with vertical and slash sparsity patterns. Most Arguments are the same with the flash_attn_varlen_func interface, except for 4 extra args: block_count and block_offset for slash sparsity patterns, and column_count and column_index for vertical sparsity patterns. For more details please refer to Appendix C.4.2 of paper https://arxiv.org/abs/2407.02490. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. block_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) block_offset: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_S) column_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) column_index: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_V) cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nr+rIcSrJrrKrLrrrrNrOz+sparse_attn_varlen_func..Fr)rPrr6rvarlen_fwd_sparse)rWrXrYrsrtrurvr2rZr9r:r]r^rAr_r`rarbrdrerorrrsparse_attn_varlen_funcs6> ry)N)rHNFrHNFF)%typingrrrrrtorch.nnnnrrr ImportErrorestrrrrflash_attn.cute.interfacer rrDEFAULT_FA_VERSIONboolrrr intr)r*r/bfloat16Tensorr7rqrwryrrrrs        1  : W