o i2@sBddlmZmZmZmZddlZddlmZddlZe dddkZ e r+ddl m Z nddlZ dd Zd d Zd d ZejdkrIejjZejjZndkdddddZdkdddddZeZeZedddddejdejdejdedededed ed!ed"eejd#ed$eejejejejffd%d&Zeddejdejdejdedededed ed!ed"eejd#ed$eejejejejffd'd(Zejdkrej j!jZ"neZ"ed)ddd * * +  ,    ,dldejdejdejd-ejd.ejd/ed0ededededed ed!ed"eejd#ed1eejd2eejd3eejd4ed$eejejejejff(d5d6Z#ed) * * +  ,    ,dldejdejdejd-ejd.ejd/ed0ededededed ed!ed"eejd#ed1eejd2eejd3eejd4ed$eejejejejff(d7d8Z$ejdkrej j!j#Z%ne#Z%ed9d:dd dkd;ejdejdejdejdeejd?eejd@eejdedededed ed!ed"eejdAedBeejd$ejf&dCdDZ&ed9 dkd;ejdejdejdejdeejd?eejd@eejdedededed ed!ed"eejdAedBeejd$ejf&dEdFZ'ejdkr+ej j!j&Z(ne&Z(edGd:dd  ,dmd;ejdejdejdejdeejd?eejd@eejd-ejd.ejd/ed0ededededed ed!ed"eejdAedBeejd4ed$ejf0dHdIZ)edG  ,dmd;ejdejdejdejdeejd?eejd@eejd-ejd.ejd/ed0ededededed ed!ed"eejdAedBeejd4ed$ejf0dJdKZ*ejdkrej j!j)Z+ne)Z+GdLdMdMej,j-Z.GdNdOdOej,j-Z/GdPdQdQej,j-Z0GdRdSdSej,j-Z1GdTdUdUej,j-Z2GdVdWdWej,j-Z3 +  , X +  , ,dndYdZZ4 +  , X +  , ,dnd[d\Z5 +  , X +  , ,dnd]d^Z6 +  , X +  , ,dnd_d`Z7 +  , X +  , ,dndadbZ8 +  , X +  , , dodcddZ9          , X + e  ,dpdfeeeejfdgeejdheejd1eejfdidjZ:dS)q)OptionalSequenceTupleUnionN!FLASH_ATTENTION_TRITON_AMD_ENABLEFALSETRUE) interface_facCs"|dur|ddkr|S|S)Nr )stride contiguous)xrU/home/ubuntu/vllm_env/lib/python3.10/site-packages/flash_attn/flash_attn_interface.pymaybe_contiguouss"rc Cs|dksJtj|\}}|dko|dk}|dko|dk}|dko%|dk}|dkr,dS|dkr6|s4dSdS|dkr.wrapr)namefn mutates_argsr*r+r0rrrnoop_custom_op_wrapper<r5)lib _stacklevelcCsdd}|dur |S|S)NcSr,r-rr.rrrr0Cr1z(noop_register_fake_wrapper..wrapr)opr3r7r8r0rrrnoop_register_fake_wrapperBr6r:zflash_attn::_flash_attn_forwardrr)r4r*qkv dropout_p softmax_scalecausalwindow_size_leftwindow_size_rightsoftcap alibi_slopesreturn_softmaxreturnc CsPdd|||fD\}}}t|||d| ||||||| d \} } } }| | | |fS)NcSg|]}t|qSrr.0rrrr Zz'_flash_attn_forward..)flash_attn_gpufwd)r;r<r=r>r?r@rArBrCrDrEout softmax_lseS_dmask rng_staterrr_flash_attn_forwardLs" rSc Csdd|||fD\}}}|j\} } } }|jd}t|}tj| | | ftj|j|jd}tjd|j|j|jd}| rQtj| | t| dt|df|j|j|jd}tjdtj |jd}||||fS) NcSrGrrHrIrrrrK{rLz,_flash_attn_forward_fake..r dtyperlayoutrrrUr) shaper empty_likeemptyfloat32rrVrUr)int64)r;r<r=r>r?r@rArBrCrDrE batch_sizeseqlen_q num_heads head_sizeseqlen_krOrPprRrrr_flash_attn_forward_fakems  , rfz&flash_attn::_flash_attn_varlen_forwardr F cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_k block_table leftpad_k seqused_k zero_tensorscCs`dd|||fD\}}}t|||d|||||| |||||| | | | |d\}}}}||||fS)NcSrGrrHrIrrrrKrLz._flash_attn_varlen_forward..)rM varlen_fwd)r;r<r=rhrirjrkr>r?r@rArBrCrDrErlrmrnrorOrPrQrRrrr_flash_attn_varlen_forwards2 rqcCsdd|||fD\}}}|du}|d}|j\}}}t|}tj||ftj|j|jd}tjd|j|j|jd}t |d}t |d}|rXtj||||f|j|j|jd}tjdtj |jd}||||fS) NcSrGrrHrIrrrrKrLz3_flash_attn_varlen_forward_fake..r rTrWrrXrZ) numelr[rr\r]r^rrVrUr)r_)r;r<r=rhrirjrkr>r?r@rArBrCrDrErlrmrnropaged_kvr`total_qrb_rOrPreseqlen_q_roundedseqlen_k_roundedrRrrr_flash_attn_varlen_forward_fakes       rxz flash_attn::_flash_attn_backward)dqdkdvdoutrOrPryrzr{ deterministicrRcCs\dd|||||fD\}}}}}t||||||||||| | | | | ||d|\}}}}|S)NcSrGrrHrIrrrrKrLz(_flash_attn_backward..)rMbwd)r|r;r<r=rOrPryrzr{r>r?r@rArBrCrDr}rR softmax_drrr_flash_attn_backwards8"rcCsdd|||||fD\}}}}}|durt|}|dur#t|}|dur,t|}|j\}}}}tj||t|df|jtjd}|S)NcSrGrrHrIrrrrK6rLz-_flash_attn_backward_fake..rrrU)rr\r[r]r)rr^)r|r;r<r=rOrPryrzr{r>r?r@rArBrCrDr}rRr`rarbrurrrr_flash_attn_backward_fake!s"    rz'flash_attn::_flash_attn_varlen_backwardcCsfdd|||||fD\}}}}}t|||||||||| | || | | |||||||d|\}}}}|S)NcSrGrrHrIrrrrKdrLz/_flash_attn_varlen_backward..)rM varlen_bwd)r|r;r<r=rOrPryrzr{rhrirjrkr>r?r@rArBrCrDr}rRrorrrr_flash_attn_varlen_backwardIsB"rcCsdd|||||fD\}}}}}| d}|j\}}}|dur&t|}|dur/t|}|dur8t|}tj||d|f|jtjd}|S)NcSrGrrHrIrrrrKrLz4_flash_attn_varlen_backward_fake..r rr)rrr[rr\r]rr^)r|r;r<r=rOrPryrzr{rhrirjrkr>r?r@rArBrCrDr}rRror`rtrbrurrrr _flash_attn_varlen_backward_fakes"      rc@$eZdZeddZeddZdS)FlashAttnQKVPackedFuncc  Csv| o|j} |dur|jdd}|dddddf|dddddf|dddddf} } }| d}|ddkrntjj| dd|dg} tjj| dd|dg} tjj|dd|dg}t| | |||||d|d||| o|dkd \}}}}| r| | | ||||||_ ||_ ||_ ||_ ||_||_||_|d d|f}| s|S|||fS) Nr rr rYrr@rArBrCrDrE.) requires_gradr[detachsizernn functionalpad_wrapped_flash_attn_forwardsave_for_backwardr>r?r@ window_sizerCrDr})ctxqkvr>r?r@rrCrDr}rEis_grad_enabledis_gradr;r<r= head_size_og out_paddedrPrQrRrOrrrforwardsB R    zFlashAttnQKVPackedFunc.forwardc Gs(|j\}}}}}}|jdddg|jddR} tj| |j|jd} |d} |} | ddkr@tjj |dd| dg} t | |||||| dddddf| dddddf| dddddf|j |j |j |jd|jd|j|j|j|d| d d|jd f} | dddddddddf S) NrrZrrr rYrR.r ) saved_tensorsr[rr]rUrrrrr_wrapped_flash_attn_backwardr>r?r@rrCrDr}) rr|argsr;r<r=rOrPrR qkv_shapedqkvr dout_paddedrrrbackwards:$  zFlashAttnQKVPackedFunc.backwardN__name__ __module__ __qualname__ staticmethodrrrrrrrs  /rc@r)FlashAttnVarlenQKVPackedFuncc Csv| o|j} |dur|jdd}|dddf|dddf|dddf}}}|d}|ddkretjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||||||d|d|| | o}|dkdd\}}}}| r| |||||||||_ ||_ ||_ ||_ ||_||_| |_| |_|dd|f}| s|S|||fS) Nr rrr rYrr@rArBrCrDrErl.)rr[rrrrrr"_wrapped_flash_attn_varlen_forwardrr> max_seqlenr?r@rrCrDr})rr cu_seqlensrr>r?r@rrCrDr}rErrr;r<r=rrrPrQrRrOrrrr sN @   z$FlashAttnVarlenQKVPackedFunc.forwardcGs(|j\}}}}}}} |jdddg|jddR} tj| |j|jd} |d} |} | ddkrAtjj |dd| dg} t | |||||| dddf| dddf| dddf|||j |j |j |j |j|jd|jd|j|j|j| d| d d|jd f} | dddddddddddf S) NrrrZrYrrr r.r )rr[rr]rUrrrrr#_wrapped_flash_attn_varlen_backwardrr>r?r@rrCrDr})rr|rr;r<r=rOrPrrRrrrrrrrrCsB$  z%FlashAttnVarlenQKVPackedFunc.backwardNrrrrrr s  7rc@r)FlashAttnKVPackedFuncc  Csj| o tdd||fD} |dur|jdd}|dddddf|dddddf} }|d}|ddkrhtjj|dd|dg}tjj| dd|dg} tjj|dd|dg}t|| |||||d|d||| o||dkd \}}}}| r| || ||||||_ ||_ ||_ ||_ ||_||_| |_|d d|f}| s|S|||fS) Ncs|]}|jVqdSr-rrIrrr x z0FlashAttnKVPackedFunc.forward..r rrr rrr.)anyr[rrrrrrrrr>r?r@rrCrDr})rr;kvr>r?r@rrCrDr}rErrr<r=rrrPrQrRrOrrrrisF 6    zFlashAttnKVPackedFunc.forwardcGs8|j\}}}}}}t|} |jdddg|jddR} tj| |j|jd} |d} |} | ddkrEtjj |dd| dg} t | |||||| | dddddf| dddddf|j |j |j|jd|jd|j|j|j|d| d d|jd f} | d d|jd f} | | dddddddddf S) NrrYrZrrrr r.r )rrr\r[r]rUrrrrrrr>r?r@rrCrDr})rr|rr;r<r=rOrPrRrykv_shapedkvrrrrrrs> $  zFlashAttnKVPackedFunc.backwardNrrrrrrh  2rc@r)FlashAttnVarlenKVPackedFunccCsx|o tdd||fD}|dur|jdd}|dddf|dddf}}|d}|ddkrbtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||||| | d| d| | |oz|dkdd \}}}}|r| ||||||||||_ ||_ ||_ ||_ | |_| |_| |_| |_| |_|d d|f}|s|S|||fS) Ncsrr-rrIrrrrrz6FlashAttnVarlenKVPackedFunc.forward..r rrr rYrr.)rr[rrrrrrrrr>rjrkr?r@rrCrDr})rr;rrhrirjrkr>r?r@rrCrDr}rErrr<r=rrrPrQrRrOrrrrsX *   z#FlashAttnVarlenKVPackedFunc.forwardcGsD|j\}}}}}}} } t|} |jdddg|jddR} tj| |j|jd} |d}|}|ddkrGtjj |dd|dg}t ||||||| | dddf| dddf|| |j |j |j|j|j|jd|jd|j|j|j| d| dd|jd f} | dd|jd f} | | dddddddddddddfS) NrrYrZrrr r.r )rrr\r[r]rUrrrrrrrjrkr>r?r@rrCrDr})rr|rr;r<r=rOrPrhrirRryrrrrrrrrsF $  "z$FlashAttnVarlenKVPackedFunc.backwardNrrrrrrs  ?rc@r) FlashAttnFuncc  Cs6| o tdd|||fD} |dur|jdd}|d}|ddkrNtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t|||||||d|d|| | ob|dkd \}}}}| r|||||||||_ ||_ ||_ ||_ ||_ | |_| |_|d d|f}| s|S|||fS) Ncsrr-rrIrrrr8rz(FlashAttnFunc.forward..r rrrrr r.)rr[rrrrrrrr>r?r@rrCrDr})rr;r<r=r>r?r@rrCrDr}rErrrrrPrQrRrOrrrr(sD     zFlashAttnFunc.forwardcGs |j\}}}}}}t|t|t|} } } |d} |} | ddkr5tjj|dd| dg} t| |||||| | | |j|j |j |j d|j d|j |j |j|d| dd|jdf} | dd|jdf} | dd|jdf} | | | dddddddddf S)Nrrrr r.r )rrr\rrrrrr>r?r@rrCrDr}r[)rr|rr;r<r=rOrPrRryrzr{rrrrrr[s<"  zFlashAttnFunc.backwardNrrrrrr'rrc@r)FlashAttnVarlenFunccCsP|o tdd|||fD}| dur|jdd} |d}|ddkrNtjj|dd|dg}tjj|dd|dg}tjj|dd|dg}t||||||||| | | d| d| | |of|dk|d \}}}}|r|||||||||||_ ||_ ||_ | |_ | |_ | |_| |_| |_||_|d d|f}|s|S|||fS) Ncsrr-rrIrrrrrz.FlashAttnVarlenFunc.forward..r rrYrrr r.)rr[rrrrrrrr>rjrkr?r@rrCrDr})rr;r<r=rhrirjrkr>r?r@rrCrDr}rErlrrrrrPrQrRrOrrrr~sV    zFlashAttnVarlenFunc.forwardcGs&|j\}}}}}}} } t|t|t|} } } |d}|}|ddkr7tjj|dd|dg}t||||||| | | || |j|j |j |j |j |j d|j d|j|j|j| d| dd|jdf} | dd|jdf} | dd|jdf} | | | ddddddddddddddfS)NrYrrr r.r )rrr\rrrrrrjrkr>r?r@rrCrDr}r[)rr|rr;r<r=rOrPrhrirRryrzr{rrrrrrsD"  &zFlashAttnVarlenFunc.backwardNrrrrrr}s  Arr r c Cs t|||||||||t S)adropout_p should be set to 0.0 during evaluation If Q, K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of Q, K, V. For multi-query and grouped-query attention (MQA/GQA), please see flash_attn_kvpacked_func and flash_attn_func. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i - window_size[0], i + window_size[1]] inclusive. Arguments: qkv: (batch_size, seqlen, 3, nheads, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rapplyrr) rr>r?r@rrCrDr}return_attn_probsrrrflash_attn_qkvpacked_funcs-rc Cs"t|||||||||| t S)a dropout_p should be set to 0.0 during evaluation If K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of K, V. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (batch_size, seqlen, nheads, headdim) kv: (batch_size, seqlen, 2, nheads_k, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) r;rr>r?r@rrCrDr}rrrrflash_attn_kvpacked_func!s?rc C$t|||||||||| | t S)a dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (batch_size, seqlen, nheads, headdim) k: (batch_size, seqlen, nheads_k, headdim) v: (batch_size, seqlen, nheads_k, headdim) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) r;r<r=r>r?r@rrCrDr}rrrrflash_attn_funcos=rc Cr)a dropout_p should be set to 0.0 during evaluation If Q, K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_varlen_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of Q, K, V. For multi-query and grouped-query attention (MQA/GQA), please see flash_attn_varlen_kvpacked_func and flash_attn_varlen_func. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i - window_size[0], i + window_size[1]] inclusive. Arguments: qkv: (total, 3, nheads, headdim), where total = total number of tokens in the batch. cu_seqlens: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into qkv. max_seqlen: int. Maximum sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr) rrrr>r?r@rrCrDr}rrrr flash_attn_varlen_qkvpacked_funcs2rcCs*t|||||||||| | | | | tS)a dropout_p should be set to 0.0 during evaluation If K, V are already stacked into 1 tensor, this function will be faster than calling flash_attn_func on Q, K, V since the backward pass avoids explicit concatenation of the gradients of K, V. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. kv: (total_k, 2, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr)r;rrhrirjrkr>r?r@rrCrDr}rrrrflash_attn_varlen_kvpacked_funcs"IrcCs.t|||||||||| | | | | ||tS)aq dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in K, V with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_attn_probs=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). S_dmask [optional, if return_attn_probs=True]: (batch_size, nheads, seqlen, seqlen). The output of softmax (possibly with different scaling). It also encodes the dropout pattern (negative means that location was dropped, nonnegative means it was kept). )rrrr)r;r<r=rhrirjrkr>r?r@rrCrDr}rrlrrrflash_attn_varlen_funcZs&IrT cache_seqlenscache_batch_idx cache_leftpadcCs|ddks Jd|ddksJddd|||fD\}}}| dur.|jdd} |durJt|trJtj|jd f|tj|jd }t|}t|}t| } t |||||||||| | |d| | | d | d|||\}}|rv||fS|S) a< If k and v are not None, k_cache and v_cache will be updated *inplace* with the new values from k and v. This is useful for incremental decoding: you can pass in the cached keys/values from the previous step, and update them with the new keys/values from the current step, and do attention with the updated cache, all in 1 kernel. If you pass in k / v, you must make sure that the cache is large enough to hold the new values. For example, the KV cache could be pre-allocated with the max sequence length, and you can use cache_seqlens to keep track of the current sequence lengths of each sequence in the batch. Also apply rotary embedding if rotary_cos and rotary_sin are passed in. The key @k will be rotated by rotary_cos and rotary_sin at indices cache_seqlens, cache_seqlens + 1, etc. If causal or local (i.e., window_size != (-1, -1)), the query @q will be rotated by rotary_cos and rotary_sin at indices cache_seqlens, cache_seqlens + 1, etc. If not causal and not local, the query @q will be rotated by rotary_cos and rotary_sin at indices cache_seqlens only (i.e. we consider all tokens in @q to be at position cache_seqlens). See tests/test_flash_attn.py::test_flash_attn_kvcache for examples of how to use this function. Supports multi-query and grouped-query attention (MQA/GQA) by passing in KV with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Note: Does not support backward pass. Arguments: q: (batch_size, seqlen, nheads, headdim) k_cache: (batch_size_cache, seqlen_cache, nheads_k, headdim) if there's no block_table, or (num_blocks, page_block_size, nheads_k, headdim) if there's a block_table (i.e. paged KV cache) page_block_size must be a multiple of 256. v_cache: (batch_size_cache, seqlen_cache, nheads_k, headdim) if there's no block_table, or (num_blocks, page_block_size, nheads_k, headdim) if there's a block_table (i.e. paged KV cache) k [optional]: (batch_size, seqlen_new, nheads_k, headdim). If not None, we concatenate k with k_cache, starting at the indices specified by cache_seqlens. v [optional]: (batch_size, seqlen_new, nheads_k, headdim). Similar to k. rotary_cos [optional]: (seqlen_ro, rotary_dim / 2). If not None, we apply rotary embedding to k and q. Only applicable if k and v are passed in. rotary_dim must be divisible by 16. rotary_sin [optional]: (seqlen_ro, rotary_dim / 2). Similar to rotary_cos. cache_seqlens: int, or (batch_size,), dtype torch.int32. The sequence lengths of the KV cache. cache_batch_idx: (batch_size,), dtype torch.int32. The indices used to index into the KV cache. If None, we assume that the batch indices are [0, 1, 2, ..., batch_size - 1]. If the indices are not distinct, and k and v are provided, the values updated in the cache might come from any of the duplicate indices. cache_leftpad: (batch_size,), dtype torch.int32. The index that the KV cache starts. If None, assume 0. block_table [optional]: (batch_size, max_num_blocks_per_seq), dtype torch.int32. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. rotary_interleaved: bool. Only applicable if rotary_cos and rotary_sin are passed in. If True, rotary embedding will combine dimensions 0 & 1, 2 & 3, etc. If False, rotary embedding will combine dimensions 0 & rotary_dim / 2, 1 & rotary_dim / 2 + 1 (i.e. GPT-NeoX style). alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. num_splits: int. If > 1, split the key/value into this many chunks along the sequence. If num_splits == 1, we don't split the key/value. If num_splits == 0, we use a heuristic to automatically determine the number of splits. Don't change this unless you know what you are doing. return_softmax_lse: bool. Whether to return the logsumexp of the attention scores. Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). r r z+k_cache must have contiguous last dimensionz+v_cache must have contiguous last dimensioncSrGrrHrIrrrrK&rLz+flash_attn_with_kvcache..NrrrZ) r r[ isinstanceintrfullint32rrrM fwd_kvcache)r;k_cachev_cacher<r= rotary_cos rotary_sinrrrrlr?r@rrCrotary_interleavedrD num_splitsreturn_softmax_lserOrPrrrflash_attn_with_kvcachesFlrr-) r r rgNFNNNF)NF)rgNFrrgNFF) rgNFrrgNFFN)NNNNNNNNNFrrgTNrF);typingrrrrrtorch.nnrosgetenvUSE_TRITON_ROCMflash_attn_triton_amdr rMflash_attn_2_cudarr'r) __version__library custom_op_torch_custom_op_wrapper register_fake_torch_register_fake_wrapperr5r:TensorfloatboolrrSrfops flash_attnrrqrxrrrrrrrautogradFunctionrrrrrrrrrrrrrrrrrs                      2       %        3      !        ?      ( R^WhVk > R Q I d b