o پi(@s ddlmZmZmZmZddlZddlmZddZ ddej dej dej d ej d e d e d e d e de ej ej ej ej ffddZ ddej dej dej d ej dej dej d e d e d e d e de ej ej ej ej ffddZ       ddddddZ       ddddddZdS))ListOptionalTupleUnionNcCs"|dur|ddkr|S|S)N)stride contiguous)xr P/home/ubuntu/.local/lib/python3.10/site-packages/sgl_kernel/sparse_flash_attn.pymaybe_contiguouss"r T q_seqlens kv_seqlensvertical_indexes slash_indexes context_size block_size_M block_size_NcausalreturncCs|d}|d} |d} |d} ||d|} tj|| | |j|jd} tj|| | | |j|jd}tj|| | |j|jd}tj|| | | |j|jd}tjjj| ||||||||||| | |||fSNrr)dtypedevice) sizetorchzerosrrops sgl_kernelconvert_vertical_slash_indexesdefault)rrrrrrrr batch_size num_heads nnz_slash nnz_verticalnum_rows block_count block_offset column_count column_indexr r r r sT     r vertical_indices_countslash_indices_countc Cs|d} |d} |d} |d} ||d|}tj| | ||j|jd}tj| | || |j|jd}tj| | ||j|jd}tj| | || |j|jd}tjjj|||||||||||||| ||||fSr) rremptyrrrr(convert_vertical_slash_indexes_mergeheadr!)rrrrr+r,rrrrr"r#r$r%r&r'r(r)r*r r r r.DsX     r.F)return_softmax_lseoutcCsx|dur |jdd}dd|||fD\}}}tjjj||||||||| ||| | | o/|dkd\}}|r:||fS|S)aCompute attention with vertical and slash sparsity patterns. Most Arguments are the same with the flash_attn_func interface, except for 4 extra args: block_count and block_offset for slash sparsity patterns, and column_count and column_index for vertical sparsity patterns. For more details please refer to Appendix C.4.2 of paper https://arxiv.org/abs/2407.02490. Arguments: q: (batch_size, seqlen, nheads, headdim) k: (batch_size, seqlen, nheads_k, headdim) v: (batch_size, seqlen, nheads_k, headdim) block_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) block_offset: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_S) column_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) column_index: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_V) dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (batch_size, seqlen, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (batch_size, nheads, seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). NrcSg|]}t|qSr r .0r r r r z$sparse_attn_func..r)shaperrr fwd_sparser!)qkvr'r(r)r* dropout_p softmax_scalersoftcap alibi_slopes deterministicreturn_attn_probsr0r1 softmax_lser r r sparse_attn_funcs*3  rEcCs| dur |jdd} dd|||fD\}}}tjjj||||||||||d|| | | | d| ||o5| dkd\}}|r@||fS|S)ah Compute attention with vertical and slash sparsity patterns. Most Arguments are the same with the flash_attn_varlen_func interface, except for 4 extra args: block_count and block_offset for slash sparsity patterns, and column_count and column_index for vertical sparsity patterns. For more details please refer to Appendix C.4.2 of paper https://arxiv.org/abs/2407.02490. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. block_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) block_offset: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_S) column_count: (batch_size, nheads, cdiv(seqlen, BLOCK_M)) column_index: (batch_size, nheads, cdiv(seqlen, BLOCK_M), NNZ_V) cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nrr2cSr3r r4r5r r r r7 r8z+sparse_attn_varlen_func..Fr)r9rrrvarlen_fwd_sparser!)r;r<r=r'r(r)r* cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_kr>r?rr@rArBrCr0r1rDr r r sparse_attn_varlen_funcs6>  rK)T)r/NFr/NFF)typingrrrrrtorch.nnnnr Tensorintbooltupler r.rErKr r r r s   C    E W