o ٷi-@sTddlZddlmZddlmZmZmZeeZGdddeZ GdddeZ dS)N) init_logger)AttentionBackend AttentionImplAttentionMetadatac@sjeZdZUdZeed<edefddZede e fddZ ede fdd Z eded fd d Zd S)FlashAttentionBackendTaccept_output_bufferreturncCdS)NT)clsr r e/home/ubuntu/.local/lib/python3.10/site-packages/vllm_omni/diffusion/attention/backends/flash_attn.pysupports_attention_maskz-FlashAttentionBackend.supports_attention_maskcCsgdS)N)@`r r r r r get_supported_head_sizessz.FlashAttentionBackend.get_supported_head_sizescCr )N FLASH_ATTNr r r r r get_namerzFlashAttentionBackend.get_nameFlashAttentionImplcCstSN)rr r r r get_impl_clsrz"FlashAttentionBackend.get_impl_clsN)__name__ __module__ __qualname__rbool__annotations__ classmethodr staticmethodlistintrstrrtyperr r r r rs rc@s eZdZ   ddedededededBd ed dfd d Zed e j e e j dfBd e j fddZ de j de j de j de j d e j f ddZ dde j de j de j ded e j f ddZ dde j de j de j ded e j f ddZ dde j de j de j ded e j f ddZdS) rFN num_heads head_size softmax_scalecausal num_kv_headsprefixrcKs||_||_||_dSr)r&r)r()selfr&r'r(r)r*r+extra_impl_argsr r r __init__%s  zFlashAttentionImpl.__init__out.cCst|tr |dS|S)Nr) isinstancetuple)r/r r r _unwrap_flash_output3sz'FlashAttentionImpl._unwrap_flash_outputquerykeyvalueattention_maskcCsddlm}m}m}m}|jdksJd|d} |||||| |\} } } } \}}\}}|| | | f||||d|j|jd}| |}||| |d| S)Nr) _pad_input _unpad_input _upad_inputflash_attn_varlen_funcz0attention_mask must be 2D, (batch_size, seq_len)) cu_seqlens_q cu_seqlens_k max_seqlen_q max_seqlen_kr)r() /vllm_omni.diffusion.attention.backends.utils.far7r8r9r:ndimsizer)r(r2)r,r3r4r5r6r7r8r9r: query_lengthqkv indices_q cu_seq_lens_q cu_seq_lens_k max_length_q max_length_k out_unpadr r r _forward_varlen_masked8s,    z)FlashAttentionImpl._forward_varlen_masked attn_metadatac Csrddlm}m}|std|dur|jnd}|dur)t|r)|||||S|||||j|j d}| |S)z)CUDA/ROCm flash attention implementation.r)HAS_FLASH_ATTNflash_attn_funczFlashAttentionBackend requires Flash Attention. Please install one of: fa3-fwd, flash-attention, or flash-attn. Otherwise, use SDPA backend by setting DIFFUSION_ATTENTION_BACKEND=TORCH_SDPANrA) rBrQrR ImportError attn_masktorchanyrOr)r(r2) r,r3r4r5rPrQrRr6r/r r r forward_cuda\s* zFlashAttentionImpl.forward_cudac Csddlm}m}|std|dur|jnd}|dur)t|r)|||||S|dd\}} tj d|d| | tj |j d} | dd}| dd}| dd}||||| | | | |j |jd } || } | j|| g| jddRS) z#XPU flash attention implementation.r)rQr:zFlashAttentionBackend requires Flash Attention. Please assure vllm-xpu-kernels properly installed. Otherwise, use SDPA backend by setting DIFFUSION_ATTENTION_BACKEND=TORCH_SDPANr;r<)stepdtypedevice)r=r>r?r@r)r()rBrQr:rSrTrUrVrOrDarangeint32rZflattenr)r(r2reshapeshape) r,r3r4r5rPrQr:r6 batch_sizeq_len cu_seqlensr/r r r forward_xpus>     zFlashAttentionImpl.forward_xpuc CsPzddlm}Wn tytdw|r|jnd}|||||dddd}|S) z,NPU attention implementation using mindiesd.r)attention_forwardaFlashAttentionBackend NPU implementation requires MindIE-SD. Please install MindIE-SD to enable NPU attention support. For installation details, see https://gitcode.com/Ascend/MindIE-SDOtherwise, use SDPA backend by setting DIFFUSION_ATTENTION_BACKEND=TORCH_SDPANmanualfused_attn_scoreBNSD)rTopt_modeop_typelayout)mindiesdrdrSrT)r,r3r4r5rPrdr6outputr r r forward_npus$  zFlashAttentionImpl.forward_npu)FNr%r)rrrr"floatrr#r.r rUTensorr1r2rOrrWrcrmr r r r r$s & ) , 9r) rU vllm.loggerr/vllm_omni.diffusion.attention.backends.abstractrrrrloggerrrr r r r s