o ٷi{g@sddlZddlZddlmZddlmZzHddlmZmZm Z m Z m Z m Z ddlm Z mZmZz ddlmZdZWn eyFdZdZYnwejd krSdd lmZndZdd lmZWn eyged wdd lmZddlmZddlmZddl m!Z!ee"Z#dZ$gdZ%defddZ&de'de'fddZ(ej)r+gdZ*      dWdddej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d&ed'fd(d)Z.        dXd*ej/j0j1dej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d+e-d&ed'fd,d-Z2d*ej/j0j1d.ej+fd/d0Z3e&ej4ej5ej4e e gdd1       dYdej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d&ed'd2ej+fd3d4Z6e#7d5        dXd*ej/j0j1dej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d+e-d&ed'fd6d7Z8d*ej/j0j1d.ej+fd8d9Z9e(d:d;e:ed:sJej5ej;e e gdd1       dYdej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d&ed'd2ej+fde&ej=ej5ej=e e e gdd1    dZdej+dej+dej+d"e-d#ee,d%e-d&ed'd2ej+fd?d@Z>e#7dA        dXd*ej/j0j1dej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d+e-d&ed'fdBdCZ?d*ej/j0j1d.ej+fdDdEZ@ere:edFrde&ejAne#7dGe(dFdHe:edFsvJej5ejAe e e gdd1    d[dej+dej+dej+d#ee,d"e-d%e-d&ed'd2ej+fdIdJZBe#7dKndZBe#7dLe&ejCej5ejCe e e gdd1    d\dej+dej+dej+d!e,d#ee,d%e-d&ed'd2ej+fdMdNZDe#7dO        dXd*ej/j0j1dej+dej+dej+d eej+d!e,d"e-d#ee,d$e-d%e-d+e-d&ed'fdPdQZEd*ej/j0j1d.ej+fdRdSZFdSddTlm6Z6m>Z>zddUlmBZBWn eyGdZBYnwdZ=0.36.dev0'.Please install latest version of diffusers from source: pip3 install git+https://github.com/huggingface/diffusers.git) init_logger)ENV)UnifiedTemplatedRingAttention) UnifiedTemplatedUlyssesAttentioni)_native_attention_sdpa_cudnn_attention_sage_attention_flash_attention_3_native_npu_attention attn_backendcCshtj|dtj|dtj|dttjtr$tj|ddS|tjvr2tj|j dSdSN) r _backendspop _constraints_supported_arg_names isinstance_supports_context_paralleldictremovevalue)rr&g/home/ubuntu/.local/lib/python3.10/site-packages/cache_dit/parallelism/attention/_attention_dispatch.py_registry_pop_attn_backend?s  r(memberr%cCsHtt|}||_||_tt|||tj|<tj||tj |<dSr) str__new__r_name__value_setattr _member_map__member_names_append_value2member_map_)r)r% new_memberr&r&r'_set_new_attn_backendIs    r4)_native_attention_forward_op _sdpa_cudnn_attention_forward_op_npu_attention_forward_op)_parallel_configquerykey attn_mask dropout_p is_causalscale enable_gqa return_lser9rc  Cs|dur| j} | tvrtd| d|rtd|rtd| jjdkr5t|||||||||| | | S| jjdkrKt|||||||||| | | Std)NzxTemplated context parallel attention with attn_mask is only supported for native attention backend, but got forward_op: .z>Causal attention is not yet supported for templated attention.z1GQA is not yet supported for templated attention.rz@Reaching this branch of code is unexpected. Please report a bug.) __name___ATTENTION_OPS_ALLOW_ATTN_MASK ValueErrorcontext_parallel_config ring_degreerapplyulysses_degreer) r:r;r%r<r=r>r?r@rA forward_op backward_opr9forward_op_namer&r&r'-_unified_templated_context_parallel_attention_sV  rMctx _save_ctxc  Cs| r||||||_||_||_||_||_| rQ|dur"tdtjj j | dd| dd| dd|||ddd\} } | dd} | dd} | | fSdd|||fD\}}}tj j j||||||||d} | dddd } | S) NzE`attn_mask` is not yet supported for native flash attention with lse.r)r=r>r?cs |] }|ddddVqdSrrPrNpermute.0xr&r&r' z/_native_attention_forward_op..r:r;r%r<r=r>r?r@rrS)save_for_backwardr<r=r>r?r@rEtorchopsaten#_scaled_dot_product_flash_attention transposenn functionalscaled_dot_product_attentionrU)rNr:r;r%r<r=r>r?r@rArOr9outlser&r&r'r5sL      r5grad_outc Os|j\}}}|d|d|ddd|||fD\}}} tjjj||| |j|j|j|j |j d} | dddd} | dddd} tj j | ||| g| d d \} } }| dddd}| dddd}| dddd}|||fS) NTcsrQrRrTrVr&r&r'rYrZz0_native_attention_backward_op..r[rrPrrSF)outputsinputs grad_outputs retain_graph) saved_tensorsrequires_grad_r]rbrcrdr<r=r>r?r@rUautogradgrad)rNrgargskwargsr:r;r%query_tkey_tvalue_tre grad_out_t grad_query_t grad_key_t grad_value_t grad_querygrad_key grad_valuer&r&r'_native_attention_backward_ops6       r|) constraintssupports_context_parallelreturnc Cs|rtd| dur0dd|||fD\}}}tjjj||||||||d} | dddd} | St|||||||||tt| d } | S) NzDNative attention backend does not support setting `return_lse=True`.csrQrRrTrVr&r&r'rYrZz$_native_attention..r[rrPrrSrJrKr9) rEr]rbrcrdrUrMr5r|) r:r;r%r<r=r>r?r@rAr9rer&r&r'rs> rzRe-registered NATIVE attention backend to enable context parallelism with attn mask in cache-dit. You can disable this behavior by: export CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCH=0.c Cs| rtd| r||||||_||_||_||_||_dd|||fD\}}}tjj tjj j j tjj j||||||||d} Wdn1sPwY| dddd} | S) Nz:cudnn attention with sdpa does not support return_lse=TruecsrQrRrTrVr&r&r'rYWrZz3_sdpa_cudnn_attention_forward_op..r[rrPrrS)rEr\r<r=r>r?r@r]rb attention sdpa_kernel SDPBackendCUDNN_ATTENTIONrcrdrU rNr:r;r%r<r=r>r?r@rArOr9rer&r&r'r6<s0 r6cOtd)Nz>Backward for cudnn attention with sdpa is not implemented yet.NotImplementedErrorrNrgrprqr&r&r'!_sdpa_cudnn_attention_backward_opgr _SDPA_CUDNN _sdpa_cudnnc Csd} | durH|sHdd|||fD\}}}tjjtjjjjtjjj||||||||d} Wdn1s:wY| dddd} nt |||||||||t t | d } |r^| \} } |rd| | fS| S) Ncss$|] }|ddddVqdSrR)rU contiguousrVr&r&r'rYs"z(_sdpa_cudnn_attention..r[rrPrrSr) r]rbrrrrrcrdrUrMr6r) r:r;r%r<r=r>r?r@rAr9rfrer&r&r'rssB  rzRegistered new attention backend: _SDPA_CUDNN to enable context parallelism with attn mask in cache-dit. You can disable it by: export CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCH=0.c Csnd}|durt|||d|||d}|r|^}}} nt|||dd||d|tt|d }|r/|\}}|r5||fS|S)NNHD)qkv tensor_layoutr>sm_scalerAr8Fr)r rMr r ) r:r;r%r>r?rAr9rfre_r&r&r'rs> rzRe-registered SAGE attention backend to enable context parallelism with FP8 Attention in cache-dit. You can disable this behavior by: export CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCH=0.c Cs|durtd|rtd|dkrtd|dur!|jdd}| r(tdd } d} d }tdid |d |d |d|d|ddddddddd| ddd| ddddd|ddd| }| rw|\}}|ddd}||fS|S) zCFlash Attention 3 forward operation for cache-dit (inference only).Nz2`attn_mask` is not yet supported for flash-attn 3.z3`enable_gqa` is not yet supported for flash-attn 3.r8z6`dropout_p` > 0 is not yet supported for flash-attn 3.gzmFlash Attention 3 is configured for inference only, but _save_ctx=True was passed. Context will not be saved.rrFrrr softmax_scalecausalqv q_descale k_descale v_descale window_sizeattention_chunkrsoftcap num_splitsrpack_gqa deterministic sm_marginreturn_attn_probsrPr&)rEshapeloggerwarningflash_attn_3_funcrU)rNr:r;r%r<r=r>r?r@rArOr9rrrrerfr&r&r'_flash_attention_3_forward_opsn     rcOr)z3Flash Attention 3 backward operation for cache-dit.zaBackward pass for Flash Attention 3 with context parallelism is not implemented yet in cache-dit.rrr&r&r'_flash_attention_3_backward_op#sr_FLASH_3z>AttentionBackendName._FLASH_3 not found, creating new backend._flash_3c Csd}|durRd}d} d} tdid|d|d|d|d|d dd dd dd dd |ddd| ddddd| ddd|} |rQ| \} }|ddd}nt|||dd||d|tt|d } |rh| \} }|rn| |fS| S)Nrr8FrrrrrrrrrrrrrrrrrrrrPrr&)rrUrMrr) r:r;r%r?r>rAr9rfrrrrer&r&r'r7sz     rzRe-registered FLASH_3 attention backend to enable context parallelism with Ulysses Anything/Float8 in cache-dit. You can disable this behavior by: export CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCH=0.zHFlash Attention 3 not available, skipping _FLASH_3 backend registration.cCs|rtd|dur,t|||dd|durdt|jdn|tt|dd d}|St|||d|d|durAdt|jdn|d|tt |d }|S) NANPU attention backend does not support setting `return_lse=True`.BSNDg?rrP atten_mask input_layoutr? pre_tockens next_tockenshead_numrr) rErmathsqrtr MAX_TOKENsizerMr7_npu_attention_backward_op)r:r;r%r=r?rAr9rer&r&r'rs@ rzRe-registered _NATIVE_NPU attention backend to enable context parallelism You can disable this behavior by: export CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCH=0.c CsJ| rtd|dur|tj}t||||d|tt|dd d} | S)NrrrPrr)rEtor]boolrrrrr&r&r'r7s$  r7cOr)Nz:Backward pass is not implemented for Npu Fusion Attention.rrr&r&r'rrr)rr)rz;Skipped custom attention backend registration in cache-dit.)Nr8FNFF)Nr8FNFFTN)Nr8FNFFN)FNFN)NFFN)r8NFN)Gr]rtypingrcache_dit.platformsr#diffusers.models.attention_dispatchrrrrrr r r r flash_attn_interfacer r_flash_attn_3_available ImportError device_type torch_npur#diffusers.models._modeling_parallelrcache_dit.loggerrcache_dit.envsr_templated_ringr_templated_ulyssesrrCrr__all__r(r*r4%CACHE_DIT_ENABLE_CUSTOM_ATTN_DISPATCHrDTensorfloatrrMrnfunction FunctionCtxr5r|NATIVEregisterrinfor6rhasattrrrSAGErrrrr _NATIVE_NPUrr7rr&r&r&r's                   I    8 (    ,     +     /   *     >       :    (     !