o ۷iڦ@sLddlmZddlZddlZddlmZddlZddlmm Z ddlmZddl m Z ddl mZmZmZddlmZmZmZdd lmZmZeeZerUddlZeraddlZddlZndZerzed d rwdd lmZdd l m!Z!dZ"ndZ"eGdddej#Z$Gdddej#Z%Gdddej#Z&Gdddej#Z'GdddZ(GdddZ)Gdddej#Z*GdddZ+Gd d!d!Z,Gd"d#d#Z-Gd$d%d%Z.Gd&d'd'Z/Gd(d)d)Z0Gd*d+d+Z1Gd,d-d-Z2Gd.d/d/Z3Gd0d1d1Z4Gd2d3d3Z5Gd4d5d5Z6Gd6d7d7Z7Gd8d9d9Z8Gd:d;d;Z9Gdd?d?Z;Gd@dAdAZGdFdGdGZ?GdHdIdIZ@GdJdKdKZAGdLdMdMZBGdNdOdOZCGdPdQdQej#ZDGdRdSdSej#ZEGdTdUdUZFGdVdWdWZGGdXdYdYej#ZHGdZd[d[ej#ZIGd\d]d]ejj#ZJGd^d_d_ejj#ZKGd`dadaejj#ZLGdbdcdcZMGdddedeZNGdfdgdgZOGdhdidiZPGdjdkdkZQGdldmdmZRGdndodoZSGdpdqdqZTGdrdsdsZUGdtduduZVGdvdwdwZWGdxdydyZXGdzd{d{ZYGd|d}d}ZZGd~ddZ[GdddZ\GdddZ]e+eGe,e7fZ^e)e:e8eFeIeJeZfZ_e)e*Be+Be,Be-Be.Be/Be0Be2Be3Be4BeWBe[BeYBe\Be5Be6Be7Be8Be;Be9Be:BeBe?Be@BeABeBBeCBeDBeEBeFBeGBeTBeUBeVBe&BeOBe%BeIBeJBeKBeLBeMBeNBePBeQBeRBeSBZ`dS)) annotationsN)Callable)nn)IPAdapterMaskProcessor) deprecateis_torch_xla_availablelogging)is_torch_npu_availableis_torch_xla_versionis_xformers_available)is_torch_versionmaybe_allow_in_graph>z2.2)flash_attention)is_spmdTFcseZdZdZ                           dbdcfd.d/ Z  ddded5d6Zdfd8d9Z dgdhd=d>Zdid@dAZdjdCdDZ dkdldGdHZ  dmdndNdOZ dodQdRZ dpdqdTdUZ dgdrdXdYZ Sdpdsd\d]Zdtd^d_Zedud`daZZS)v Attentiona A cross attention layer. Parameters: query_dim (`int`): The number of channels in the query. cross_attention_dim (`int`, *optional*): The number of channels in the encoder_hidden_states. If not given, defaults to `query_dim`. heads (`int`, *optional*, defaults to 8): The number of heads to use for multi-head attention. kv_heads (`int`, *optional*, defaults to `None`): The number of key and value heads to use for multi-head attention. Defaults to `heads`. If `kv_heads=heads`, the model will use Multi Head Attention (MHA), if `kv_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. dim_head (`int`, *optional*, defaults to 64): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. bias (`bool`, *optional*, defaults to False): Set to `True` for the query, key, and value linear layers to contain a bias parameter. upcast_attention (`bool`, *optional*, defaults to False): Set to `True` to upcast the attention computation to `float32`. upcast_softmax (`bool`, *optional*, defaults to False): Set to `True` to upcast the softmax computation to `float32`. cross_attention_norm (`str`, *optional*, defaults to `None`): The type of normalization to use for the cross attention. Can be `None`, `layer_norm`, or `group_norm`. cross_attention_norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the group norm in the cross attention. added_kv_proj_dim (`int`, *optional*, defaults to `None`): The number of channels to use for the added key and value projections. If `None`, no projection is used. norm_num_groups (`int`, *optional*, defaults to `None`): The number of groups to use for the group norm in the attention. spatial_norm_dim (`int`, *optional*, defaults to `None`): The number of channels to use for the spatial normalization. out_bias (`bool`, *optional*, defaults to `True`): Set to `True` to use a bias in the output linear layer. scale_qk (`bool`, *optional*, defaults to `True`): Set to `True` to scale the query and key by `1 / sqrt(dim_head)`. only_cross_attention (`bool`, *optional*, defaults to `False`): Set to `True` to only use cross attention and not added_kv_proj_dim. Can only be set to `True` if `added_kv_proj_dim` is not `None`. eps (`float`, *optional*, defaults to 1e-5): An additional value added to the denominator in group normalization that is used for numerical stability. rescale_output_factor (`float`, *optional*, defaults to 1.0): A factor to rescale the output by dividing it with this value. residual_connection (`bool`, *optional*, defaults to `False`): Set to `True` to add the residual connection to the output. _from_deprecated_attn_block (`bool`, *optional*, defaults to `False`): Set to `True` if the attention block is loaded from a deprecated state dict. processor (`AttnProcessor`, *optional*, defaults to `None`): The attention processor to use. If `None`, defaults to `AttnProcessor2_0` if `torch 2.x` is used and `AttnProcessor` otherwise. N@F Th㈵>? query_dimintcross_attention_dim int | Noneheadskv_headsdim_headdropoutfloatbiasboolupcast_attentionupcast_softmaxcross_attention_norm str | Nonecross_attention_norm_num_groupsqk_normadded_kv_proj_dimadded_proj_bias bool | Nonenorm_num_groupsspatial_norm_dimout_biasscale_qkonly_cross_attentionepsrescale_output_factorresidual_connection_from_deprecated_attn_block processor'AttnProcessor' | Noneout_dimout_context_dimelementwise_affine is_causalc#svtddlm}m} m}!|dur|n|||_|dur!|jn|||_||_||_ |du|_ |dur7|n||_ ||_ | |_ ||_||_||_d|_|durR|n||_|dur[|n||_||_||_||_||_||_|jrt|dnd|_|dur||n||_||_| |_||_|jdur|jrtd|durtj |||dd|_!nd|_!|durt"||d |_#nd|_#| durd|_$d|_%n| d krtj&|||d |_$tj&|||d |_%n}| d kr||dd|d |_$||dd|d |_%nf| dkr tj&|||d|_$tj&|||d|_%nL| dkr!|!|||d |_$|!|||d |_%n6| dkr9|!|||d|_$|!|||d|_%n| dkrO| dd|d|_$| dd|d|_%ntd| d| dur`d|_'n1| d krmt&|j |_'n$| dkr|jdur{| }"n|j }"tj |"| ddd|_'ntd| dtj(||j|d|_)|jstj(|j |j|d|_*tj(|j |j|d|_+nd|_*d|_+||_,|jdurtj(| |j|d|_-tj(| |j|d|_.|jdurtj(| |j|d|_/n d|_/d|_-d|_.|jst0g|_1|j12tj(|j|j|d|j12t3|nd|_1|jdur.|js.tj(|j|j|d|_4nd|_4| dur| dur| d krStj&|||d |_5tj&|||d |_6nL| d krk||dd|d |_5||dd|d |_6n4| dkr|!||d|_5|!||d|_6n | dkrd|_5|!|||d|_6ntd| dd|_5d|_6|durt7t8dr|jrt9nt:}|;|dS)N) FP32LayerNormLpNormRMSNormFgrz`only_cross_attention` can only be set to True if `added_kv_proj_dim` is not None. Make sure to set either `only_cross_attention=False` or define `added_kv_proj_dim`.T num_channels num_groupsr2affine) f_channels zq_channels layer_norm)r2r:fp32_layer_norm)r:r"r2layer_norm_across_heads)r2rms_normrms_norm_across_headsl2r)pdimr2zunknown qk_norm: z. Should be one of None, 'layer_norm', 'fp32_layer_norm', 'layer_norm_across_heads', 'rms_norm', 'rms_norm_across_heads', 'l2'. group_normrzunknown cross_attention_norm: z.. Should be None, 'layer_norm' or 'group_norm'r"zC. Should be one of `None,'layer_norm','fp32_layer_norm','rms_norm'`scaled_dot_product_attention)<super__init__ normalizationr=r>r? inner_dim inner_kv_dimruse_biasis_cross_attentionrr$r%r3r4r fused_projectionsr8r9context_pre_onlypre_onlyr;r5r0scalersliceable_head_dimr*r1 ValueErrorr GroupNormrO SpatialNorm spatial_normnorm_qnorm_k LayerNorm norm_crossLinearto_qto_kto_vr+ add_k_proj add_v_proj add_q_proj ModuleListto_outappendDropout to_add_out norm_added_q norm_added_khasattrFAttnProcessor2_0 AttnProcessor set_processor)#selfrrrrrr r"r$r%r&r(r)r*r+r-r.r/r0r1r2r3r4r5r6r8r9rZr[r:r;r=r>r?norm_cross_num_channels __class__Z/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/attention_processor.pyrSks !                     zAttention.__init__use_xla_flash_attentionpartition_spectuple[str | None, ...] | NonereturnNonecCsp|r#tsdtddr dtrtddrd|rt|}nt|}nttdr.|jr.tnt }| |dS) ak Set whether to use xla flash attention from `torch_xla` or not. Args: use_xla_flash_attention (`bool`): Whether to use pallas flash attention kernel from `torch_xla` or not. partition_spec (`tuple[]`, *optional*): Specify the partition specification if using SPMD. Otherwise None. ztorch_xla is not available<2.3zEflash attention pallas kernel is supported from torch_xla version 2.32.4zPflash attention pallas kernel using SPMD is supported from torch_xla version 2.4rQN) rr rXLAFluxFlashAttnProcessor2_0XLAFlashAttnProcessor2_0rtrur0rvrwrx)ryrris_fluxr6r}r}r~set_use_xla_flash_attention7s   z%Attention.set_use_xla_flash_attentionuse_npu_flash_attentioncCs6|rt}nttdr|jrtnt}||dS)zR Set whether to use npu flash attention from `torch_npu` or not. rQN)AttnProcessorNPUrtrur0rvrwrx)ryrr6r}r}r~set_use_npu_flash_attentionXs z%Attention.set_use_npu_flash_attention'use_memory_efficient_attention_xformers attention_opCallable | Nonec Cst|do t|jtttf}t|dot|jtttt f}t|do*t|jt t t f}t|do7t|jt tf}|r|rG|rGtd|jtsPtdddtjsYtdz!d}|duri|\}} |j^}} tjdd |d } tj| | | } Wnty} z| d} ~ ww|rt|jj|jj|jj|jj|d } | |j!t|jd r| "|jj#j$j%n|rt&'d t |d} n|rt |jj|jj|jj(|jj)|d} | |j!t|jdr| j"|jj*dj$j%|jj*dj$j+d n|rt|d} nt,|d} n||r:tt-drtnt}||jj|jj|jj|jjd} | |j!t|jd r9| "|jj#j$j%nF|rpt |jj|jj|jj(|jj)d} | |j!t|jdro| j"|jj*dj$j%|jj*dj$j+d ntt-dr}|j.r}t/nt0} |1| dS)a Set whether to use memory efficient attention from `xformers` or not. Args: use_memory_efficient_attention_xformers (`bool`): Whether to use memory efficient attention from `xformers` or not. attention_op (`Callable`, *optional*): The attention operation to use. Defaults to `None` which uses the default attention operation from `xformers`. r6zhMemory efficient attention is currently not supported for custom diffusion for attention processor type zeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformersxformers)namezvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU N)r<r(cudadevicedtype)train_kv train_q_out hidden_sizerrto_k_custom_diffusionzMemory efficient attention with `xformers` might currently not work correctly if an attention mask is required for the attention operation.r)rr num_tokensr\rto_k_iprrQ)rrrr)rrrr\)2rt isinstancer6CustomDiffusionAttnProcessor$CustomDiffusionXFormersAttnProcessorCustomDiffusionAttnProcessor2_0AttnAddedKVProcessorAttnAddedKVProcessor2_0SlicedAttnAddedKVProcessorXFormersAttnAddedKVProcessorIPAdapterAttnProcessorIPAdapterAttnProcessor2_0IPAdapterXFormersAttnProcessorJointAttnProcessor2_0XFormersJointAttnProcessorNotImplementedErrorr ModuleNotFoundErrortorchr is_availabler^SUPPORTED_DTYPESrandnropsmemory_efficient_attention Exceptionrrrrload_state_dict state_dicttorweightrloggerinforr\rrXFormersAttnProcessorrur0rvrwrx)ryrris_custom_diffusionis_added_kv_processor is_ip_adapteris_joint_processorrop_fwop_bw_qer6attn_processor_classr}r}r~+set_use_memory_efficient_attention_xformersis              z5Attention.set_use_memory_efficient_attention_xformers slice_sizecCs|dur||jkrtd|d|jd|dur#|jdur#t|}n |dur,t|}n|jdur5t}nttdr@|jr@t nt }| |dS)z Set the slice size for attention computation. Args: slice_size (`int`): The slice size for attention computation. Nz slice_size z has to be smaller or equal to .rQ) r]r^r*rSlicedAttnProcessorrrtrur0rvrwrx)ryrr6r}r}r~set_attention_slices   zAttention.set_attention_slice'AttnProcessor'cCsVt|dr&t|jtjjr&t|tjjs&td|jd||j d||_dS)z Set the attention processor to use. Args: processor (`AttnProcessor`): The attention processor to use. r6z-You are removing possibly trained weights of z with N) rtrr6rrModulerr_modulespop)ryr6r}r}r~rxs    zAttention.set_processorreturn_deprecated_lora'AttentionProcessor'cCs|s|jSdS)a7 Get the attention processor in use. Args: return_deprecated_lora (`bool`, *optional*, defaults to `False`): Set to `True` to return the deprecated LoRA attention processor. Returns: "AttentionProcessor": The attention processor in use. Nr6)ryrr}r}r~ get_processor+s zAttention.get_processor hidden_states torch.Tensorencoder_hidden_statestorch.Tensor | Noneattention_maskc stt|jjjddhfdd|D}t|dkr1t d|d|jj j dfd d |D}|j||f||d |S) ah The forward method of the `Attention` class. Args: hidden_states (`torch.Tensor`): The hidden states of the query. encoder_hidden_states (`torch.Tensor`, *optional*): The hidden states of the encoder. attention_mask (`torch.Tensor`, *optional*): The attention mask to use. If `None`, no mask is applied. **cross_attention_kwargs: Additional keyword arguments to pass along to the cross attention. Returns: `torch.Tensor`: The output of the attention layer. ip_adapter_masksip_hidden_statescs$g|]\}}|vr|vr|qSr}r}).0krattn_parametersquiet_attn_parametersr}r~ Vsz%Attention.forward..rzcross_attention_kwargs z are not expected by z and will be ignored.csi|] \}}|vr||qSr}r})rrw)rr}r~ ]sz%Attention.forward..rr) setinspect signaturer6__call__ parameterskeysitemslenrwarningr|__name__)ryrrrcross_attention_kwargs unused_kwargsr}rr~forward9s&  zAttention.forwardtensorcCsL|j}|j\}}}||||||}|dddd|||||}|S)ac Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size // heads, seq_len, dim * heads]`. `heads` is the number of heads initialized while constructing the `Attention` class. Args: tensor (`torch.Tensor`): The tensor to reshape. Returns: `torch.Tensor`: The reshaped tensor. rrr<)rshapereshapepermute)ryr head_size batch_sizeseq_lenrNr}r}r~batch_to_head_dimgs  "zAttention.batch_to_head_dimrcCs~|j}|jdkr|j\}}}d}n|j\}}}}|||||||}|dddd}|dkr=|||||||}|S)a  Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size, seq_len, heads, dim // heads]` `heads` is the number of heads initialized while constructing the `Attention` class. Args: tensor (`torch.Tensor`): The tensor to reshape. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the tensor. If `3`, the tensor is reshaped to `[batch_size * heads, seq_len, dim // heads]`. Returns: `torch.Tensor`: The reshaped tensor. rr<rr)rndimrrr)ryrr8rrrrN extra_dimr}r}r~head_to_batch_dimxs  zAttention.head_to_batch_dimquerykeyc Cs|j}|jr|}|}|dur*tj|jd|jd|jd|j|jd}d}n|}d}tj|||dd||j d}~|j rE|}|j dd}~| |}|S) ak Compute the attention scores. Args: query (`torch.Tensor`): The query tensor. key (`torch.Tensor`): The key tensor. attention_mask (`torch.Tensor`, *optional*): The attention mask to use. If `None`, no mask is applied. Returns: `torch.Tensor`: The attention probabilities/scores. Nrr<rrrL)betaalpharN) rr$r!remptyrrbaddbmm transposer\r%softmaxr) ryrrrr baddbmm_inputrattention_scoresattention_probsr}r}r~get_attention_scoress2    zAttention.get_attention_scores target_lengthrc Cs|j}|dur |S|jd}||krA|jjdkr7|jd|jd|f}tj||j|jd}tj||gdd}n tj |d|fd d }|d kr]|jd||kr[|j |d|jd|d }|S|d krs| d}|j |d|jd|d }|S)a Prepare the attention mask for the attention computation. Args: attention_mask (`torch.Tensor`): The attention mask to prepare. target_length (`int`): The target length of the attention mask. This is the length of the attention mask after padding. batch_size (`int`): The batch size, which is used to repeat the attention mask. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the attention mask. Can be either `3` or `4`. Returns: `torch.Tensor`: The prepared attention mask. NrLmpsrr<rrrr)valuerrN output_size) rrrtyperzerosrcatrupadrepeat_interleave unsqueeze) ryrr rr8rcurrent_length padding_shapepaddingr}r}r~prepare_attention_masks,    z Attention.prepare_attention_maskcCsf|jdus Jdt|jtjr||}|St|jtjr1|dd}||}|dd}|SJ)aG Normalize the encoder hidden states. Requires `self.norm_cross` to be specified when constructing the `Attention` class. Args: encoder_hidden_states (`torch.Tensor`): Hidden states of the encoder. Returns: `torch.Tensor`: The normalized encoder hidden states. NzGself.norm_cross must be defined to call self.norm_encoder_hidden_statesr<r)rerrrdr_r)ryrr}r}r~norm_encoder_hidden_statess      z$Attention.norm_encoder_hidden_statescCs|jjjj}|jjjj}|jsYt|jjj|jjj|j jjg}|j d}|j d}t j |||j ||d|_|jj||j rXt|jjj|jjj|j jjg}|jj|nAt|jjj|j jjg}|j d}|j d}t j |||j ||d|_|jj||j rt|jjj|j jjg}|jj|t|dddurt|dddurt|dddurt|jjj|jjj|jjjg}|j d}|j d}t j |||j||d|_|jj||jrt|jjj|jjj|jjjg}|jj|||_dS)Nr<r)r"rrrlrjrk)rgrdatarrrXrrrhrirrrfrWto_qkvcopy_r"to_kvgetattrrlrjrkr+ to_added_qkvrY)ryfuserrconcatenated_weights in_features out_featuresconcatenated_biasr}r}r~fuse_projectionssL  "  "      zAttention.fuse_projections)NrNrrFFFNrNNTNNTTFrrFFNNNNFTF)8rrrrrrrrrrr r!r"r#r$r#r%r#r&r'r(rr)r'r*rr+r,r-rr.rr/r#r0r#r1r#r2r!r3r!r4r#r5r#r6r7r8rr9rr:r#r;r#)NF)rr#rrrr)rr#rrN)rr#rrrr)rrrr)r6rrr)F)rr#rrNN)rrrrrrrr)rrrr)r)rrr8rrr)rrrrrrrr) rrr rrrr8rrr)rrrr)T)r __module__ __qualname____doc__rSrrrrrxrrrrr rrrno_gradr' __classcell__r}r}r{r~r3sn9P !     .  0 3rcs(eZdZd fdd Zdd d ZZS)!SanaMultiscaleAttentionProjection in_channelsrnum_attention_heads kernel_sizerrc sPtd|}tj||||d|dd|_tj||dddd|dd|_dS)NrrF)rgroupsr"r<r)r3r")rRrSrConv2dproj_inproj_out)ryr0r1r2channelsr{r}r~rSIs "z*SanaMultiscaleAttentionProjection.__init__rrcCs||}||}|Sr()r5r6ryrr}r}r~r\s  z)SanaMultiscaleAttentionProjection.forward)r0rr1rr2rrrrrrrrr*r+rSrr.r}r}r{r~r/Hsr/csPeZdZdZ       d&d'fdd Zd(dd Zd(d!d"Zd)d$d%ZZS)*SanaMultiscaleLinearAttentionz(Lightweight multi-scale linear attentionNrr batch_normV瞯||}||} ||} | |} | d|jdf} | d|jdf} | d|jdf} |j durr| | } |j dur|| | } |durdd} | |g|R}| |g|R}| dd| dd| dd}}}| dd| dd| dd} } } | d} | d}| |}|j\}}}}g}t|D]}||dddf}tj|dd}| ||ddd|ddf}| ||ddd|ddf}| ||ddd|ddf}tj|||d|gdd}tj|||d|gdd}tj|||d|gdd}tj|||d dd }| d}t|d d d ||f}||qtj|d d}| dddd }|j| |fdd\}}|jd |}|jd|}t|d r||}||fS)NrrLcSst|ddddf}|ddddf}|||||j}|||||j}tj||gdddS)N.rrr<rLrr)r!rrrstackflatten)x freqs_cos freqs_sinx_evenx_oddcossinr}r}r~apply_rotary_embs z8MochiAttnProcessor2_0.__call__..apply_rotary_embr<F)as_tuplerr dropout_pr;rrrq)rgrhri unflattenrrbrcrlrjrkrrrsrsizerrangernonzerorirrurQrrosplit_with_sizesrnrtrq) ryrerrrrgrrr  encoder_query encoder_key encoder_valuerqsequence_lengthencoder_sequence_length total_lengthrrrrN attn_outputsidxmaskvalid_prompt_token_indicesvalid_encoder_queryvalid_encoder_keyvalid_encoder_value valid_query valid_key valid_value attn_outputvalid_sequence_lengthr}r}r~rsv               (               zMochiAttnProcessor2_0.__call__r() rerfrrrrrrrgrrrrr*r+r,rSrr}r}r}r~r_s  r_c@s"eZdZdZ   ddd d ZdS)rwzJ Default processor for performing attention-related computations. NrerrrrrrtembrcOst|dks|dddurd}tdd||} |jdur#|||}|j} | dkr>|j\} } } }|| | | |dd}|durE|jn|j\} }}|||| }|j durd| |dddd}| |}|durp|}n|j rx| |}| |}||}||}||}||}||||}t||}||}|jd|}|jd|}| dkr|dd | | | |}|jr|| }||j}|S) Nrr\The `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.1.0.0rr<rrLr)rgetrrarrviewrrrOrgrerrhrirr rbmmrrnrr4r3)ryrerrrrargsr^deprecation_messageresidual input_ndimrchannelheightwidthr}rrrr r r}r}r~rTsH               zAttnProcessor.__call__NNN rerrrrrrrrrrrrr*r+r,rr}r}r}r~rwOs rwc@eZdZdZ      ddfdd Z  ddddZZS)raK Processor for implementing attention for the Custom Diffusion method. Args: train_kv (`bool`, defaults to `True`): Whether to newly train the key and value matrices corresponding to the text features. train_q_out (`bool`, defaults to `True`): Whether to newly train query matrices corresponding to the latent image features. hidden_size (`int`, *optional*, defaults to `None`): The hidden size of the attention layer. cross_attention_dim (`int`, *optional*, defaults to `None`): The number of channels in the `encoder_hidden_states`. out_bias (`bool`, defaults to `True`): Whether to include the bias parameter in `train_q_out`. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. TNrrr#rrrrr/r r!ct||_||_||_||_|jr*tj|p||dd|_tj|p$||dd|_ |jrStj||dd|_ t g|_ |j tj|||d|j t|dSdSNFrPrRrSrrrrrrfrto_v_custom_diffusionto_q_custom_diffusionrmto_out_custom_diffusionrorpryrrrrr/r r{r}r~rS  z%CustomDiffusionAttnProcessor.__init__rerrrrrrrcCs|j\}}}||||}|jr|||jjj}n |||jjj}|dur0d} |}n d} |jr:| |}|j rd| ||j jj} | ||j jj} | |jjj} | |jjj} n | |} ||} | rt| } | ddddddfd| ddddddf<| | d| | } | | d| | } ||}|| } || } ||| |} t| | }||}|jr|jd|}|jd|}|S|jd|}|jd|}|S)NFTr<rr)rrrrrrgrrrerrrrrhrir ones_likedetachrr rrrrn)ryrerrrrr}rr crossattnrr rr r}r}r~rsH     4     z%CustomDiffusionAttnProcessor.__call__TTNNTr rr#rr#rrrrr/r#r r!r) rerrrrrrrrrrr*r+r,rSrr.r}r}r{r~rsrc@ eZdZdZ  d dd d ZdS)rz Processor for performing attention-related computations with extra learnable key and value matrices for the text encoder. NrerrrrrrrcOst|dks|dddurd}tdd||}||jd|jdddd}|j\} } } ||| | }|dur=|}n|jrE||}| |dddd}| |} | | } | |} | |}| | } | |}|js||}||}| |}| |}tj| |gdd}tj||gdd}n| }|}|| ||}t||}||}|jd|}|jd|}|dd |j}||}|S) Nrr\rrr<rLrrr)rrrrrrrrerrOrgrrjrkr1rhrirrr rrrnr)ryrerrrrr^rrrr}rrencoder_hidden_states_key_proj encoder_hidden_states_value_projrr r r}r}r~rsF  "              zAttnAddedKVProcessor.__call__r)rrr}r}r}r~rs  rc@(eZdZdZddZ  ddd dZdS)rz Processor for performing scaled dot-product attention (enabled by default if you're using PyTorch 2.0), with extra learnable key and value matrices for the text encoder. cCr`)NrQzWAttnAddedKVProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSH z AttnAddedKVProcessor2_0.__init__NrerrrrrrrcOst|dks|dddurd}tdd||}||jd|jdddd}|j\} } } |j|| | dd }|dur?|}n|jrG||}| |dddd}| |} |j | dd } | |} | |}|j | dd } |j |dd }|js||}||}|j |dd }|j |dd }tj| |gdd }tj||gdd }n| }|}tj| |||d d d }|dd| d|jd}|jd|}|jd|}|dd|j}||}|S)Nrr\rrr<rLrr)r8rrF attn_maskrtr;r)rrrrrrrrerrOrgrrjrkr1rhrirrrurQrrn)ryrerrrrr^rrrr}rrrrrr r}r}r~rNsH  "        z AttnAddedKVProcessor2_0.__call__r)rrr}r}r}r~rBs  rc@r)rYAttention processor used typically in processing the SD3-like self-attention projections.cCr`)NrQzUJointAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrdzJointAttnProcessor2_0.__init__Nrerrtorch.FloatTensorrrtorch.FloatTensor | NonercOsH|}|jd}||} ||} ||} | jd} | |j} | |d|j| dd} | |d|j| dd} | |d|j| dd} |jdurQ|| } |jdur[|| } |dur| |}| |}| |}||d|j| dd}||d|j| dd}||d|j| dd}|j dur| |}|j dur| |}tj| |gdd} tj| |gdd} tj| |gdd} tj| | | ddd}|dd|d|j| }|| j}|dur |ddd|jdf|dd|jddf}}|js ||}|jd|}|jd|}|dur"||fS|S) NrrLr<rrrFrs)rrgrhrirrrrbrcrlrjrkrrrsrrrurQrrrrZrqrn)ryrerrrrr^rrrrr rUhead_dim encoder_hidden_states_query_projrrr}r}r~rsl                        zJointAttnProcessor2_0.__call__r) rerrrrrrrrrrr}r}r}r~r rc@r)PAGJointAttnProcessor2_0rcCr`)NrQzXPAGJointAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrz!PAGJointAttnProcessor2_0.__init__Nrerrrrrrrc!Cs|}|j}|dkr|j\}}} } |||| | dd}|j} | dkr8|j\}}} } |||| | dd}|jd} |d\} }|d\}}|jd}|| }|| }|| }||}| |}| |}t j ||gdd}t j ||gdd}t j ||gdd}|jd}||j }||d|j |dd}||d|j |dd}||d|j |dd}tj|||ddd } | dd|d|j |} | |j} | ddd|jdf| dd|jddf} }|jd| } |jd| } |js ||}|dkr| dd ||| | } | dkr,|dd ||| | }|jd}||}||}||}||}| |}| |}t j ||gdd}t j ||gdd}t j ||gdd}|jd}||j }||d|j |dd}||d|j |dd}||d|j |dd}|d}t j||f|j|jd } td | d| d| f<| d| d| fd| dd} tj|||| ddd }|dd|d|j |}||j}|ddd|jdf|dd|jddf}}|jd|}|jd|}|js||}|dkr/|dd ||| | }| dkr@|dd ||| | }t | |g}t ||g}||fS)Nrr<rrrrLrFrsrr-infr)rrrrchunkrgrhrirlrjrkrrrrurQrrrrnrZrqrvrrr!fill_diagonal_r)!ryrerrrrrrrrrcontext_input_ndimidentity_block_sizehidden_states_orghidden_states_ptbencoder_hidden_states_orgencoder_hidden_states_ptb query_orgkey_org value_org$encoder_hidden_states_org_query_proj"encoder_hidden_states_org_key_proj$encoder_hidden_states_org_value_projrUr query_ptbkey_ptb value_ptb$encoder_hidden_states_ptb_query_proj"encoder_hidden_states_ptb_key_proj$encoder_hidden_states_ptb_value_projr full_maskr}r}r~rs                                z!PAGJointAttnProcessor2_0.__call__r)rrr}r}r}r~r  rc@r)PAGCFGJointAttnProcessor2_0rcCr`)NrQz[PAGCFGJointAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrz$PAGCFGJointAttnProcessor2_0.__init__Nrerrrrrrrc%Os|}|j}|dkr|j\} } } } || | | | dd}|j} | dkr8|j\} } } } || | | | dd}|jd}|d\}}}t||g}|d\}}}t||g}|jd} ||}||}| |}| |}| |}| |}tj||gdd}tj||gdd}tj||gdd}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}tj|||dd d }|dd| d|j |}||j}|ddd|jdf|dd|jddf}}|jd|}|jd|}|js||}|dkr+|dd | | | | }| dkr<|dd | | | | }|jd} ||}||}| |}| |} | |}!| |}"tj|| gdd}tj||!gdd}tj||"gdd}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|d}#tj|#|#f|j|jd }$td |$d|d|f<|$d|d|fd|$dd}$tj||||$dd d}|dd| d|j |}||j}|ddd|jdf|dd|jddf}}|jd|}|jd|}|js.||}|dkr?|dd | | | | }| dkrP|dd | | | | }t||g}t||g}||fS)Nrr<rrrrrLrFrsrrrr)rrrrrrrrgrhrirlrjrkrrurQrrrrnrZrqrvrrr!rr)%ryrerrrrr^rrrrrrrrhidden_states_uncondrrencoder_hidden_states_uncondrrrrrrrrrUrrrrrrrrrr}r}r~rs                                z$PAGCFGJointAttnProcessor2_0.__call__r)rrr}r}r}r~rrrc@r)FusedJointAttnProcessor2_0rcCr`NrQzPAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rS(rdz#FusedJointAttnProcessor2_0.__init__NrerrrrrrrcOsZ|}|j}|dkr|j\} } } } || | | | dd}|j} | dkr8|j\} } } } || | | | dd}|jd} ||}|jdd}tj||dd\}}}||}|jdd}tj||dd\}}}tj||gdd}tj||gdd}tj||gdd}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}t j |||dd d }|dd | d|j |}| |j}|ddd|jdf|dd|jddf}}|jd|}|jd|}|js||}|dkr|dd  | | | | }| dkr)|dd  | | | | }||fS) Nrr<rrrLrrrFrsr)rrrrrrsplitr!rrrurQrrrrnrZrq)ryrerrrrr^rrrrrrrqkv split_sizerrr  encoder_qkvrrrrUrr}r}r~r,sV          z#FusedJointAttnProcessor2_0.__call__r)rrr}r}r}r~r%rrc@,eZdZdZddddZ  ddddZdS)r Processor for implementing memory efficient attention using xFormers. Args: attention_op (`Callable`, *optional*, defaults to `None`): The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. NrrcC ||_dSr(rryrr}r}r~rS z#XFormersJointAttnProcessor.__init__rerrrrrrrcOs|}||}||} ||} ||}|| } || } |jdur0||}|jdur:|| } |dur||} ||} | |} || } || } || } |j durl| | } |j durv| | } t j || gdd}t j | | gdd} t j | | gdd} tjj|| | ||j|jd}||j}||}|dur|ddd|jdf|dd|jddf}}|js||}|jd|}|jd|}|dur||fS|S)Nr<r attn_biasopr\r)rgrhrir contiguousrbrcrlrjrkrrrsrrrrrrr\rrrrrZrqrn)ryrerrrrr^rrrr rrrr}r}r~rsR                 z#XFormersJointAttnProcessor.__call__r(rrr)rrr}r}r}r~rt  rc@,eZdZdZddZ    ddddZdS)AllegroAttnProcessor2_0z Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the Allegro model. It applies a normalization layer and rotary embedding on the query and key vector. cCr`)NrQzWAllegroAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrz AllegroAttnProcessor2_0.__init__NrerrrrrrrrgrcCs6|}|jdur |||}|j}|dkr(|j\} } } } || | | | dd}|dur/|jn|j\} } }|durL||| | }|| |jd|jd}|jdur^||dddd}||}|durj|}n|j rr| |}| |}| |}|jd}||j}|| d|j|dd}|| d|j|dd}|| d|j|dd}|dur|j sddlm}|||d|d}|||d|d}tj||||ddd }|dd| d|j|}||j}|jd|}|jd|}|dkr |dd | | | | }|jr||}||j}|S) Nrr<rrL)apply_rotary_emb_allegrorrFrr)rarrrrrrrOrgrerrhrirX embeddingsrrurQrrrrnr4r3)ryrerrrrrgrrrrrrr}rrrr rUrrr}r}r~rsV              z AllegroAttnProcessor2_0.__call__NNNNrerrrrrrrrrrgrrrrr}r}r}r~rs rc@&eZdZdZddZ d dd d ZdS)AuraFlowAttnProcessor2_0z;Attention processor used typically in processing Aura Flow.cC$ttdstddrtddSdS)NrQr2.1zAuraFlowAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to at least 2.1 or above as we use `scale` in `F.scaled_dot_product_attention()`. rtrur rbrcr}r}r~rS* z!AuraFlowAttnProcessor2_0.__init__NrerrrrrcOsD|jd}||}||}||} |dur'||} ||} ||} |jd} | |j}||d|j|}||d|j|}| |d|j|} |j durV| |}|j dur`| |}|dur| |d|j|} | |d|j|} | |d|j|} |j dur| | } |j dur| | } t j| |gdd}t j| |gdd}t j| | gdd} |dd}|dd}| dd} tj||| d|jdd}|dd|d|j|}||j}|dur|dd|jddf|ddd|jdf}}|jd|}|jd|}|dur||}|dur ||fS|S) NrrLr<rrrFrtr\r;)rrgrhrirlrjrkrrrbrcrrrsrrrrurQr\rrrrnrq)ryrerrrr^rrrr rrrrUrr}r}r~r0sf                          z!AuraFlowAttnProcessor2_0.__call__r(rerrrrrrrrr}r}r}r~r'  rc@r)FusedAuraFlowAttnProcessor2_0zRAttention processor used typically in processing Aura Flow with fused projections.cCr)NrQrrzFusedAuraFlowAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to at least 2.1 or above as we use `scale` in `F.scaled_dot_product_attention()`. rrcr}r}r~rSrz&FusedAuraFlowAttnProcessor2_0.__init__NrerrrrrcOsf|jd}||}|jdd}tj||dd\} } } |dur7||} | jdd}tj| |dd\} }}| jd}||j}| |d|j|} | |d|j|} | |d|j|} |jdurf|| } |jdurp|| } |dur| |d|j|} ||d|j|}||d|j|}|j dur| | } |j dur| |}tj | | gdd} tj || gdd} tj || gdd} | dd} | dd} | dd} t j| | | d|jdd }| dd|d|j|}|| j}|dur|dd|jddf|ddd|jdf}}|jd|}|jd|}|dur(||}|dur1||fS|S) NrrLrrr<rrFr)rrrrr!rrrbrcrrrsrrrurQr\rrrrnrq)ryrerrrr^rrrrrr rrrrrUrr}r}r~rsn                       z&FusedAuraFlowAttnProcessor2_0.__call__r(rrr}r}r}r~rrrc@(eZdZdZddZ  ddddZdS)CogVideoXAttnProcessor2_0 Processor for implementing scaled dot-product attention for the CogVideoX model. It applies a rotary embedding on query and key vectors, but does not include spatial normalization. cCr`NrQzVCogVideoXAttnProcessor requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrdz"CogVideoXAttnProcessor2_0.__init__NrerrrrrrrgrcCs|d}tj||gdd}|j\}}} |dur+||||}|||jd|jd}||} ||} | |} | jd} | |j}| |d|j| dd} | |d|j| dd} | |d|j| dd} |j duru| | } |j dur| | } |durddl m}|| dddd|df|| dddd|df<|js|| dddd|df|| dddd|df<tj| | | |ddd}| dd|d|j|}|jd |}|jd|}|j||d|gdd\}}||fS) Nr<rrLrrqrFrr)rvrrrrrrrgrhrirrbrcrrqrXrurQrrnr)ryrerrrrgtext_seq_lengthrr}rrrr rUrrqr}r}r~rsB            66  z"CogVideoXAttnProcessor2_0.__call__r) rerrrrrrrrgrrrrr}r}r}r~r  rc@r)FusedCogVideoXAttnProcessor2_0rcCr`rrarcr}r}r~rS0 rdz'FusedCogVideoXAttnProcessor2_0.__init__NrerrrrrrrgrcCs|d}tj||gdd}|dur|jn|j\}}} |dur2||||}|||jd|jd}||} | jdd} tj| | dd\} } }| jd}||j}| |d|j| dd} | |d|j| dd} ||d|j| dd}|j dur| | } |j dur| | } |durddl m }|| dddd|df|| dddd|df<|js|| dddd|df|| dddd|df<tj| | ||ddd }| dd|d|j|}|jd |}|jd|}|j||d|gdd\}}||fS) Nr<rrLrrrrFrr)rvrrrrrrrrrrbrcrrqrXrurQrrn)ryrerrrrgrrr}rrrrrr rUrrqr}r}r~r4 sD         66  z'FusedCogVideoXAttnProcessor2_0.__call__r)rrr}r}r}r~r* rrc@r)rrNrrcCrr(rrr}r}r~rS} rz%XFormersAttnAddedKVProcessor.__init__rerrrrrrrcCs~|}||jd|jdddd}|j\}}}||||}|dur'|}n|jr/||}||dddd}||} || } | |} | |} || } || } |j s| |} | |} || } || } tj| | gdd} tj| | gdd} n| } | } tjj| | | ||j|jd}|| j}||}|jd|}|jd|}|dd|j}||}|S)Nrr<rLrrrr)rrrrrerrOrgrrjrkr1rhrirrrrrrr\rrrrnr)ryrerrrrrr}rrrrrr r}r}r~r sD"              z%XFormersAttnAddedKVProcessor.__call__r(rr)rrr}r}r}r~rq rrc@.eZdZdZddddZ   ddddZdS)rrNrrcCrr(rrr}r}r~rS rzXFormersAttnProcessor.__init__rerrrrrrrrcOst|dks|dddurd}tdd||} |jdur#|||}|j} | dkr>|j\} } } }|| | | |dd}|durE|jn|j\} }}|||| }|durc|j\}}}| d|d}|j duru| |dddd}| |}|dur|}n|j r| |}||}||}||}||}||}tjj|||||j|jd }||j}||}|jd|}|jd|}| dkr|dd | | | |}|jr|| }||j}|S) Nrr\rrrr<rrLrr)rrrrarrrrrexpandrOrgrerrhrirrrrrrr\rrrrnrr4r3)ryrerrrrrr^rrrrrrr key_tokensr query_tokensrrr r}r}r~r sR             zXFormersAttnProcessor.__call__r(rrrrr}r}r}r~r s rc@*eZdZdZddZ   ddddZdS)ra Processor for implementing flash attention using torch_npu. Torch_npu supports only fp16 and bf16 data types. If fp32 is used, F.scaled_dot_product_attention will be used for computation, but the acceleration effect on NPU is not significant. cCststddS)NzTAttnProcessorNPU requires torch_npu extensions and is supported only on npu devices.)r rbrcr}r}r~rS szAttnProcessorNPU.__init__NrerrrrrrrrcOst|dks|dddurd}tdd||} |jdur#|||}|j} | dkr>|j\} } } }|| | | |dd}|durE|jn|j\} }}|dur|||| }|| |j d|jd}| dd|jdd}|j t j kr{t | }n| }|jdur||dddd}||}|dur|}n|jr||}||}||}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|j t jt jfvrtj||||j d d|d t|jdd d d d dd d}n tj||||dd d}|dd| d|j |}||j }|jd|}|jd|}| dkrF|dd| | | |}|j rN|| }||j!}|S)Nrr\rrrr<rrLBNSDriF) input_layoutpse atten_maskr\ pre_tockens next_tockens keep_probsync inner_preciserrr)"rrrrarrrrrrrepeatrrr# logical_notrOrgrerrhrifloat16bfloat16 torch_npunpu_fusion_attentionmathsqrtrurQrrrnr4r3ryrerrrrrr^rrrrrrrr}rrrr rUrr}r}r~r" s                zAttnProcessorNPU.__call__rrrr}r}r}r~r src@r)rvs Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). cCr`rrarcr}r}r~rS rdzAttnProcessor2_0.__init__NrerrrrrrrrcOsHt|dks|dddurd}tdd||} |jdur#|||}|j} | dkr>|j\} } } }|| | | |dd}|durE|jn|j\} }}|durb|||| }|| |j d|jd}|j durt| |dddd}| |}|dur|}n|j r| |}||}||}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|jdur||}|jdur||}tj||||d d d }|dd| d|j |}||j}|jd|}|jd|}| dkr|dd | | | |}|jr|| }||j}|S) Nrr\rrrr<rrLrFrr)rrrrarrrrrrrOrgrerrhrirbrcrurQrrrrnr4r3rr}r}r~r s\                   zAttnProcessor2_0.__call__rrrr}r}r}r~rv srvc@r)rz Processor for implementing scaled dot-product attention with pallas flash attention kernel if using `torch_xla`. NrrcCsFttds tdtddrtdtrtddrtd||_dS)NrQzXXLAFlashAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rr6XLA flash attention requires torch_xla version >= 2.3.rDSPMD support for XLA flash attention needs torch_xla version >= 2.4.)rtrurbr rr)ryrr}r}r~rS s   z!XLAFlashAttnProcessor2_0.__init__rerrrrrrrrcOs|}|jdur |||}|j} | dkr(|j\} } } } || | | | dd}|dur/|jn|j\} }}|durL|||| }|| |jd|jd}|jdur^||dddd}||}|durj|}n|j rr| |}| |}| |}|jd}||j}|| d|j|dd}|| d|j|dd}|| d|j|dd}|j dur| |}|jdur||}tdd|||fDr|dur|| dd|jd}||dktd|dktd }||}|t|jd }tr|jnd}t|||d |d }ntd tj||||d d d}|dd| d|j|}||j}|jd|}|jd|}| dkrU|dd| | | | }|jr]||}||j }|S)Nrr<rrLcss|] }|jddkVqdS)riN)r)rrr}r}r~ 0 sz4XLAFlashAttnProcessor2_0.__call__..rrrrF)causalrz[Unable to use the flash attention pallas kernel API call due to QKV sequence length < 4096.rr)!rarrrrrrrOrgrerrhrirbrcallr! masked_fillrrrrrrrrurQrrrrnr4r3)ryrerrrrrr^rrrrrrr}rrrr rUrrr}r}r~r sr                 z!XLAFlashAttnProcessor2_0.__call__r()rrrrrr}r}r}r~r s rc@r)MochiVaeAttnProcessor2_0z0 Attention processor used in Mochi VAE. cCr`rrarcr}r}r~rS_ rdz!MochiVaeAttnProcessor2_0.__init__NrerrrrrrrcCs|}|jddk}|dur|jn|j\}}} |dur-||||}|||jd|jd}|rP||}|jd|}|jd|}|jrI||}||j}|S||} |dur[|}| |} ||} | jd} | |j}| |d|j| dd} | |d|j| dd} | |d|j| dd} |j dur| | } |j dur| | } t j| | | |d|jd}| dd|d|j|}|| j}|jd|}|jd|}|jr||}||j}|S)Nr<rLrrrr)rrrrrirnr4r3rgrhrrbrcrurQr;rrr)ryrerrrris_single_framerr}rrrr rUrr}r}r~rc sP             z!MochiVaeAttnProcessor2_0.__call__r)rrr}r}r}r~rZ s rc@s4eZdZdZddZdd d Z ddddZd S)StableAudioAttnProcessor2_0z Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the Stable Audio model. It applies rotary embedding on query and key vector, and allows MHA, GQA or MQA. cCr`)NrQz[StableAudioAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rS rz$StableAudioAttnProcessor2_0.__init__rjr freqs_cistuple[torch.Tensor]rc Csbddlm}|djd}|dd|f|d|df}}|||ddd}tj||fdd }|S) Nr<rrrL.Truse_realuse_real_unbind_dimr)rrqrrr) ryrjr"rqrot_dim x_to_rotate x_unrotated x_rotatedoutr}r}r~apply_partial_rotary_emb s "z4StableAudioAttnProcessor2_0.apply_partial_rotary_embNrerrrrr rotary_embcCs ddlm}|}|j}|dkr#|j\} } } } || | | | dd}|dur*|jn|j\} } }|durG||| | }|| |jd|jd}||}|durS|}n|j r[| |}| |}| |}|jd|j}|jd|}|| d|j|dd}|| d||dd}|| d||dd}||jkr|j|}t j||d|jd|d}t j||d|jd|d}|jdur||}|jdur||}|durC|j}|j}|t j}|t j}|djd}|dd|f|d|df}}|||d d d }t j||fdd }|js9|dd|f|d|df}}|||d d d }t j||fdd }||}||}tj||||d dd}|dd| d|j|}||j}|jd|}|jd|}|dkr|dd | | | | }|jr||}||j}|S)Nr<rrrrLrr.Trr$rrFr)rrqrrrrrrrgrerrhrirrrbrcrrrTrrXrurQrrnr4r3)ryrerrrr-rqrrrrrrr}rrrr rrheads_per_kv_head query_dtype key_dtyper'query_to_rotatequery_unrotated query_rotated key_to_rotate key_unrotated key_rotatedr}r}r~r sz              ""      z$StableAudioAttnProcessor2_0.__call__)rjrr"r#rrr) rerrrrrrrr-rrr)rr*r+r,rSr,rr}r}r}r~r! s r!c@r)HunyuanAttnProcessor2_0z Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the HunyuanDiT model. It applies a s normalization layer and rotary embedding on query and key vector. cCr`rrarcr}r}r~rS: rdz HunyuanAttnProcessor2_0.__init__NrerrrrrrrrgrcCsJddlm}|}|jdur|||}|j} | dkr.|j\} } } } || | | | dd}|dur5|jn|j\} }}|durR|||| }|| |jd|jd}|j durd| |dddd}| |}|durp|}n|j rx| |}| |}||}|jd}||j}|| d|j|dd}|| d|j|dd}|| d|j|dd}|jdur||}|jdur||}|dur|||}|js|||}tj||||ddd}|dd| d|j|}||j}|jd |}|jd|}| dkr|dd | | | | }|jr||}||j}|S Nr<rrrrLrFrrr)rrqrarrrrrrrOrgrerrhrirbrcrXrurQrrrrnr4r3)ryrerrrrrgrqrrrrrrr}rrrr rUrr}r}r~r> s`                    z HunyuanAttnProcessor2_0.__call__rrrr}r}r}r~r74 sr7c@r)FusedHunyuanAttnProcessor2_0a Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0) with fused projection layers. This is used in the HunyuanDiT model. It applies a s normalization layer and rotary embedding on query and key vector. cCr`)NrQz\FusedHunyuanAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rS rz%FusedHunyuanAttnProcessor2_0.__init__NrerrrrrrrrgrcCsddlm}|}|jdur|||}|j} | dkr.|j\} } } } || | | | dd}|dur5|jn|j\} }}|durR|||| }|| |jd|jd}|j durd| |dddd}|dur| |}|jdd}t j ||dd\}}}n#|j r||}||}||}|jdd}t j ||dd\}}|jd}||j}|| d|j|dd}|| d|j|dd}|| d|j|dd}|jdur||}|jdur||}|dur|||}|js|||}tj||||dd d }|dd| d|j|}||j}|jd |}|jd|}| dkr7|dd | | | | }|jr?||}||j}|S) Nr<rrrrLrrrFrrr)rrqrarrrrrrrOrrrrerrgrrbrcrXrurQrrrrnr4r3)ryrerrrrrgrqrrrrrrr}rrrrrr kvrUrr}r}r~r sf                    z%FusedHunyuanAttnProcessor2_0.__call__rrrr}r}r}r~r9  r9c@r)PAGHunyuanAttnProcessor2_0_ Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the HunyuanDiT model. It applies a normalization layer and rotary embedding on query and key vector. This variant of the processor employs [Pertubed Attention Guidance](https://huggingface.co/papers/2403.17377). cCr`)NrQzZPAGHunyuanAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rS rz#PAGHunyuanAttnProcessor2_0.__init__NrerrrrrrrrgrcCsddlm}|}|jdur|||}|j} | dkr.|j\} } } } || | | | dd}|d\}}|dur<|jn|j\} }}|durY|||| }|| |j d|jd}|j durk| |dddd}| |}|durw|}n|j r| |}||}||}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|jdur||}|jdur||}|dur|||}|js|||}tj||||ddd}|dd| d|j |}||j}|jd |}|jd|}| dkr|dd | | | | }|j dur0| |dddd}||}||j}|jd |}|jd|}| dkrZ|dd | | | | }t||g}|jri||}||j}|Sr8)rrqrarrrrrrrrOrgrerrhrirbrcrXrurQrrrrnrrr4r3)ryrerrrrrgrqrrrrrrrrr}rrrr rUrr}r}r~r st                        z#PAGHunyuanAttnProcessor2_0.__call__rrrr}r}r}r~r< r;r<c@r)PAGCFGHunyuanAttnProcessor2_0r=cCr`)NrQz]PAGCFGHunyuanAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rS rz&PAGCFGHunyuanAttnProcessor2_0.__init__NrerrrrrrrrgrcCsddlm}|}|jdur|||}|j} | dkr.|j\} } } } || | | | dd}|d\}}}t ||g}|durD|jn|j\} }}|dura| ||| }|| |j d|jd}|j durs| |dddd}| |}|dur|}n|jr||}||}||}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|jdur||}|jdur||}|dur|||}|js|||}tj||||ddd }|dd| d|j |}||j}|jd |}|jd|}| dkr%|dd | | | | }|j dur8| |dddd}||}||j}|jd |}|jd|}| dkrb|dd | | | | }t ||g}|jrq||}||j}|S) Nr<rrrrrLrFrrr)rrqrarrrrrrrrrrOrgrerrhrirbrcrXrurQrrrrnr4r3)ryrerrrrrgrqrrrrrrrrrr}rrrr rUrr}r}r~r sv                        z&PAGCFGHunyuanAttnProcessor2_0.__call__rrrr}r}r}r~r>x r;r>c@s,eZdZdZddZ    ddddZdS)LuminaAttnProcessor2_0z Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the LuminaNextDiT model. It applies a s normalization layer and rotary embedding on query and key vector. cCr`rrarcr}r}r~rS rdzLuminaAttnProcessor2_0.__init__Nrerrrrrrquery_rotary_embkey_rotary_embbase_sequence_lengthrrcCs0ddlm}|j} | dkr!|j\} } } } || | | | dd}|j\} }}||}||}||}|jd}|jd}||j }|j }||}|j durV| |}|j dur`| |}|| d|j |}|| d||}|| d||}|dur|||dd}|dur|||dd}| || |}}|durd}n|durtt|||j}n|j}|j |}|dkr|dddd|ddd}|dddd|ddd}|| ddd}|d|j |d}|dd}|dd}|dd}tj|||||d }|dd |}|S) Nr<rrrrLF)r%r)rr\)rrqrrrrrgrhrirrrbrcrrrlogr\rrrir#rrurQ)ryrerrrr@rArBrqrrrrrr}rrrr rrUrrr softmax_scalen_repr}r}r~r sZ                   zLuminaAttnProcessor2_0.__call__r)rerrrrrrrr@rrArrBrrrrr}r}r}r~r? s r?c@r)FusedAttnProcessor2_0u Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). It uses fused projection layers. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is currently 🧪 experimental in nature and can change in future. cCr`)NrQz`FusedAttnProcessor2_0 requires at least PyTorch 2.0, to use it. Please upgrade PyTorch to > 2.0.rarcr}r}r~rS]rzFusedAttnProcessor2_0.__init__NrerrrrrrrrcOst|dks|dddurd}tdd||} |jdur#|||}|j} | dkr>|j\} } } }|| | | |dd}|durE|jn|j\} }}|durb|||| }|| |j d|jd}|j durt| |dddd}|dur| |}|jdd }t j ||dd \}}}n#|jr||}||}||}|jdd}t j ||dd \}}|jd}||j }|| d|j |dd}|| d|j |dd}|| d|j |dd}|jdur||}|jdur||}tj||||d d d }|dd| d|j |}||j}|jd|}|jd|}| dkr6|dd| | | |}|jr>|| }||j}|S)Nrr\rrrr<rrLrrrFrr)rrrrarrrrrrrOrrrrerrgrrbrcrurQrrrrnr4r3)ryrerrrrrr^rrrrrrrr}rrrrrr r:rUrr}r}r~rcsb                   zFusedAttnProcessor2_0.__call__rrrr}r}r}r~rFTs rFcsBeZdZdZ       ddfdd Z  dd ddZZS)!raz Processor for implementing memory efficient attention using xFormers for the Custom Diffusion method. Args: train_kv (`bool`, defaults to `True`): Whether to newly train the key and value matrices corresponding to the text features. train_q_out (`bool`, defaults to `True`): Whether to newly train query matrices corresponding to the latent image features. hidden_size (`int`, *optional*, defaults to `None`): The hidden size of the attention layer. cross_attention_dim (`int`, *optional*, defaults to `None`): The number of channels in the `encoder_hidden_states`. out_bias (`bool`, defaults to `True`): Whether to include the bias parameter in `train_q_out`. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. attention_op (`Callable`, *optional*, defaults to `None`): The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. TFNrrr#rrrrr/r r!rrcst||_||_||_||_||_|jr-tj|p||dd|_ tj|p'||dd|_ |jrVtj||dd|_ t g|_ |j tj|||d|j t|dSdSr)rRrSrrrrrrrfrrrrmrrorp)ryrrrrr/r rr{r}r~rSs  z-CustomDiffusionXFormersAttnProcessor.__init__rerrrrrrrc Cs|dur|jn|j\}}}||||}|jr#|||jjj}n |||jjj}|dur7d} |}n d} |jrA| |}|j rk| ||j jj} | ||j jj} | |jjj} | |jjj} n | |} ||} | rt| } | ddddddfd| ddddddf<| | d| | } | | d| | } ||}|| } || } tjj|| | ||j|jd}||j}||}|jr|jd|}|jd|}|S|jd|}|jd|}|S)NFTr<rrr)rrrrrrgrrrerrrrrhrirrrrrrrrrr\rrrn) ryrerrrrr}rrrrr rr}r}r~rsN    4  z-CustomDiffusionXFormersAttnProcessor.__call__)TFNNTrN)rr#rr#rrrrr/r#r r!rrr)rrr}r}r{r~rs rcr)ru Processor for implementing attention for the Custom Diffusion method using PyTorch 2.0’s memory-efficient scaled dot-product attention. Args: train_kv (`bool`, defaults to `True`): Whether to newly train the key and value matrices corresponding to the text features. train_q_out (`bool`, defaults to `True`): Whether to newly train query matrices corresponding to the latent image features. hidden_size (`int`, *optional*, defaults to `None`): The hidden size of the attention layer. cross_attention_dim (`int`, *optional*, defaults to `None`): The number of channels in the `encoder_hidden_states`. out_bias (`bool`, defaults to `True`): Whether to include the bias parameter in `train_q_out`. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. TNrrr#rrrrr/r r!crrrrr{r}r~rSBrz(CustomDiffusionAttnProcessor2_0.__init__rerrrrrrrcCs|j\}}}||||}|jr||}n||}|dur$d} |}n d} |jr.||}|jrX|| |jj j } | | |j j j } | |jj j } | |jj j } n | |} ||} | rt| } | ddddddfd| ddddddf<| | d| | } | | d| | } |jd} | |j}||d|j|dd}| |d|j|dd} | |d|j|dd} tj|| | |ddd}|dd|d|j|}| |j }|jr|jd|}|jd|}|S|jd|}|jd|}|S) NFTr<rrLrrr)rrrrrgrerrrrrrrrhrirrrrrrrurQrrrn)ryrerrrrr}rrrrr rrUrr}r}r~r\sP       4    z(CustomDiffusionAttnProcessor2_0.__call__rrr)rrr}r}r{r~r.src@s*eZdZdZdddZ  ddddZdS)ra' Processor for implementing sliced attention. Args: slice_size (`int`, *optional*): The number of steps to compute attention. Uses as many slices as `attention_head_dim // slice_size`, and `attention_head_dim` must be a multiple of the `slice_size`. rrcCrr(rryrr}r}r~rSrzSlicedAttnProcessor.__init__NrerrrrrrrcCs |}|j}|dkr|j\}}} } |||| | dd}|dur$|jn|j\}} } ||| |}|jdurC||dddd}||} | jd}|| } |durY|}n|jra| |}| |}| |}||}||}| j\}}} t j ||||jf| j| jd}t|d|jdD]=}||j}|d|j}| ||}|||}|dur|||nd}||||}t ||||}||||<q||}|jd|}|jd|}|dkr|dd||| | }|jr||}||j}|S)Nrr<rrLrrr)rrrrrrOrgrrerrhrirrrrrrwrr rrrnrr4r3)ryrerrrrrrrrrr}rrrNrr batch_size_attentionri start_idxend_idx query_slice key_sliceattn_mask_slice attn_slicer}r}r~rsV               zSlicedAttnProcessor.__call__)rrr)rrr}r}r}r~rs  rc@r)rah Processor for implementing sliced attention with extra learnable key and value matrices for the text encoder. Args: slice_size (`int`, *optional*): The number of steps to compute attention. Uses as many slices as `attention_head_dim // slice_size`, and `attention_head_dim` must be a multiple of the `slice_size`. cCrr(rGrHr}r}r~rSrz#SlicedAttnAddedKVProcessor.__init__Nre 'Attention'rrrrrrrcCs4|}|jdur |||}||jd|jdddd}|j\}}} ||||}|dur2|}n|jr:||}||dddd}||} | jd} | | } | |} | |} | | } | | } |j s| |}||}| |}| |}tj| |gdd}tj| |gdd}n| }| }| j\}}} tj||| |jf| j| jd}t|d|jdD]=}||j}|d|j}| ||}|||}|dur|||nd}||||}t||||}||||<q||}|jd|}|jd|}|dd|j}||}|S)Nrr<rLrrrr)rarrrrrerrOrgrrjrkr1rhrirrrrrrrwrr rrrnr)ryrerrrrrrr}rrrNrrrr rIrrJrKrLrMrNrOrPr}r}r~rs\  "                  z#SlicedAttnAddedKVProcessor.__call__r) rerQrrrrrrrrrrrr}r}r}r~rs rcs,eZdZdZd fdd Zdd d ZZS)r`aq Spatially conditioned normalization as defined in https://huggingface.co/papers/2209.09002. Args: f_channels (`int`): The number of channels for input to group normalization layer, and output of the spatial norm layer. zq_channels (`int`): The number of channels for the quantized vector as described in the paper. rDrrEcsNttj|dddd|_tj||dddd|_tj||dddd|_dS)Nrư>Tr@r<r)r2strider)rRrSrr_ norm_layerr4conv_yconv_b)ryrDrEr{r}r~rS^s zSpatialNorm.__init__frzqrcCsD|jdd}tj||dd}||}|||||}|S)Nrnearest)rvrQ)rru interpolaterTrUrV)ryrWrXf_sizenorm_fnew_fr}r}r~rhs  zSpatialNorm.forward)rDrrEr)rWrrXrrr)rr*r+r,rSrr.r}r}r{r~r`Ss  r`c8eZdZdZdfdd Z     ddddZZS)ra Attention processor for Multiple IP-Adapters. Args: hidden_size (`int`): The hidden size of the attention layer. cross_attention_dim (`int`): The number of channels in the `encoder_hidden_states`. num_tokens (`int`, `tuple[int]` or `list[int]`, defaults to `(4,)`): The context length of the image features. scale (`float` or list[`float`], defaults to 1.0): the weight scale of image prompt. Nrrcst|_|_t|ttfs|g}||_t|ts$|gt|}t|t|kr0t d||_ t fddt t|D|_t fddt t|D|_dS)NJ`scale` should be a list of integers with the same length as `num_tokens`.cg|] }tjddqSFrPrrfrrrrr}r~rz3IPAdapterAttnProcessor.__init__..crarbrcrdrer}r~rrf)rRrSrrrtuplelistrrr^r\rrmrwrto_v_ipryrrrr\r{rer~rSs"   zIPAdapterAttnProcessor.__init__rerrrrrrrr\r!rc% Cs|}|dur@t|tr|\}} n0d} tdd| dd|jd|jd} |ddd| ddf|dd| dddfg}} |jdurK|||}|j} | dkrf|j\} }}}|| |||dd }|durm|jn|j\} }}| ||| }|j dur| |dd dd }| |}|dur|}n|j r| |}||}||}||}||}||}||||}t||}||}|durot|tst|d}t|t|jkrt| ksntd t|d t|jd t| d tt||j| D]`\}\}}}|durq t|tjr(|jdkr,td|jd|jdkrJtd|jdd|jdd|t|trlt||jdksltd|jddt|d|q ndgt|j}t| |j|j|j|D]\}}}}}d}t|trtdd|Drd}n|dkrd}|s]|dur3t|ts|g|jd}|jd}t |D]l}||dd|ddddf} ||dd|ddddf}!|| } ||!}!||| d}"t|"|!}#||#}#t!"|dd|ddddf| |#jd|#jd }$|$j#|j$|j%d}$||||#|$}qĐq||} ||}!|| } ||!}!||| d}"t|"|!}||}|||}q|j&d|}|j&d|}| dkr~|dd'| |||}|j(r||}||j)}|S)NYou have passed a tensor as `encoder_hidden_states`. This is deprecated and will be removed in a future release. Please make sure to update your script to pass `encoder_hidden_states` as a tuple to suppress this warning.!encoder_hidden_states not a tuplerF standard_warnr<rrr"Length of ip_adapter_masks array ()) must match length of self.scale array (") and number of ip_hidden_states ()Each element of the ip_adapter_masks array should be a tensor with shape [1, num_images_for_ip_adapter, height, width]. Please use `IPAdapterMaskProcessor` to preprocess your maskNumber of masks (&) does not match number of ip images ( ) at index #) does not match number of scales (cs|]}|dkVqdSrNr}rsr}r}r~rz2IPAdapterAttnProcessor.__call__..TrrLr)*rrgrrrrarrrrrOrgrerrhrirr rrrrhrrr\r^ enumeratezipTensorrrirrwr downsamplerrrrnrr4r3)%ryrerrrrr\rrrrend_posrrrrrr}rrrr r indexrip_statecurrent_ip_hidden_statesrriskipcurrent_num_imagesrJip_keyip_valueip_attention_probs_current_ip_hidden_statesmask_downsampler}r}r~rs                 &                       zIPAdapterAttnProcessor.__call__Nr_rNNNrNrerrrrrrrrrr\r!rrrr}r}r{r~rpsrcr^)ra Attention processor for IP-Adapter for PyTorch 2.0. Args: hidden_size (`int`): The hidden size of the attention layer. cross_attention_dim (`int`): The number of channels in the `encoder_hidden_states`. num_tokens (`int`, `tuple[int]` or `list[int]`, defaults to `(4,)`): The context length of the image features. scale (`float` or `list[float]`, defaults to 1.0): the weight scale of image prompt. Nr_rcstttdst|jjd|_|_t |t t fs#|g}||_ t |t s2|gt |}t |t |kr>td||_tfddtt |D|_tfddtt |D|_dS)NrQz@ requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.r`crarbrcrdrer}r~r]rfz6IPAdapterAttnProcessor2_0.__init__..crarbrcrdrer}r~r`rf)rRrSrtrurbr|rrrrrgrhrrr^r\rrmrwrrirjr{rer~rSGs*     z"IPAdapterAttnProcessor2_0.__init__rerrrrrrrr\r!rc% Cs|}|dur@t|tr|\}} n0d} tdd| dd|jd|jd} |ddd| ddf|dd| dddfg}} |jdurK|||}|j} | dkrf|j\} }}}|| |||dd }|durm|jn|j\} }}|dur| ||| }|| |j d |jd }|j dur| |dd dd }| |}|dur|}n|j r||}||}||}|jd }||j }|| d |j |dd }|| d |j |dd }|| d |j |dd }tj||||d dd }|dd | d |j |}||j}|durt|tst|d}t|t|jkr/t| ksEntd t|dt|jdt| dtt||j| D]`\}\}}}|dur\qNt|tjri|jdkrmtd|jd|jdkrtd|jdd|jdd|t|trt||jdkstd|jddt|d|qNndgt|j}t| |j|j|j |D]\}}}}}d}t|trt!dd|Drd}n|dkrd}|s|durt|ts|g|jd}|jd}t"|D]} ||dd| ddddf}!||dd| ddddf}"|!| d |j |dd }!|"| d |j |dd }"tj||!|"dd dd }#|#dd | d |j |}#|#|j}#t#$|dd| ddddf| |#jd|#jd }$|$j|j|j%d}$||| |#|$}qq||}!||}"|!| d |j |dd }!|"| d |j |dd }"tj||!|"dd dd }|dd | d |j |}||j}|||}q|j&d|}|j&d|}| dkr|d d| |||}|j'r||}||j(}|S)NrkrlrFrmr<rrrrLrrrorprqrrrsrtrurvrwcsrxryr}rzr}r}r~rr|z5IPAdapterAttnProcessor2_0.__call__..Trr))rrgrrrrarrrrrrOrgrerrhrirurQrrrrhrrr\r^r}r~rrrrirrwrrrrnr4r3)%ryrerrrrr\rrrrrrrrrrr}rrrr rUrrrrrrrirrrJrrrrr}r}r~rcs                (                       z"IPAdapterAttnProcessor2_0.__call__rrrrr}r}r{r~r8s rcsBeZdZdZ    ddfdd Z     ddddZZS)raG Attention processor for IP-Adapter using xFormers. Args: hidden_size (`int`): The hidden size of the attention layer. cross_attention_dim (`int`): The number of channels in the `encoder_hidden_states`. num_tokens (`int`, `tuple[int]` or `list[int]`, defaults to `(4,)`): The context length of the image features. scale (`float` or `list[float]`, defaults to 1.0): the weight scale of image prompt. attention_op (`Callable`, *optional*, defaults to `None`): The base [operator](https://facebookresearch.github.io/xformers/components/ops.html#xformers.ops.AttentionOpBase) to use as the attention operator. It is recommended to set to `None`, and allow xFormers to choose the best operator. Nr_rrrcst|_|_||_t|ttfs|g}||_t|ts'|gt |}t |t |kr3t d||_ t fddtt |D|_t fddtt |D|_dS)Nr`c g|] }tjp ddqSrbrcrdrer}r~rM z;IPAdapterXFormersAttnProcessor.__init__..crrbrcrdrer}r~rPr)rRrSrrrrrgrhrrr^r\rrmrwrri)ryrrrr\rr{rer~rS4s$   z'IPAdapterXFormersAttnProcessor.__init__rerrrrrrrr\r!rc$ Cs|}|dur@t|tr|\}} n0d} tdd| dd|jd|jd} |ddd| ddf|dd| dddfg}} |jdurK|||}|j} | dkrf|j\} }}}|| |||dd }|durm|jn|j\} }}|dur| ||| }|j\}}}| d |d }|j dur| |dd dd }| |}|dur|}n|j r||}||}||}||}||}||}tjj|||||jd }||j}||}| r|durt|tst|d}t|t|jkrt| ks$ntd t|d t|jdt| dtt ||j| D]`\}\}}}|dur;q-t|t!j"rH|jdkrLtd|jd|jdkrjtd|jdd|jdd|t|trt||jdkstd|jddt|d|q-ndgt|j}t | |j|j#|j$|D]\}}}}}d}t|trt%dd|Drd}n|dkrd}|s|dura|t!j&}t|ts|g|jd}|jd}t'|D]t}||dd|ddddf} ||dd|ddddf}!|| } ||!}!tjj|| |!|jd}"|"|j}"||"}"t()|dd|ddddf| |"jd|"jd }#|#j|j|j*d}#||||"|#}qq||} ||}!|| } ||!}!tjj|| |!|jd}||j}||}|||}q|j+d|}|j+d|}| dkr|d d,| |||}|j-r||}||j.}|S)NrkrlrFrmr<rrrrL)rrrorprqrrrsrtrurvrwcsrxryr}rzr}r}r~rr|z:IPAdapterXFormersAttnProcessor.__call__..T)rrr)/rrgrrrrarrrrrrOrgrerrhrirrrrrrrrrrhrrr\r^r}r~rrrrirrrwrrrrnrr4r3)$ryrerrrrr\rrrrrrrrrrr}rrrrr rrrrrrirrrJrrrrr}r}r~rSs                 (                      z'IPAdapterXFormersAttnProcessor.__call__)Nr_rNrr)rerrrrrrrrrr\r!rrrr}r}r{r~r s#rcs<eZdZdZ  ddfd d Z ddddZZS)!SD3IPAdapterJointAttnProcessor2_0aZ Attention processor for IP-Adapter used typically in processing the SD3-like self-attention projections, with additional image-based information and timestep embeddings. Args: hidden_size (`int`): The number of hidden channels. ip_hidden_states_dim (`int`): The image feature dimension. head_dim (`int`): The number of head channels. timesteps_emb_dim (`int`, defaults to 1280): The number of input channels for timestep embedding. scale (`float`, defaults to 0.5): IP-Adapter scale. ?rrip_hidden_states_dimrtimesteps_emb_dimr\r!cstddlm}m}|||dddd|_tj||dd|_tj||dd|_ ||d|_ ||d|_ ||d|_ ||_ dS)Nr<) AdaLayerNormr?rrR) output_dimnorm_eps chunk_dimFrP)rRrSrTrr?norm_iprrfrrirbrc norm_ip_kr\)ryrrrrr\rr?r{r}r~rSs     z*SD3IPAdapterJointAttnProcessor2_0.__init__NrerrrrrrrrrcCsL|}|jd}||} ||} ||} | jd} | |j} | |d|j| dd} | |d|j| dd} | |d|j| dd} | }| }| }|jdurW|| } |jdura|| } |dur| |}| |}| |}||d|j| dd}||d|j| dd}||d|j| dd}|j dur| |}|j dur| |}tj| |gdd} tj| |gdd} tj| |gdd} tj| | | ddd }|dd|d|j| }|| j}|dur|ddd|jdf|dd|jddf}}|js||}|jdkr|dur|j||d }||}||}||d|j| dd}||d|j| dd}||} ||}||}tj||gdd} tj||gdd} tj| | | ddd }|dd|d|j| }|| j}|||j}|jd|}|jd|}|dur||fS|S) aP Perform the attention computation, integrating image features (if provided) and timestep embeddings. If `ip_hidden_states` is `None`, this is equivalent to using JointAttnProcessor2_0. Args: attn (`Attention`): Attention instance. hidden_states (`torch.FloatTensor`): Input `hidden_states`. encoder_hidden_states (`torch.FloatTensor`, *optional*): The encoder hidden states. attention_mask (`torch.FloatTensor`, *optional*): Attention mask. ip_hidden_states (`torch.FloatTensor`, *optional*): Image embeddings. temb (`torch.FloatTensor`, *optional*): Timestep embeddings. Returns: `torch.FloatTensor`: Output hidden states. rrLr<rNrrFrs)r)rrgrhrirrrrbrcrlrjrkrrrsrrrurQrrrrZrqr\rrrirrn)ryrerrrrrrrrrr rUr img_queryimg_key img_valuerrrnorm_ip_hidden_statesrrr}r}r~r/s                              z*SD3IPAdapterJointAttnProcessor2_0.__call__)rr) rrrrrrrrr\r!r)rerrrrrrrrrrrrrrr}r}r{r~rsrc@*eZdZdZddZ   ddddZdS)PAGIdentitySelfAttnProcessor2_0 Processor for implementing PAG using scaled dot-product attention (enabled by default if you're using PyTorch 2.0). PAG reference: https://huggingface.co/papers/2403.17377 cCr`)NrQz_PAGIdentitySelfAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrz(PAGIdentitySelfAttnProcessor2_0.__init__NrerrrrrrrrrcCsf|}|jdur |||}|j}|dkr(|j\}} } } ||| | | dd}|d\} } | j\}}}|durL||||}|||jd|jd}|jdur^|| dddd} | | }| | }| | }|jd}||j}||d|j|dd}||d|j|dd}||d|j|dd}t j ||||ddd} | dd|d|j|} | |j} |jd| } |jd| } |dkr| dd || | | } | j\}}}|jdur|| dddd} | | } | |j} |jd| } |jd| } |dkr| dd || | | } t| | g}|jr,||}||j}|S) Nrr<rrLrFrrr)rarrrrrrrrOrgrhrirurQrrrrnrrr4r3)ryrerrrrrrrrrrrrr}rrrr rUrr}r}r~rsZ                 z(PAGIdentitySelfAttnProcessor2_0.__call__r rerrrrrrrrrrrrr}r}r}r~r rc@r)"PAGCFGIdentitySelfAttnProcessor2_0rcCr`)NrQzbPAGCFGIdentitySelfAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.rarcr}r}r~rSrz+PAGCFGIdentitySelfAttnProcessor2_0.__init__NrerrrrrrrrrcCsz|}|jdur |||}|j}|dkr(|j\}} } } ||| | | dd}|d\} } }t| | g} | j\}}}|durT||||}|||j d|jd}|j durf| | dddd} | | }| | }| | }|jd}||j }||d|j |dd}||d|j |dd}||d|j |dd}tj||||ddd} | dd|d|j |} | |j} |jd | } |jd| } |dkr| dd || | | } |j\}}}|j dur| |dddd}| |}|}||j}|jd |}|jd|}|dkr'|dd || | | }t| |g}|jr6||}||j}|S) Nrr<rrrLrFrrr)rarrrrrrrrrrOrgrhrirurQrrrrnr4r3)ryrerrrrrrrrrrrrrr}rrrr rUrr}r}r~r"s^                 z+PAGCFGIdentitySelfAttnProcessor2_0.__call__rrrr}r}r}r~rrrc@seZdZdZd ddZd S) rMzD Processor for implementing multiscale quadratic attention. rer;rrrcCs|jdd\}}|||jkrd}nd}|}t|\}}}}|j} |dd}||} ||} ||} t j | | | gdd}|dd}|g} |j D] }| ||qOt j | dd}|ri|j t jd}||dd|j||}|jdd d\} } } || } || } |r|| | | }|j | d}n|| | | }t ||d||f}||dddd}|jd kr||dddd}n||}|jr||}|S) NrTFr<rLrrrRrrI)rrArhrvrmovedimrgrhrirrrIrorrTrrrKrWrZrnrCrLr4)ryrerrruse_linear_attentionrrroriginal_dtyperrr multi_scale_qkvblockr}r}r~rsF          z'SanaMultiscaleAttnProcessor2_0.__call__N)rer;rrrrrr}r}r}r~rM}srMc@eZdZdZddZdS)LoRAAttnProcessorz9 Processor for implementing attention with LoRA. cCdSr(r}rcr}r}r~rSzLoRAAttnProcessor.__init__Nrr*r+r,rSr}r}r}r~r rc@r)LoRAAttnProcessor2_0zj Processor for implementing attention with LoRA (enabled by default if you're using PyTorch 2.0). cCrr(r}rcr}r}r~rSrzLoRAAttnProcessor2_0.__init__Nrr}r}r}r~rrrc@r)LoRAXFormersAttnProcessorzH Processor for implementing attention with LoRA using xFormers. cCrr(r}rcr}r}r~rSrz"LoRAXFormersAttnProcessor.__init__Nrr}r}r}r~rrrc@r)LoRAAttnAddedKVProcessorzz Processor for implementing attention with LoRA with extra learnable key and value matrices for the text encoder. cCrr(r}rcr}r}r~rSrz!LoRAAttnAddedKVProcessor.__init__Nrr}r}r}r~rrrc@r)SanaLinearAttnProcessor2_0I Processor for implementing scaled dot-product linear attention. Nrerrrrrrrc Cs|j}|dur |}||}||}||}|jdur"||}|jdur,||}|ddd|jdf}|ddd|jdfdd}|ddd|jdf}t |}t |}| | | }}}t j |dddd}t ||} t | |}|ddddddf|ddddddfd }|dddd}||}|jd |}|jd|}|t jkr|d d }|S) Nr<rrLrrNrOrrPr?r )rrgrhrirbrcrrurrurelur!rrrSrirrnrclip) ryrerrrrrrr rVr}r}r~rs6       "    8   z#SanaLinearAttnProcessor2_0.__call__r)rrr}r}r}r~r rc@r) PAGCFGSanaLinearAttnProcessor2_0rNrerrrrrrrc Cs|j}|d\}}}t||g}||} ||} ||} | ddd|j df} | ddd|j dfdd} | ddd|j df} t | } t | } | | | } } } t j | dddd} t| | } t| | }|ddddddf|ddddddfd }|dddd}||}|jd |}|jd|}|||}|jd |}|jd|}t||g}|tjkr|d d }|S) Nrr<rrLrNrOrrPr?rrr)rrrrrgrhrirrurrurr!rrSrirrnrr) ryrerrrrrrrrrr rVr}r}r~rs6   "    8   z)PAGCFGSanaLinearAttnProcessor2_0.__call__r)rrr}r}r}r~rrrc@r)%PAGIdentitySanaLinearAttnProcessor2_0rNrerrrrrrrc Cs|j}|d\}}||}||} ||} |ddd|jdf}| ddd|jdfdd} | ddd|jdf} t |}t | } | | | }} } tj | dddd} t | | } t | |}|jt jt jfvr~| }|ddddddf|ddddddfd }|dddd}||}|jd |}|jd|}|||}|jd |}|jd|}t ||g}|t jkr|d d }|S) Nrr<rLrrNrOrrPr?rrr)rrrgrhrirrurrurr!rrrSrrrirrnrr) ryrerrrrrrrrr rVr}r}r~rMs8   "    8   z.PAGIdentitySanaLinearAttnProcessor2_0.__call__r)rrr}r}r}r~rHrrc@eZdZddZdS)FluxAttnProcessor2_0cO*d}tdd|ddlm}||i|S)Nzq`FluxAttnProcessor2_0` is deprecated and this will be removed in a future version. Please use `FluxAttnProcessor`rrr<FluxAttnProcessorrtransformers.transformer_fluxrclsrr^rrr}r}r~__new__  zFluxAttnProcessor2_0.__new__Nrr*r+rr}r}r}r~r rc@r)FluxSingleAttnProcessor2_0rcOr)Nz|`FluxSingleAttnProcessor` is deprecated and will be removed in a future version. Please use `FluxAttnProcessorSDPA` instead.rrr<rrrr}r}r~rrz"FluxSingleAttnProcessor2_0.__new__Nrr*r+r,rr}r}r}r~rrrc@r)FusedFluxAttnProcessor2_0cOr)Nzv`FusedFluxAttnProcessor2_0` is deprecated and this will be removed in a future version. Please use `FluxAttnProcessor`rrr<rrrr}r}r~rrz!FusedFluxAttnProcessor2_0.__new__Nrr}r}r}r~rrrc@r)"FluxIPAdapterJointAttnProcessor2_0cOr)Nz`FluxIPAdapterJointAttnProcessor2_0` is deprecated and this will be removed in a future version. Please use `FluxIPAdapterAttnProcessor`rrr<)FluxIPAdapterAttnProcessor)rrr)rrr^rrr}r}r~rrz*FluxIPAdapterJointAttnProcessor2_0.__new__Nrr}r}r}r~rrrc@r)FluxAttnProcessor2_0_NPUcOs0d}tdd|ddddlm}|}d|_|S) NzFluxAttnProcessor2_0_NPU is deprecated and will be removed in a future version. An alternative solution to use NPU Flash Attention will be provided in the future.rrFrmr<r _native_npurrr_attention_backendrrr^rrr6r}r}r~r z FluxAttnProcessor2_0_NPU.__new__Nrr}r}r}r~rrrc@r)FusedFluxAttnProcessor2_0_NPUcCs0d}tdd|ddddlm}|}d|_|S) NzFusedFluxAttnProcessor2_0_NPU is deprecated and will be removed in a future version. An alternative solution to use NPU Flash Attention will be provided in the future.rrFrmr<r _fused_npur)ryrrr6r}r}r~rrz%FusedFluxAttnProcessor2_0_NPU.__new__Nrr}r}r}r~rrrc@r)rrcOsd}tdd|ddtddrtdtrtdd rtd d d lm}t|d ks3|dddur;d}tdd|||i|}d|_|S)NzXLAFluxFlashAttnProcessor2_0 is deprecated and will be removed in diffusers 1.0.0. An alternative solution to using XLA Flash Attention will be provided in the future.rrFrmrrrrrr<rrrzpartition_spec was not used in the processor implementation when it was added. Passing it is a no-op and support for it will be removed. _native_xla) rr rbrrrrrrrr}r}r~rs   z$XLAFluxFlashAttnProcessor2_0.__new__Nrr}r}r}r~rrr)a __future__rrrtypingrrtorch.nn.functionalr functionalruimage_processorrutilsrrr utils.import_utilsr r r utils.torch_utilsr r get_loggerrrrr xformers.ops$torch_xla.experimental.custom_kernelrtorch_xla.runtimer XLA_AVAILABLErrr/r;r[r_rwrrrrrrrrrrrrrrrrrvrrr!r7r9r<r>r?rFrrrrr`rrrrrrrMrrrrrrrrrrrrrrADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorr}r}r}r~s         BBiHhCLV&OU^]aEGG^r^tUbg{|`ftrW\Iii,cg<    479           !"#$%&'()*+,-./012