o Gi@sddlmZmZddlZddlmZddlmmZddl m Z m Z ddl m Z mZmZddlmZddlmZmZmZmZmZmZdd lmZmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$eriddl%Z&ndZ&e 'e(Z)Gd d d Z*GdddZ+dej,dej-de.de.fddZ/eGdddej,Z0eGdddej,Z1eGdddej,Z2Gdddej,Z3eGdddej,Z4Gd d!d!ej,Z5eGd"d#d#ej,Z6Gd$d%d%ej,Z7dS)&)AnyCallableN) deprecatelogging)is_torch_npu_availableis_torch_xla_availableis_xformers_available)maybe_allow_in_graph)GEGLUGELUApproximateGELUFP32SiLULinearActivationSwiGLU) AttentionAttentionProcessorJointAttnProcessor2_0)SinusoidalPositionalEmbedding) AdaLayerNormAdaLayerNormContinuousAdaLayerNormZeroRMSNormSD35AdaLayerNormZeroXc@sPeZdZedeeeffddZdeeeefBfddZddZ d d Z d S) AttentionMixinreturncsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorscsHt|dr|||d<|D]\}}|d|||q|S)N get_processor .processor.)hasattrr named_children)rrrsub_namechildfn_recursive_add_processorsN/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/attention.pyr(2s zCAttentionMixin.attn_processors..fn_recursive_add_processors)strtorchnnModuledictrr$)selfrrrr)r'r*attn_processors(s & zAttentionMixin.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrcsbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorr!r")r# isinstancer/r3popr$)rrr2r%r&fn_recursive_attn_processorr)r*r7Us   zFAttentionMixin.set_attn_processor..fn_recursive_attn_processorN) lenr1keysr4r/ ValueErrorr+r,r-r.r$)r0r2countrrr)r6r*set_attn_processor@s   z!AttentionMixin.set_attn_processorcCsV|jD]\}}dt|jjvrtdq|D]}t|tr(|j r(| qdS)z Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. AddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.N) r1itemsr+ __class____name__r:modulesr4AttentionModuleMixin_supports_qkv_fusionfuse_projections)r0_attn_processorrr)r)r*fuse_qkv_projectionsbs z#AttentionMixin.fuse_qkv_projectionscCs*|D]}t|tr|jr|qdS)umDisables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)rAr4rBrCunfuse_projections)r0rr)r)r*unfuse_qkv_projectionsos z%AttentionMixin.unfuse_qkv_projectionsN) r@ __module__ __qualname__propertyr/r+rr1r<rGrIr)r)r)r*r's " rc @sveZdZdZgZdZdZdeddfddZd5de dd fd d Z d e fd dZ de ddfddZ  d6de dee dBdfdBddfddZ d7de dedBddfddZeddZeddZdeddfd d!Zd"ejdejfd#d$Zd8d"ejd&edejfd'd(Z d7d)ejd*ejd+ejdBdejfd,d-Z %d8d+ejd.ed/ed&edejf d0d1Zd2ejdejfd3d4ZdS)9rBNTFr2rcCsVt|dr&t|jtjjr&t|tjjs&td|jd||j d||_dS)z Set the attention processor to use. Args: processor (`AttnProcessor`): The attention processor to use. r2z-You are removing possibly trained weights of z with N) r#r4r2r,r-r.loggerinfo_modulesr5)r0r2r)r)r*r3s    z"AttentionModuleMixin.set_processorreturn_deprecated_lorarcCs|s|jSdS)a7 Get the attention processor in use. Args: return_deprecated_lora (`bool`, *optional*, defaults to `False`): Set to `True` to return the deprecated LoRA attention processor. Returns: "AttentionProcessor": The attention processor in use. N)r2)r0rPr)r)r*r s z"AttentionModuleMixin.get_processorbackendcCsZddlm}dd|jD}||vr!td|dd|||}||j_dS)Nr )AttentionBackendNamecSsh|]}|jqSr)value).0xr)r)r* sz=AttentionModuleMixin.set_attention_backend..z `backend=z ` must be one of the following: z, ) attention_dispatchrR __members__valuesr:joinlowerr2_attention_backend)r0rQrRavailable_backendsr)r)r*set_attention_backends   z*AttentionModuleMixin.set_attention_backenduse_npu_flash_attentioncC |r ts td|ddS)z Set whether to use NPU flash attention from `torch_npu` or not. Args: use_npu_flash_attention (`bool`): Whether to use NPU flash attention or not. ztorch_npu is not available _native_npuN)r ImportErrorr_)r0r`r)r)r*set_use_npu_flash_attentionsz0AttentionModuleMixin.set_use_npu_flash_attentionuse_xla_flash_attentionpartition_spec.cCra)a Set whether to use XLA flash attention from `torch_xla` or not. Args: use_xla_flash_attention (`bool`): Whether to use pallas flash attention kernel from `torch_xla` or not. partition_spec (`tuple[]`, *optional*): Specify the partition specification if using SPMD. Otherwise None. is_flux (`bool`, *optional*, defaults to `False`): Whether the model is a Flux model. ztorch_xla is not available _native_xlaN)rrcr_)r0rerfis_fluxr)r)r*set_use_xla_flash_attentionsz0AttentionModuleMixin.set_use_xla_flash_attention'use_memory_efficient_attention_xformers attention_opc Cs|rNts tdddtjstdz$tr7d}|dur'|\}}|j^}}tjdd|d}tj |||}Wnt yF}z|d}~ww| ddSdS) a Set whether to use memory efficient attention from `xformers` or not. Args: use_memory_efficient_attention_xformers (`bool`): Whether to use memory efficient attention from `xformers` or not. attention_op (`Callable`, *optional*): The attention operation to use. Defaults to `None` which uses the default attention operation from `xformers`. zeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformersxformers)rzvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU N)r r(cudadevicedtype) r ModuleNotFoundErrorr,rn is_availabler:SUPPORTED_DTYPESrandnxopsopsmemory_efficient_attention Exceptionr_) r0rjrkrqop_fwop_bwrEqer)r)r*+set_use_memory_efficient_attention_xformerss2   z@AttentionModuleMixin.set_use_memory_efficient_attention_xformerscCsT|jst|jjddSt|ddrdS|jjjj }|jjjj }t |drr|j rrt |jjj|jjjg}|jd}|jd}tj|||j||d|_|jj|t |d rq|jrqt |jjj|jjjg}|jj|nNt |jjj|jjj|jjjg}|jd}|jd}tj|||j||d|_|jj|t |d r|jrt |jjj|jjj|jjjg}|jj|t|d ddur%t|d ddur%t|d ddur%t |jjj|jjj|jjjg}|jd}|jd}tj|||j||d|_|jj||jr%t |jjj|jjj|jjjg}|jj|d |_dS)ze Fuse the query, key, and value projections into a single projection for efficiency. zK does not support fusing QKV projections, so `fuse_projections` will no-op.Nfused_projectionsFis_cross_attentionr r)biasrprquse_bias add_q_proj add_k_proj add_v_projT)rCrMdebugr?r@getattrto_qweightdatarprqr#rr,catto_kto_vshaper-Linearrto_kvcopy_rto_qkvrrradded_proj_bias to_added_qkvr)r0rprqconcatenated_weights in_features out_featuresconcatenated_biasr)r)r*rDsZ      "  "    z%AttentionModuleMixin.fuse_projectionscCs`|jsdSt|dds dSt|drt|dt|dr!t|dt|dr+t|dd|_dS)z\ Unfuse the query, key, and value projections back to separate projections. NrFrrr)rCrr#delattrr)r0r)r)r*rH:s        z'AttentionModuleMixin.unfuse_projections slice_sizecCsht|dr|dur||jkrtd|d|jdd}|dur%|d}|dur-|}||dS)z Set the slice size for attention computation. Args: slice_size (`int`): The slice size for attention computation. sliceable_head_dimNz slice_size z has to be smaller or equal to r"sliced)r#rr:_get_compatible_processordefault_processor_clsr3)r0rr2r)r)r*set_attention_sliceTs z(AttentionModuleMixin.set_attention_slicetensorcCsL|j}|j\}}}||||||}|dddd|||||}|S)a Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size // heads, seq_len, dim * heads]`. Args: tensor (`torch.Tensor`): The tensor to reshape. Returns: `torch.Tensor`: The reshaped tensor. rrr )headsrreshapepermute)r0r head_size batch_sizeseq_lendimr)r)r*batch_to_head_dimks  "z&AttentionModuleMixin.batch_to_head_dimrout_dimcCs~|j}|jdkr|j\}}}d}n|j\}}}}|||||||}|dddd}|dkr=|||||||}|S)a5 Reshape the tensor for multi-head attention processing. Args: tensor (`torch.Tensor`): The tensor to reshape. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the tensor. Returns: `torch.Tensor`: The reshaped tensor. rr rr)rndimrrr)r0rrrrrr extra_dimr)r)r*head_to_batch_dim{s  z&AttentionModuleMixin.head_to_batch_dimquerykeyattention_maskc Cs|j}|jr|}|}|dur*tj|jd|jd|jd|j|jd}d}n|}d}tj|||dd||j d}~|j rE|}|j dd}~| |}|S) aL Compute the attention scores. Args: query (`torch.Tensor`): The query tensor. key (`torch.Tensor`): The key tensor. attention_mask (`torch.Tensor`, *optional*): The attention mask to use. Returns: `torch.Tensor`: The attention probabilities/scores. Nrr rqrp)betaalphar) rqupcast_attentionfloatr,emptyrrpbaddbmm transposescaleupcast_softmaxsoftmaxto) r0rrrrq baddbmm_inputrattention_scoresattention_probsr)r)r*get_attention_scoress2    z)AttentionModuleMixin.get_attention_scores target_lengthrc Cs|j}|dur |S|jd}||krA|jjdkr7|jd|jd|f}tj||j|jd}tj||gdd}n tj |d|fd d }|d krW|jd||krU|j |dd}|S|d krg| d}|j |dd}|S) a Prepare the attention mask for the attention computation. Args: attention_mask (`torch.Tensor`): The attention mask to prepare. target_length (`int`): The target length of the attention mask. batch_size (`int`): The batch size for repeating the attention mask. out_dim (`int`, *optional*, defaults to `3`): Output dimension. Returns: `torch.Tensor`: The prepared attention mask. Nrmpsrr rrrrSr) rrrptyper,zerosrqrFpadrepeat_interleave unsqueeze) r0rrrrrcurrent_length padding_shapepaddingr)r)r*prepare_attention_masks$   z+AttentionModuleMixin.prepare_attention_maskencoder_hidden_statescCsf|jdus Jdt|jtjr||}|St|jtjr1|dd}||}|dd}|SJ)z Normalize the encoder hidden states. Args: encoder_hidden_states (`torch.Tensor`): Hidden states of the encoder. Returns: `torch.Tensor`: The normalized encoder hidden states. NzGself.norm_cross must be defined to call self.norm_encoder_hidden_statesr r) norm_crossr4r- LayerNorm GroupNormr)r0rr)r)r*norm_encoder_hidden_statess      z/AttentionModuleMixin.norm_encoder_hidden_states)F)NFN)r)r@rJrK_default_processor_cls_available_processorsrCrrr3boolr r+r_rdtuplerirr~r,no_gradrDrHintrTensorrrrrrr)r)r)r*rBysp   ' B  0 +rBff hidden_states chunk_dim chunk_sizecsf|j||dkrtd|j|d|d|j||}tjfdd|j||dD|d}|S)Nrz)`hidden_states` dimension to be chunked: z$ has to be divisible by chunk size: z[. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`.csg|]}|qSr)r))rU hid_slicerr)r* sz)_chunked_feed_forward..r)rr:r,rchunk)rrrr num_chunks ff_outputr)rr*_chunked_feed_forward srcsNeZdZdZdedededeffdd Zdejd ejd ejfd d ZZ S) GatedSelfAttentionDenseat A gated self-attention dense layer that combines visual features and object features. Parameters: query_dim (`int`): The number of channels in the query. context_dim (`int`): The number of channels in the context. n_heads (`int`): The number of heads to use for attention. d_head (`int`): The number of channels in each head. query_dim context_dimn_headsd_headcstt|||_t|||d|_t|dd|_t ||_ t ||_ | dt td| dt tdd|_dS)N)rrdim_headgeglu activation_fn alpha_attnr alpha_denseT)super__init__r-rlinearrattn FeedForwardrrnorm1norm2register_parameter Parameterr,renabled)r0rrrrr?r)r*r%s    z GatedSelfAttentionDense.__init__rVobjsrc Cs|js|S|jd}||}||j||tj||gddddd|ddf}||j | | |}|S)Nr r) rrrrtanhrrr,rrrr)r0rVrn_visualr)r)r*forward6s  BzGatedSelfAttentionDense.forward) r@rJrK__doc__rrr,rr __classcell__r)r)rr*rs $rc seZdZdZ   ddedededededBd ef fd d Zdd edBdefddZ dde j de j de j de ee fdBde e je jff ddZZS)JointTransformerBlocka, A Transformer block following the MMDiT architecture, introduced in Stable Diffusion 3. Reference: https://huggingface.co/papers/2403.03206 Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. context_pre_only (`bool`): Boolean to determine if we should add some blocks associated with the processing of `context` conditions. FNrnum_attention_headsattention_head_dimcontext_pre_onlyqk_normuse_dual_attentionc sHt||_||_|rdnd}|rt||_nt||_|dkr.t||ddddd|_n|dkr8t||_nt d|d t t d rIt }nt d t |d|||||d||dd |_|rnt |d|||d||dd |_nd|_tj|ddd|_t||dd|_|stj|ddd|_t||dd|_nd|_d|_d|_d|_dS)Nada_norm_continous ada_norm_zeroFư>T layer_norm)elementwise_affineepsr norm_typezUnknown context_norm_type: z>, currently only support `ada_norm_continous`, `ada_norm_zero`scaled_dot_product_attentionzYThe current PyTorch version does not support the `scaled_dot_product_attention` function.) rcross_attention_dimadded_kv_proj_dimrrrr rr2r r) rrrrrrr2r rrrgelu-approximate)rdim_outrr)rrr r rrrr norm1_contextr:r#rrrrattn2r-rrrr norm2_context ff_context _chunk_size _chunk_dim) r0rrrr r r context_norm_typer2rr)r*rRsp            zJointTransformerBlock.__init__rrcC||_||_dSrrrr0rrr)r)r*set_chunk_feed_forward z,JointTransformerBlock.set_chunk_feed_forwardrrtembjoint_attention_kwargsrcCs|pi}|jr|j||d\}}}}} } } n |j||d\}}}}} |jr,|||} n |j||d\} } }}}|jd|| d|\}}|d|}||}|jrh|jdd| i|}| d|}||}||}|d|dddf|dddf}|jdurt |j ||j |j}n| |}| d|}||}|jrd}||fS| d|}||}| |} | d|dddf|dddf} |jdurt |j | |j |j}n| | }||d|}||fS)N)emb)rrr rr))r rr rrrrrrrrrrr)r0rrr%r&norm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlpnorm_hidden_states2 gate_msa2r c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlp attn_outputcontext_attn_output attn_output2rcontext_ff_outputr)r)r*rsX  (   (  zJointTransformerBlock.forward)FNFrr)r@rJrKrrrr+rr#r, FloatTensorr/rrrrrr)r)rr*rCs>R rc-sNeZdZdZ                   d1d ed ed ed edBdededBdededededededededededBdedBdedBdedBdedBdedef,fd d! Zd2d#edBd efd$d%Z       d3d&e j d'e j dBd(e j dBd)e j dBd*e j dBd+e eefd,e j dBd-e ee j fdBd.e j fd/d0ZZS)4BasicTransformerBlocka A basic Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (: obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (: obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, *optional*): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, *optional*): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, *optional*): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, *optional*, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, *optional*, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` *optional*, defaults to False): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, *optional*, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*, defaults to `None`): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. rNrFTrh㈵>defaultrrrrrnum_embeds_ada_normattention_biasonly_cross_attentiondouble_self_attentionrnorm_elementwise_affinernorm_eps final_dropoutattention_typepositional_embeddingsnum_positional_embeddings-ada_norm_continous_conditioning_embedding_dim ada_norm_bias ff_inner_dimff_biasattention_out_biasc st||_||_||_||_||_||_||_| |_ | |_ ||_ ||_ | |_ |duo0| dk|_|duo9| dk|_| dk|_| dk|_| dk|_| dvr]|dur]td| d| d | |_||_|rm|durmtd |d kryt||d |_nd|_| dkrt|||_n#| dkrt|||_n| dkrt||| ||d |_n tj|| |d|_t|||||| r|nd| |d|_|dus| r| dkrt|||_ n| dkrt||| ||d |_ nt||| |_ t|| s|nd||||| |d|_!n| dkrt||| |_ nd|_ d|_!| dkrt||| ||d|_"n| dvr't||| |_"n| dkr/d|_"t#||||||d|_$|dksD|dkrLt%|||||_&| dkr_t't()d||d|_*d|_+d|_,dS)Nr ada_normada_norm_singlerada_norm_continuousrKr `norm_type` is set to w, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to r"\If `positional_embedding` type is defined, `num_positition_embeddings` must also be defined. sinusoidalmax_seq_lengthrms_normrrrrdropoutrrrout_biasrrrrrWrrrX)r rKrlayer_norm_i2vgenrWrrB inner_dimrgatedzgated-text-imageg?r)-rrrrrrWrrr=r?r@rDrEr>use_ada_layer_norm_zerouse_ada_layer_normuse_ada_layer_norm_singleuse_layer_normuse_ada_layer_norm_continuousr:rr<r pos_embedrrrrr-rrattn1rrnorm3rrrfuserrr,ruscale_shift_tablerr)r0rrrrWrrr<r=r>r?rr@rrArBrCrDrErFrGrHrIrJrr)r*rs                 zBasicTransformerBlock.__init__rrcCr rr!r"r)r)r*r#r$z,BasicTransformerBlock.set_chunk_feed_forwardrrrencoder_attention_masktimestepcross_attention_kwargs class_labelsadded_cond_kwargsrc Csp|dur|dddurtd|jd} |jdkr"|||} n\|jdkr7|j||||jd\} } } } }nG|jdvrB||} n<|jdkrP|||d } n.|jd krz|jd|| d d j d d d\}}} } } }||} | d ||} nt d|j dur| | } |dur| ni}| dd}|j| f|jr|nd|d|}|jdkr| d |}n |jd kr| |}||}|jdkr|d }|dur|||d}|jdur2|jdkr|||} n&|jdvr||} n|jd kr|} n|jdkr |||d } nt d|j dur"|jd kr"| | } |j| f||d|}||}|jdkrA|||d } n |jd ksL||} |jdkrf| d | dddf| dddf} |jd kry||} | d | | } |jdurt|j| |j|j}n|| }|jdkr|d |}n |jd kr||}||}|jdkr|d }|S)NrSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.rrKr ) hidden_dtype)rrZrMpooled_text_embrLr^rr rzIncorrect norm usedgligenrrrr)r rrZzIncorrect norm)getrMwarningrrrrqrhrrr:rdcopyr5rer>rrsqueezergrrrfrrrr)r0rrrrirjrkrlrmrr(r)r*r+r, shift_msa scale_msa gligen_kwargsr3rr)r)r*rs                              (        zBasicTransformerBlock.forward)rNrNFFFFTrr:Fr;NNNNNTTr7)NNNNNNN)r@rJrKrrr+rrrr#r,r LongTensorr/rrrr)r)rr*r9s'     )   r9c sHeZdZdZ  d dedededBdedBffdd Zd d ZZS) LuminaFeedForwarda' A feed-forward layer. Parameters: hidden_size (`int`): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. intermediate_size (`int`): The intermediate dimension of the feedforward layer. multiple_of (`int`, *optional*): Value to ensure hidden dimension is a multiple of this value. ffn_dim_multiplier (float, *optional*): Custom multiplier for hidden dimension. Defaults to None. Nrr\ multiple_offfn_dim_multipliercstt|durt||}|||d|}tj||dd|_tj||dd|_tj||dd|_t|_ dS)Nr Fr) rrrr-rlinear_1linear_2linear_3rsilu)r0rr\r}r~rr)r*rJs(   zLuminaFeedForward.__init__cCs ||||||Sr)rrrr)r0rVr)r)r*rhs zLuminaFeedForward.forward)r|N) r@rJrKrrrrrrr)r)rr*r{;sr{c sxeZdZdZ ddedededededBf fdd Zd edBfd d Z dd ejdedejdBdejfddZ Z S)TemporalBasicTransformerBlocka A basic Transformer block for video like data. Parameters: dim (`int`): The number of channels in the input and output. time_mix_inner_dim (`int`): The number of channels for temporal attention. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. Nrtime_mix_inner_dimrrrcst||k|_t||_t||dd|_t||_t |||dd|_ |dur;t||_ t ||||d|_ nd|_ d|_ t||_ t|dd|_d|_d|_dS)Nr)rr)rrrr)rrrrr)rris_resr-rnorm_inrff_inrrrerrrfrrr)r0rrrrrrr)r*rys:        z&TemporalBasicTransformerBlock.__init__rcKs||_d|_dS)Nr r!)r0rkwargsr)r)r*r#s z4TemporalBasicTransformerBlock.set_chunk_feed_forwardr num_framesrrc Csv|jd}|j\}}}||}|dddf||||}|dddd}|||||}|}||}|jdurEt|j||j|j}n||}|jrQ||}| |} |j | dd} | |}|j durv| |} |j | |d} | |}| |} |jdurt|j| |j|j} n|| } |jr| |}n| }|dddf||||}|dddd}|||||}|S)Nrrr r)r)rrrrrrrrrrrerrrfr) r0rrrr batch_frames seq_lengthchannelsresidualr(r3rr)r)r*rs>            z%TemporalBasicTransformerBlock.forwardr) r@rJrKrrrr#r,rrrr)r)rr*rls25 rcsTeZdZ    ddedededed ed edBd ed effd d ZddZZS)SkipFFTransformerBlockrNFTrrr kv_input_dimkv_input_dim_proj_use_biasrr=rJc svt||krt||||_nd|_t|d|_t||||||| d|_t|d|_ t||||||| d|_ dS)Nr)rrrrWrrrX)rrrrrWrrX) rrr-r kv_mapperrrrrerr) r0rrrrrrWrr=rJrr)r*rs0    zSkipFFTransformerBlock.__init__cCs|dur|ni}|jdur|t|}||}|j|fd|i|}||}||}|j|fd|i|}||}|S)Nr)rurrrrrerr)r0rrrkr(r3r)r)r*rs,   zSkipFFTransformerBlock.forward)rNFT)r@rJrKrrrrrr)r)rr*rs.  *rc/szeZdZdZ                  d8d ed ededededBdededBdedededededededededBdedBdedBdeded ed!ed"ef.fd#d$ Zd%ed&e e eeffd'd(Z d9d%ed"ed&e efd)d*Z d9d ed!ed"ed&dfd+d,Z d:d.edBd ed&dfd/d0Z    d;d1ejd2ejdBd3ejdBd4ejdBd5eeefd&ejf d6d7ZZS)<FreeNoiseTransformerBlocka A FreeNoise Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (`int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (`bool`, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, defaults to `False`): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, defaults to `False`): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, defaults to `False`): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` defaults to `False`): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. ff_inner_dim (`int`, *optional*): Hidden dimension of feed-forward MLP. ff_bias (`bool`, defaults to `True`): Whether or not to use bias in feed-forward MLP. attention_out_bias (`bool`, defaults to `True`): Whether or not to use bias in attention output project layer. context_length (`int`, defaults to `16`): The maximum number of frames that the FreeNoise block processes at once. context_stride (`int`, defaults to `4`): The number of frames to be skipped before starting to process a new batch of `context_length` frames. weighting_scheme (`str`, defaults to `"pyramid"`): The weighting scheme to use for weighting averaging of processed latent frames. As described in the Equation 9. of the [FreeNoise](https://huggingface.co/papers/2310.15169) paper, "pyramid" is the default setting used. rNrFTrr:rpyramidrrrrWrrr<r=r>r?rr@rrArBrDrErHrIrJcontext_lengthcontext_strideweighting_schemec st||_||_||_||_||_||_||_| |_ | |_ ||_ ||_ | |_ |||||duo7| dk|_|duo@| dk|_| dk|_| dk|_| dk|_| dvrd|durdtd| d| d | |_||_|rt|durttd |d krt||d |_nd|_tj|| |d |_t|||||| r|nd| |d|_|dus| rt||| |_t|| s|nd||||| |d|_t ||||||d|_!t||| |_"d|_#d|_$dS)Nr rKrLrrMrNrOrPr"rQrRrSrrVrYr[r)%rrrrrrWrrr=r?r@rDrEr>set_free_noise_propertiesr_r`rarbrcr:rr<rrdr-rrrrerrrrrfrr)r0rrrrWrrr<r=r>r?rr@rrArBrDrErHrIrJrrrrr)r*rns           z"FreeNoiseTransformerBlock.__init__rrcCsHg}td||jd|jD]}|}t|||j}|||fq|S)Nrr )rangerrminappend)r0r frame_indicesi window_start window_endr)r)r*_get_frame_indicess z,FreeNoiseTransformerBlock._get_frame_indicescCs|dkr dg|}|S|dkrH|ddkr-|d}ttd|d}||ddd}|S|dd}ttd|}||g|ddd}|S|dkr|ddkrl|d}d g|d|g}|tt|dd}|S|dd}d g|}|tt|dd}|Std |) Nflatg?rrrr rdelayed_reverse_sawtoothg{Gz?z'Unsupported value for weighting_scheme=)listrr:)r0rrweightsmidr)r)r*_get_frame_weightss0     z,FreeNoiseTransformerBlock._get_frame_weightscCs||_||_||_dSr)rrr)r0rrrr)r)r*rs z3FreeNoiseTransformerBlock.set_free_noise_propertiesrrcCr rr!r"r)r)r*r#r$z0FreeNoiseTransformerBlock.set_chunk_feed_forwardrrrrirkc OsN|dur|dddurtd|dur|ni}|j}|j} |d} || } ||j |j } t j | || d d d} | dd| k} | sm| |j kr[td| d|j | | dd}| | |j | ft jd| df|d }t |}t| D]\}\}}t |dd||f}|| 9}|dd||f}||}|jdur||}|j|f|jr|nd|d |}||}|jd kr|d}|jdur||}|jdur|jd kr||}|j|f||d |}||}|t| dkr>| s>|dd| df|dd| df|dd| df7<|dd| df|dd| f7<q|dd||f||7<|dd||f|7<qt jd dt|j |j dd|j |j ddDdd!| }|"|}|j#durt$|j%||j&|j#}n|%|}||}|jd kr|d}|S)Nrrnr rorrzExpected num_frames=z1 to be greater or equal than self.context_length=)rprrrrLcSs&g|]\}}t|dk|||qSr7)r,where)rUaccumulated_splitnum_times_splitr)r)r*rysz5FreeNoiseTransformerBlock.forward..r)'rsrMrtrurprqsizerrrrr,rrr:rr zeros_like enumerate ones_likerrdrer>rrvrrrr8rzipsplitrrfrrrr)r0rrrrirkargsrrprqrr frame_weightsis_last_frame_batch_completelast_frame_batch_lengthnum_times_accumulatedaccumulated_valuesr frame_start frame_endrhidden_states_chunkr(r3rr)r)r*rs               *.       z!FreeNoiseTransformerBlock.forward)rNrNFFFFTrr:FNNNTTrrr)rr7)NNNN)r@rJrKrrrr+rrrrrrrr#r,rr/rrrr)r)rr*r6s;     r!   rcsheZdZdZ       dded edBd ed ed ed edeffdd Zde j de j fddZ Z S)ra A feed-forward layer. Parameters: dim (`int`): The number of channels in the input. dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`. mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. final_dropout (`bool` *optional*, defaults to False): Apply a final dropout. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. NrrrFTrrmultrWrrBrc st|durt||}|dur|n|}|dkr"t|||d} |dkr/t||d|d} n0|dkr;t|||d} n$|dkrGt|||d} n|dkrSt|||d} n |d kr_t|||d d } t g|_ |j | |j t ||j tj |||d|r|j t |dSdS) Ngelurrr) approximaterrzgeglu-approximateswigluz linear-silur)r activation)rrrr r rrrr- ModuleListnetrDropoutr) r0rrrrWrrBr\ract_fnrr)r*rs.    zFeedForward.__init__rrcOsDt|dks|dddurd}tdd||jD]}||}q|S)NrrzThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)r8rsrr)r0rrrdeprecation_messagerr)r)r*rs    zFeedForward.forward)NrrrFNT) r@rJrKrrrr+rrr,rrrr)r)rr*rs2 (r)8typingrrr,torch.nnr-torch.nn.functional functionalrutilsrrutils.import_utilsrrr utils.torch_utilsr activationsr r rrrrattention_processorrrr embeddingsr normalizationrrrrrrlrv get_loggerr@rMrrBr.rrrrrr9r{rrrrr)r)r)r*sP      R),M1H]