o pi^@s~ddlmZmZmZmZmZddlZddlmm Z ddlmZddl m Z m Z ddlmZddlmZmZmZmZmZdd lmZmZdd lmZdd lmZmZmZmZe e!Z"d ej#d ej$de%de%fddZ&eGdddej#Z'eGdddej#Z(eGdddej#Z)Gdddej#Z*eGdddej#Z+Gdddej#Z,eGdddej#Z-Gd d!d!ej#Z.dS)")AnyDictListOptionalTupleN)nn) deprecatelogging)maybe_allow_in_graph)GEGLUGELUApproximateGELUFP32SiLUSwiGLU) AttentionJointAttnProcessor2_0)SinusoidalPositionalEmbedding) AdaLayerNormAdaLayerNormContinuousAdaLayerNormZeroRMSNormff hidden_states chunk_dim chunk_sizecsf|j||dkrtd|j|d|d|j||}tjfdd|j||dD|d}|S)Nrz)`hidden_states` dimension to be chunked: z$ has to be divisible by chunk size: z[. Make sure to set an appropriate `chunk_size` when calling `unet.enable_forward_chunking`.csg|]}|qSr).0 hid_slicerrX/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/attention.py (sz)_chunked_feed_forward..dim)shape ValueErrortorchcatchunk)rrrr num_chunks ff_outputrr r!_chunked_feed_forwardsr,csNeZdZdZdedededeffdd Zdejd ejd ejfd d ZZ S) GatedSelfAttentionDenseat A gated self-attention dense layer that combines visual features and object features. Parameters: query_dim (`int`): The number of channels in the query. context_dim (`int`): The number of channels in the context. n_heads (`int`): The number of heads to use for attention. d_head (`int`): The number of channels in each head. query_dim context_dimn_headsd_headcstt|||_t|||d|_t|dd|_t ||_ t ||_ | dt td| dt tdd|_dS)N)r.headsdim_headgeglu activation_fn alpha_attn alpha_denseT)super__init__rLinearlinearrattn FeedForwardr LayerNormnorm1norm2register_parameter Parameterr'tensorenabled)selfr.r/r0r1 __class__rr!r;:s    z GatedSelfAttentionDense.__init__xobjsreturnc Cs|js|S|jd}||}||j||tj||gddddd|ddf}||j | | |}|S)Nr r#) rFr%r=r7tanhr>rAr'r(r9rrB)rGrJrKn_visualrrr!forwardKs  BzGatedSelfAttentionDense.forward) __name__ __module__ __qualname____doc__intr;r'TensorrO __classcell__rrrHr!r-.s $r-csVeZdZdZdfdd Zddeedefdd Zd ej d ej d ej fd dZ Z S)JointTransformerBlocka$ A Transformer block following the MMDiT architecture, introduced in Stable Diffusion 3. Reference: https://arxiv.org/abs/2403.03206 Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. context_pre_only (`bool`): Boolean to determine if we should add some blocks associated with the processing of `context` conditions. Fc st||_|r dnd}t||_|dkr#t||ddddd|_n|dkr-t||_ntd|d tt d r>t }ntd t |d|||||d|d |_ t j|ddd |_t||dd|_|sut j|ddd |_t||dd|_nd|_d|_d|_d|_dS)Nada_norm_continous ada_norm_zeroFư>T layer_norm)elementwise_affineepsbias norm_typezUnknown context_norm_type: z>, currently only support `ada_norm_continous`, `ada_norm_zero`scaled_dot_product_attentionzYThe current PyTorch version does not support the `scaled_dot_product_attention` function.) r.cross_attention_dimadded_kv_proj_dimr3r2out_dimcontext_pre_onlyr^ processorr\r]gelu-approximate)r$dim_outr6r)r:r;rdrrAr norm1_contextr&hasattrFrrr>rr@rBr?r norm2_context ff_context _chunk_size _chunk_dim)rGr$num_attention_headsattention_head_dimrdcontext_norm_typererHrr!r;gsL          zJointTransformerBlock.__init__rrr$cC||_||_dSNrnrorGrr$rrr!set_chunk_feed_forward z,JointTransformerBlock.set_chunk_feed_forwardrencoder_hidden_statestembcCs||j||d\}}}}}|jr|||} n |j||d\} } } } } |j|| d\}}|d|}||}||}|d|dddf|dddf}|jdur_t|j||j |j}n||}|d|}||}|jrxd}||fS| d|}||}| |} | d| dddf| dddf} |jdurt|j | |j |j}n| | }|| d|}||fS)N)emb)rryr ) rArdrir> unsqueezerBrnr,rrorlrm)rGrryrznorm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlpnorm_encoder_hidden_states c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlp attn_outputcontext_attn_outputr+context_ff_outputrrr!rOs@  (   (  zJointTransformerBlock.forward)Fr) rPrQrRrSr;rrTrwr' FloatTensorrOrVrrrHr!rWXs 3rWc-sNeZdZdZ                   d1d ed ed ed eededeededededededededededeedeedeedeedeededef,fd d! Z d2d#eed efd$d%Z       d3d&e j d'ee j d(ee j d)ee j d*ee j d+eeefd,ee j d-eeee j fd.e j fd/d0ZZS)4BasicTransformerBlocka A basic Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (: obj: `int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (: obj: `bool`, *optional*, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, *optional*): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, *optional*): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, *optional*): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, *optional*, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, *optional*, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` *optional*, defaults to False): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, *optional*, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*, defaults to `None`): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. r8Nr4FTr[h㈵>defaultr$rprqrar6num_embeds_ada_normattention_biasonly_cross_attentiondouble_self_attentionupcast_attentionnorm_elementwise_affiner_norm_eps final_dropoutattention_typepositional_embeddingsnum_positional_embeddings-ada_norm_continous_conditioning_embedding_dim ada_norm_bias ff_inner_dimff_biasattention_out_biasc st||_||_||_||_||_||_||_| |_ | |_ ||_ ||_ | |_ |duo0| dk|_|duo9| dk|_| dk|_| dk|_| dk|_| dvr]|dur]td| d| d | |_||_|rm|durmtd |d kryt||d |_nd|_| dkrt|||_n#| dkrt|||_n| dkrt||| ||d |_n tj|| |d|_t|||||| r|nd| |d|_|dus| r| dkrt|||_ n| dkrt||| ||d |_ nt||| |_ t|| s|nd||||| |d|_!n| dkrt||| |_ nd|_ d|_!| dkrt||| ||d|_"n| dvr't||| |_"n| dkr/d|_"t#||||||d|_$|dksD|dkrLt%|||||_&| dkr_t't()d||d|_*d|_+d|_,dS)NrYada_normada_norm_singler[ada_norm_continuousrrY`norm_type` is set to w, but `num_embeds_ada_norm` is not defined. Please make sure to define `num_embeds_ada_norm` if setting `norm_type` to .\If `positional_embedding` type is defined, `num_positition_embeddings` must also be defined. sinusoidalmax_seq_lengthrms_normrfr.r2r3dropoutr^rarout_biasr.rar2r3rr^rr)rYrr[layer_norm_i2vgenrr6r inner_dimr^gatedzgated-text-imageg?r)-r:r;r$rprqrrar6rrrrrruse_ada_layer_norm_zerouse_ada_layer_normuse_ada_layer_norm_singleuse_layer_normuse_ada_layer_norm_continuousr&r_rr pos_embedrrArrrr@rattn1rBattn2norm3r?rr-fuserrDr'randnscale_shift_tablernro)rGr$rprqrrar6rrrrrrr_rrrrrrrrrrrHrr!r;s                 zBasicTransformerBlock.__init__rrcCrsrtrurvrrr!rwrxz,BasicTransformerBlock.set_chunk_feed_forwardrattention_maskryencoder_attention_masktimestepcross_attention_kwargs class_labelsadded_cond_kwargsrLc Csp|dur|dddurtd|jd} |jdkr"|||} n\|jdkr7|j||||jd\} } } } }nG|jdvrB||} n<|jdkrP|||d } n.|jd krz|jd|| d d j d d d\}}} } } }||} | d ||} nt d|j dur| | } |dur| ni}| dd}|j| f|jr|nd|d|}|jdkr| d |}n |jd kr| |}||}|jdkr|d }|dur|||d}|jdur2|jdkr|||} n&|jdvr||} n|jd kr|} n|jdkr |||d } nt d|j dur"|jd kr"| | } |j| f||d|}||}|jdkrA|||d } n |jd ksL||} |jdkrf| d | dddf| dddf} |jd kry||} | d | | } |jdurt|j| |j|j}n|| }|jdkr|d |}n |jd kr||}||}|jdkr|d }|S)NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.rrrY) hidden_dtype)r[rrpooled_text_embrrr r#zIncorrect norm usedgligenryrrK)rYr[rzIncorrect norm)getloggerwarningr%r_rAdtyperreshaper)r&rcopypoprrr|ndimsqueezerrrBrrnr,rro)rGrrryrrrrr batch_sizer}r~rrr shift_msa scale_msa gligen_kwargsrr+rrr!rOs                              (        zBasicTransformerBlock.forward)r8Nr4NFFFFTr[rFrNNNNNTTr)NNNNNNN)rPrQrRrSrTrstrboolfloatr;rwr'rU LongTensorrrrOrVrrrHr!rs'     )   rc sHeZdZdZ  d dededeedeeffdd Zd d ZZ S) LuminaFeedForwarda' A feed-forward layer. Parameters: hidden_size (`int`): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. intermediate_size (`int`): The intermediate dimension of the feedforward layer. multiple_of (`int`, *optional*): Value to ensure hidden dimension is a multiple of this value. ffn_dim_multiplier (float, *optional*): Custom multiplier for hidden dimension. Defaults to None. Nr$r multiple_offfn_dim_multipliercsttd|d}|durt||}|||d|}tj||dd|_tj||dd|_tj||dd|_t|_ dS)Nrr Fr^) r:r;rTrr<linear_1linear_2linear_3rsilu)rGr$rrrrHrr!r;/s*   zLuminaFeedForward.__init__cCs ||||||Srt)rrrr)rGrJrrr!rONs zLuminaFeedForward.forward)rN) rPrQrRrSrTrrr;rOrVrrrHr!r src sxeZdZdZ ddededededeef fdd Zd eefd d Z dd ej dedeej dej fddZ Z S)TemporalBasicTransformerBlocka A basic Transformer block for video like data. Parameters: dim (`int`): The number of channels in the input and output. time_mix_inner_dim (`int`): The number of channels for temporal attention. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. Nr$time_mix_inner_dimrprqracst||k|_t||_t||dd|_t||_t |||dd|_ |dur;t||_ t ||||d|_ nd|_ d|_ t||_ t|dd|_d|_d|_dS)Nr4)rhr6)r.r2r3ra)r.rar2r3r5)r:r;is_resrr@norm_inr?ff_inrArrrBrrrrnro)rGr$rrprqrarHrr!r;_s:        z&TemporalBasicTransformerBlock.__init__rcKs||_d|_dS)Nr ru)rGrkwargsrrr!rws z4TemporalBasicTransformerBlock.set_chunk_feed_forwardr num_framesryrLc Csv|jd}|j\}}}||}|dddf||||}|dddd}|||||}|}||}|jdurEt|j||j|j}n||}|jrQ||}| |} |j | dd} | |}|j durv| |} |j | |d} | |}| |} |jdurt|j| |j|j} n|| } |jr| |}n| }|dddf||||}|dddd}|||||}|S)Nrrr r)ry)r%rpermuterrnr,rrorrArrrBrr) rGrrryr batch_frames seq_lengthchannelsresidualr}rr+rrr!rOs>            z%TemporalBasicTransformerBlock.forwardrt) rPrQrRrSrTrr;rwr'rUrOrVrrrHr!rRs25 rcsTeZdZ    ddedededed ed eed ed effd d ZddZZS)SkipFFTransformerBlockr8NFTr$rprq kv_input_dimkv_input_dim_proj_use_biasrarrc svt||krt||||_nd|_t|d|_t||||||| d|_t|d|_ t||||||| d|_ dS)NrZ)r.r2r3rr^rar)r.rar2r3rr^r) r:r;rr< kv_mapperrrArrrBr) rGr$rprqrrrrarrrHrr!r;s0    zSkipFFTransformerBlock.__init__cCs|dur|ni}|jdur|t|}||}|j|fd|i|}||}||}|j|fd|i|}||}|S)Nry)rrrkrrArrBr)rGrryrr}rrrr!rOs,   zSkipFFTransformerBlock.forward)r8NFT) rPrQrRrTrrr;rOrVrrrHr!rs.  *rc/szeZdZdZ                  d8d ed edededeededeedededededededededeedeedeededed ed!ed"ef.fd#d$ Z d%ed&e e eeffd'd(Z d9d%ed"ed&e efd)d*Z d9d ed!ed"ed&dfd+d,Zd:d.eed ed&dfd/d0Z    d;d1ejd2eejd3eejd4eejd5eeefd&ejf d6d7ZZS)<FreeNoiseTransformerBlocka A FreeNoise Transformer block. Parameters: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The size of the encoder_hidden_states vector for cross attention. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. num_embeds_ada_norm (`int`, *optional*): The number of diffusion steps used during training. See `Transformer2DModel`. attention_bias (`bool`, defaults to `False`): Configure if the attentions should contain a bias parameter. only_cross_attention (`bool`, defaults to `False`): Whether to use only cross-attention layers. In this case two cross attention layers are used. double_self_attention (`bool`, defaults to `False`): Whether to use two self-attention layers. In this case no cross attention layers are used. upcast_attention (`bool`, defaults to `False`): Whether to upcast the attention computation to float32. This is useful for mixed precision training. norm_elementwise_affine (`bool`, defaults to `True`): Whether to use learnable elementwise affine parameters for normalization. norm_type (`str`, defaults to `"layer_norm"`): The normalization layer to use. Can be `"layer_norm"`, `"ada_norm"` or `"ada_norm_zero"`. final_dropout (`bool` defaults to `False`): Whether to apply a final dropout after the last feed-forward layer. attention_type (`str`, defaults to `"default"`): The type of attention to use. Can be `"default"` or `"gated"` or `"gated-text-image"`. positional_embeddings (`str`, *optional*): The type of positional embeddings to apply to. num_positional_embeddings (`int`, *optional*, defaults to `None`): The maximum number of positional embeddings to apply. ff_inner_dim (`int`, *optional*): Hidden dimension of feed-forward MLP. ff_bias (`bool`, defaults to `True`): Whether or not to use bias in feed-forward MLP. attention_out_bias (`bool`, defaults to `True`): Whether or not to use bias in attention output project layer. context_length (`int`, defaults to `16`): The maximum number of frames that the FreeNoise block processes at once. context_stride (`int`, defaults to `4`): The number of frames to be skipped before starting to process a new batch of `context_length` frames. weighting_scheme (`str`, defaults to `"pyramid"`): The weighting scheme to use for weighting averaging of processed latent frames. As described in the Equation 9. of the [FreeNoise](https://arxiv.org/abs/2310.15169) paper, "pyramid" is the default setting used. r8Nr4FTr[rrpyramidr$rprqrrar6rrrrrrr_rrrrrrrcontext_lengthcontext_strideweighting_schemec st||_||_||_||_||_||_||_| |_ | |_ ||_ ||_ | |_ |||||duo7| dk|_|duo@| dk|_| dk|_| dk|_| dk|_| dvrd|durdtd| d| d | |_||_|rt|durttd |d krt||d |_nd|_tj|| |d |_t|||||| r|nd| |d|_|dus| rt||| |_t|| s|nd||||| |d|_t ||||||d|_!t||| |_"d|_#d|_$dS)NrYrrr[rrrrrrrrrfrrrr)%r:r;r$rprqrrar6rrrrrrset_free_noise_propertiesrrrrrr&r_rrrrr@rArrrBrr?rrrnro)rGr$rprqrrar6rrrrrrr_rrrrrrrrrrrHrr!r;Ts           z"FreeNoiseTransformerBlock.__init__rrLcCsHg}td||jd|jD]}|}t|||j}|||fq|S)Nrr )rangerrminappend)rGr frame_indicesi window_start window_endrrr!_get_frame_indicess z,FreeNoiseTransformerBlock._get_frame_indicescCs|dkr=|ddkr ttd|dd}||ddd}|Sttd|dd}||ddg|ddd}|Std|)Nrrrr rz'Unsupported value for weighting_scheme=)listrr&)rGrrweightsrrr!_get_frame_weightss  z,FreeNoiseTransformerBlock._get_frame_weightscCs||_||_||_dSrt)rrr)rGrrrrrr!rs z3FreeNoiseTransformerBlock.set_free_noise_propertiesrrcCrsrtrurvrrr!rwrxz0FreeNoiseTransformerBlock.set_chunk_feed_forwardrrryrrc Os.|dur|dddurtd|dur|ni}|j}|j} |d} || } ||j |j } t j | || d d d} | dd| k} | sm| |j kr[td| d|j | | dd}| | |j | ft jd| df|d }t |}t| D]\}\}}t |dd||f}|| 9}|dd||f}||}|jdur||}|j|f|jr|nd|d |}||}|jd kr|d}|jdur||}|jdur|jd kr||}|j|f||d |}||}|t| dkr>| s>|dd| df|dd| df|dd| df7<|dd| df|dd| f7<q|dd||f||7<|dd||f|7<qt |dk|||| }| |}|j!durt"|j#||j$|j!}n|#|}||}|jd kr|d}|S) Nrrr )devicerrrzExpected num_frames=z1 to be greater or equal than self.context_length=)r rrr)%rrrrr rsizerr rrr'rEr|r&rzeros zeros_like enumerate ones_likerArrrrrrrBr_lenwheretorrnr,rro)rGrrryrrargsrr rrr frame_weightsis_last_frame_batch_completelast_frame_batch_lengthnum_times_accumulatedaccumulated_valuesr frame_start frame_endr hidden_states_chunkr}rr+rrr!rOs               *.      z!FreeNoiseTransformerBlock.forward)r8Nr4NFFFFTr[rFNNNTTrrr)rr)NNNN)rPrQrRrSrTrrrrr;rrrr rrwr'rUrrrOrVrrrHr!rs;     r   rcsheZdZdZ       dded eed ed ed ed edeffdd Z de j de j fddZ Z S)r?a A feed-forward layer. Parameters: dim (`int`): The number of channels in the input. dim_out (`int`, *optional*): The number of channels in the output. If not given, defaults to `dim`. mult (`int`, *optional*, defaults to 4): The multiplier to use for the hidden dimension. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to be used in feed-forward. final_dropout (`bool` *optional*, defaults to False): Apply a final dropout. bias (`bool`, defaults to True): Whether to use a bias in the linear layer. Nrr8r4FTr$rhmultrr6rr^c st|durt||}|dur|n|}|dkr"t|||d} |dkr/t||d|d} n#|dkr;t|||d} n|dkrGt|||d} n |dkrRt|||d} tg|_ |j | |j t ||j tj |||d|r|j t |dSdS) NgelurrgrM) approximater^r4zgeglu-approximateswiglu) r:r;rTrr rrr ModuleListnetrDropoutr<) rGr$rhrrr6rrr^act_fnrHrr!r;cs*    zFeedForward.__init__rrLcOsDt|dks|dddurd}tdd||jD]}||}q|S)NrrzThe `scale` argument is deprecated and will be ignored. Please remove it, as passing it will raise an error in the future. `scale` should directly be passed while calling the underlying pipeline component i.e., via `cross_attention_kwargs`.z1.0.0)rrr r")rGrrrdeprecation_messagemodulerrr!rOs    zFeedForward.forward)Nrr8r4FNT)rPrQrRrSrTrrrrr;r'rUrOrVrrrHr!r?Us2 &r?)/typingrrrrrr'torch.nn.functionalr functionalrkutilsr r utils.torch_utilsr activationsr rrrrattention_processorrr embeddingsr normalizationrrrr get_loggerrPrModulerUrTr,r-rWrrrrrr?rrrr!s<     ){M2H: