o pi@s(ddlmZmZmZmZmZddlZddlmZddl Zddl m Z m Z ddl mZddlmZddlmZdd lmZmZdd lmZmZmZmZmZmZdd lmZmZdd l m!Z!dd l"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-e.e/Z0Gdddej1Z2Gddde!e eZ3dS))AnyDictOptionalTupleUnionN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin)logging)get_activation) Attention FeedForward)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)TransformerTemporalModel)CrossAttnDownBlock3DCrossAttnUpBlock3D DownBlock3DUNetMidBlock3DCrossAttn UpBlock3Dget_down_block get_up_block)UNet3DConditionOutputcs^eZdZ    ddedededed ed eed effd d Zdej dej fddZ Z S)"I2VGenXLTransformerTemporalEncodergegluFNdimnum_attention_headsattention_head_dim activation_fnupcast_attention ff_inner_dimdropoutc sNttj|ddd|_t||||d|dd|_t|||d|dd|_dS)NTh㈵>)elementwise_affineepsF) query_dimheadsdim_headr,biasr*out_bias)r,r) final_dropout inner_dimr3) super__init__nn LayerNormnorm1rattn1rff)selfr&r'r(r)r*r+r, __class__c/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/unets/unet_i2vgen_xl.pyr85s&   z+I2VGenXLTransformerTemporalEncoder.__init__ hidden_statesreturncCs^||}|j|dd}||}|jdkr|d}||}||}|jdkr-|d}|S)N)encoder_hidden_statesr)r;r<ndimsqueezer=)r>rCnorm_hidden_states attn_output ff_outputrArArBforwardSs      z*I2VGenXLTransformerTemporalEncoder.forward)r$FNr%) __name__ __module__ __qualname__intstrboolrr8torchTensorrL __classcell__rArAr?rBr#4s2r#cseZdZdZdZe        d@d eed ededee dfdee dfdeedfdedeedede eeefdee eeefffdd Z e de e effddZde ee e efffddZdAd!eed"eddfd#d$Zd%d&Zd'd(ZdBd)eddfd*d+Zd,d-Zd.d/Zd0d1Zd2d3Z     4dCd5ejd6e ejeefd7ejd8ejd9eejd:eejd;eejdd?ZZ S)D I2VGenXLUNeta  I2VGenXL UNet. It is a conditional 3D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample-shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `Tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 4): The number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): The number of channels in the output. down_block_types (`Tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): The tuple of downsample blocks to use. up_block_types (`Tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): The tuple of upsample blocks to use. block_out_channels (`Tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. If `None`, normalization and activation layers is skipped in post-processing. cross_attention_dim (`int`, *optional*, defaults to 1280): The dimension of the cross attention features. attention_head_dim (`int`, *optional*, defaults to 64): Attention head dim. num_attention_heads (`int`, *optional*): The number of attention heads. FNrFrrrrrrrri@irZr @ sample_size in_channels out_channelsdown_block_types.up_block_typesblock_out_channelslayers_per_blocknorm_num_groupscross_attention_dimr(r'c st| } t|t|krtd|d|dt|t|kr-td|d|dt| tsEt| t|krEtd| d|dtj|||ddd d |_t d | |dd |d |_ t tjd |d dd dt tj|d |d dd d dt tj|d |dd d d|_ t|d|d |dd|_t tjd |d dd dt tdtj|d |dddd dt tj|d| ddd d|_|dd } t|ddd|_|d} t| | dd|_t t| | t t| | ||_t t| | t t| | |_tg|_tg|_t| tr#| ft|} |d}t|D]-\}}|}||}|t|d k}t||||| | dd|| | |d dd }|j|q+t|d| ddd | | d|dd |_ d|_!t"t#|}t"t#| }|d}t|D]M\}}|t|d k}|}||}|t$|d t|d }|sd}|j!d 7_!nd}t%||d |||| |dd|| ||d|d}|j||}qtj&|d|dd|_'t(d|_)tj|d|dd d |_*dS)Nz\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: rrr) kernel_sizepadding)r'r(r_ num_layersrerF)ri)striderir gelu)r&r'r+r(r))r[r[Tsilu)act_fnr-F) rkr_r` temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsrfr'downsample_paddingdual_cross_attention) r_rqrsrtoutput_scale_factorrfr'rurw) rkr_r`prev_output_channelrq add_upsamplersrtrurfr'rwresolution_idx) num_channels num_groupsr/)+r7r8len ValueError isinstancerPr9Conv2dconv_inrtransformer_in SequentialSiLUimage_latents_proj_inr#image_latents_temporal_encoderAdaptiveAvgPool2dimage_latents_context_embeddingr time_projrtime_embeddingLinearcontext_embedding fps_embedding ModuleList down_blocks up_blocks enumerater appendr mid_blocknum_upsamplerslistreversedminr! GroupNorm conv_norm_outr conv_actconv_out)r>r^r_r`rarbrcrdrerfr(r'time_embed_dimtimestep_input_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_heads up_block_typerzr{up_blockr?rArBr8s           zI2VGenXLUNet.__init__rDcsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorscsHt|dr|||d<|D]\}}|d|||q|S)N get_processor .processorrg)hasattrrnamed_children)rrrsub_namechildfn_recursive_add_processorsrArBrLs zAI2VGenXLUNet.attn_processors..fn_recursive_add_processors)rQrSr9Modulerrr)r>rrrrArrBattn_processorsAs  & zI2VGenXLUNet.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrcsbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorrrg)rrdictrpopr)rrrrrfn_recursive_attn_processorrArBrps   zDI2VGenXLUNet.set_attn_processor..fn_recursive_attn_processorN) rrkeysrrrrQrSr9rr)r>rcountrrrArrBset_attn_processor[s   zI2VGenXLUNet.set_attn_processorr chunk_sizer&csZ|dvr td||pd}dtjjdtdtffdd |D]}|||q"d S) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rrrr&c6t|dr |j||d|D]}|||qdSNset_chunk_feed_forward)rr&rrchildrenrrr&rfn_recursive_feed_forwardrArBr  zGI2VGenXLUNet.enable_forward_chunking..fn_recursive_feed_forwardN)rrSr9rrPr)r>rr&rrArrBenable_forward_chunking~s  z$I2VGenXLUNet.enable_forward_chunkingcs<dtjjdtdtffdd |D]}|ddqdS)Nrrr&crrrrrrArBrrzHI2VGenXLUNet.disable_forward_chunking..fn_recursive_feed_forwardr)rSr9rrPr)r>rrArrBdisable_forward_chunkings z%I2VGenXLUNet.disable_forward_chunkingcCsjtdd|jDrt}ntdd|jDr t}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. cs|]}|jtvVqdSN)r@r.0procrArArB z:I2VGenXLUNet.set_default_attn_processor..csrr)r@rrrArArBrrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)r>rrArArBset_default_attn_processorsz'I2VGenXLUNet.set_default_attn_processorvaluecCs t|ttttfr||_dSdSr)rrrrrgradient_checkpointing)r>rrrArArB_set_gradient_checkpointings z(I2VGenXLUNet._set_gradient_checkpointingcCsHt|jD]\}}t|d|t|d|t|d|t|d|qdS)a>Enables the FreeU mechanism from https://arxiv.org/abs/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. s1s2b1b2N)rrsetattr)r>rrrrrupsample_blockrArArB enable_freeus    zI2VGenXLUNet.enable_freeucCsPhd}t|jD]\}}|D]}t||st||ddur$t||dqq dS)zDisables the FreeU mechanism.>rrrrN)rrrgetattrr)r> freeu_keysrrkrArArB disable_freeus zI2VGenXLUNet.disable_freeucCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsritemsrQr@rMrmodulesrrfuse_projectionsrr)r>_attn_processorrrArArBfuse_qkv_projectionss    z!I2VGenXLUNet.fuse_qkv_projectionscCs|jdur ||jdSdS)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rr)r>rArArBunfuse_qkv_projectionss z#I2VGenXLUNet.unfuse_qkv_projectionsTsampletimestepfps image_latentsimage_embeddingsrE timestep_condcross_attention_kwargs return_dictc % s|j\} } } } }d|jd}d}tfdd|jddDr(tdd}|}t|sV|jjd k}t |t rC|r?tj ntj }n|rHtj ntj}tj|g||jd }nt|jd kre|d|j}||jd }||}|j|jd }|||}||jd }|||j|jd }||}|j| d d }|| d |jj}tj||gdd}|ddddddddf}|d dddd|jd |jd|jd|jd|jd}||}|j\}}}}|d ddd||||}tj||gdd}||}| d|jj!|jj}tj||gdd}|j| d d }|d dddd|jd |jd|jd|jd|jd}|"|}|dddf| | | | |d dddd| | || | }|#|}|| | || | d dddd}tj||gdd}|d dddd|jd | df|jdd}|$|}|j%|| |ddd }|f}|j&D]&} t'| dr| j(r| |||| |d\}}!n | ||| d\}}!||!7}q|j)dur|j)|||| |d}t*|j+D]Q\}"}#|"t|j+dk}$|t|#j, d}!|dt|#j, }|$s#|r#|djdd}t'|#dr:|#j(r:|#|||!||| |d}q|#|||!|| d}q|-|}|.|}|/|}|dddfd| f|jddd dddd}| su|fSt0|dS)a The [`I2VGenXLUNet`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, num_frames, channel, height, width`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. fps (`torch.Tensor`): Frames per second for the video being generated. Used as a "micro-condition". image_latents (`torch.Tensor`): Image encodings from the VAE. image_embeddings (`torch.Tensor`): Projection embeddings of the conditioning image computed with a vision encoder. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] instead of a plain tuple. Returns: [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. r FNc3s|] }|dkVqdS)rNrA)rsdefault_overall_up_factorrArBr8sz'I2VGenXLUNet.forward..z9Forward upsample size to force interpolation output size.Tmps)dtypedevicer)r)repeatsr&r)r&rrFrx) num_framesrrhas_cross_attention)rCtembrErr)rCrr)rErr)rCrres_hidden_states_tuplerE upsample_sizerr)rCrrrr)r)1shaperanyloggerinforS is_tensorrtyperfloatfloat32float64int32int64tensorrtoexpandrrrrrepeat_interleave new_zerosconfigrfcatpermutereshaperrviewr_rrrrrrrrrrresnetsrrrr")%r>rrrrrrErrr batch_sizechannelsrheightwidthforward_upsample_sizer timestepsis_mpsrt_embfps_embemb context_embimage_latents_for_context_embdsimage_latents_context_embs _batch_size _channels_height_width image_embdown_block_res_samplesdownsample_block res_samplesrrrrArrBrLs&               4          6 zI2VGenXLUNet.forward) NrFrFrWrXrYr r[r\r]N)Nr)F)NNNNT)!rMrNrO__doc__ _supports_gradient_checkpointingr rrPrrQrr8propertyrrrrrrrrRrrrrrrSrTr rr"rLrUrArAr?rBrVes   ?#     rV)4typingrrrrrrStorch.nnr9torch.utils.checkpointconfiguration_utilsrr loadersr utilsr activationsr attentionrrattention_processorrrrrrr embeddingsrrmodeling_utilsr!transformers.transformer_temporalrunet_3d_blocksrrrrrr r!unet_3d_conditionr" get_loggerrMrrr#rVrArArArBs$       $ 1