o Gis@sddlmZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z dd lmZmZmZdd lmZmZmZmZmZdd lmZmZdd lmZdd lmZddlm Z m!Z!m"Z"ddl#m$Z$e %e&Z'Gdddej(Z)Gdddeeee Z*dS))AnyN) ConfigMixinregister_to_config)UNet2DConditionLoadersMixin)logging)get_activation) AttentionAttentionMixin FeedForward)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)TransformerTemporalModel)UNetMidBlock3DCrossAttnget_down_block get_up_block)UNet3DConditionOutputcs^eZdZ    ddedededed ed edBd effd d ZdejdejfddZ Z S)"I2VGenXLTransformerTemporalEncodergegluFNdimnum_attention_headsattention_head_dim activation_fnupcast_attention ff_inner_dimdropoutc sNttj|ddd|_t||||d|dd|_t|||d|dd|_dS)NTh㈵>)elementwise_affineepsF) query_dimheadsdim_headr$biasr"out_bias)r$r! final_dropout inner_dimr+) super__init__nn LayerNormnorm1r attn1r ff)selfrrr r!r"r#r$ __class__Y/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/unets/unet_i2vgen_xl.pyr0/s&   z+I2VGenXLTransformerTemporalEncoder.__init__ hidden_statesreturncCs^||}|j|dd}||}|jdkr|d}||}||}|jdkr-|d}|S)N)encoder_hidden_statesr)r3r4ndimsqueezer5)r6r;norm_hidden_states attn_output ff_outputr9r9r:forwardMs      z*I2VGenXLTransformerTemporalEncoder.forward)rFNr) __name__ __module__ __qualname__intstrboolr0torchTensorrD __classcell__r9r9r7r:r.s2rcsdeZdZdZdZe        d8d edBd ededeedfdeedfdeedfdededBdedeeeBdeeeBdBffdd Z d9dedBdeddfddZ d d!Z d"d#Z d$d%Z d&d'Zd(d)Zd*d+Z     ,d:d-ejd.ejeBeBd/ejd0ejd1ejdBd2ejdBd3ejdBd4eeefdBd5edeeejBfd6d7ZZS); I2VGenXLUNeta  I2VGenXL UNet. It is a conditional 3D UNet model that takes a noisy sample, conditional state, and a timestep and returns a sample-shaped output. This model inherits from [`ModelMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Parameters: sample_size (`int` or `tuple[int, int]`, *optional*, defaults to `None`): Height and width of input/output sample. in_channels (`int`, *optional*, defaults to 4): The number of channels in the input sample. out_channels (`int`, *optional*, defaults to 4): The number of channels in the output. down_block_types (`tuple[str]`, *optional*, defaults to `("CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D")`): The tuple of downsample blocks to use. up_block_types (`tuple[str]`, *optional*, defaults to `("UpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D", "CrossAttnUpBlock2D")`): The tuple of upsample blocks to use. block_out_channels (`tuple[int]`, *optional*, defaults to `(320, 640, 1280, 1280)`): The tuple of output channels for each block. layers_per_block (`int`, *optional*, defaults to 2): The number of layers per block. norm_num_groups (`int`, *optional*, defaults to 32): The number of groups to use for the normalization. If `None`, normalization and activation layers is skipped in post-processing. cross_attention_dim (`int`, *optional*, defaults to 1280): The dimension of the cross attention features. attention_head_dim (`int`, *optional*, defaults to 64): Attention head dim. num_attention_heads (`int`, *optional*): The number of attention heads. FNr>CrossAttnDownBlock3DrPrP DownBlock3D UpBlock3DCrossAttnUpBlock3DrTrTi@irVr @ sample_size in_channels out_channelsdown_block_types.up_block_typesblock_out_channelslayers_per_blocknorm_num_groupscross_attention_dimr rc st| } t|t|krtd|d|dt|t|kr-td|d|dt| tsEt| t|krEtd| d|dtj|||ddd d |_t d | |dd |d |_ t tjd |d dd dt tj|d |d dd d dt tj|d |dd d d|_ t|d|d |dd|_t tjd |d dd dt tdtj|d |dddd dt tj|d| ddd d|_|dd } t|ddd|_|d} t| | dd|_t t| | t t| | ||_t t| | t t| | |_tg|_tg|_t| tr#| ft|} |d}t|D]-\}}|}||}|t|d k}t||||| | dd|| | |d dd }|j|q+t|d| ddd | | d|dd |_ d|_!t"t#|}t"t#| }|d}t|D]M\}}|t|d k}|}||}|t$|d t|d }|sd}|j!d 7_!nd}t%||d |||| |dd|| ||d|d}|j||}qtj&|d|dd|_'t(d|_)tj|d|dd d |_*dS)Nz\Must provide the same number of `down_block_types` as `up_block_types`. `down_block_types`: z. `up_block_types`: .zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: rrr) kernel_sizepadding)rr r[ num_layersrar>)re)striderergelu)rrr#r r!)rWrWTsilu)act_fnr%F) rgr[r\ temb_channelsadd_downsample resnet_eps resnet_act_fn resnet_groupsrbrdownsample_paddingdual_cross_attention) r[rmrorpoutput_scale_factorrbrrqrs) rgr[r\prev_output_channelrm add_upsamplerorprqrbrrsresolution_idx) num_channels num_groupsr')+r/r0len ValueError isinstancerHr1Conv2dconv_inrtransformer_in SequentialSiLUimage_latents_proj_inrimage_latents_temporal_encoderAdaptiveAvgPool2dimage_latents_context_embeddingr time_projrtime_embeddingLinearcontext_embedding fps_embedding ModuleList down_blocks up_blocks enumeraterappendr mid_blocknum_upsamplerslistreversedminr GroupNorm conv_norm_outr conv_actconv_out)r6rZr[r\r]r^r_r`rarbr rtime_embed_dimtimestep_input_dimoutput_channelidown_block_type input_channelis_final_block down_blockreversed_block_out_channelsreversed_num_attention_heads up_block_typervrwup_blockr7r9r:r0|s           zI2VGenXLUNet.__init__r chunk_sizerr<csZ|dvr td||pd}dtjjdtdtffdd |D]}|||q"d S) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rmodulerrc6t|dr |j||d|D]}|||qdSNset_chunk_feed_forward)rrhasattrrchildrenrrrchildfn_recursive_feed_forwardr9r:rO  zGI2VGenXLUNet.enable_forward_chunking..fn_recursive_feed_forwardN)r|rKr1ModulerHr)r6rrrr9rr:enable_forward_chunking<s  z$I2VGenXLUNet.enable_forward_chunkingcs<dtjjdtdtffdd |D]}|ddqdS)Nrrrcrrrrrr9r:r[rzHI2VGenXLUNet.disable_forward_chunking..fn_recursive_feed_forwardr)rKr1rrHr)r6rr9rr:disable_forward_chunkingZs z%I2VGenXLUNet.disable_forward_chunkingcCsjtdd|jDrt}ntdd|jDr t}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. cs|]}|jtvVqdSN)r8r .0procr9r9r: jz:I2VGenXLUNet.set_default_attn_processor..csrr)r8rrr9r9r:rlrzOCannot call `set_default_attn_processor` when attention processors are of type N) allattn_processorsvaluesrrr|nextiterset_attn_processor)r6 processorr9r9r:set_default_attn_processorfsz'I2VGenXLUNet.set_default_attn_processorcCsHt|jD]\}}t|d|t|d|t|d|t|d|qdS)aFEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. s1s2b1b2N)rrsetattr)r6rrrrrupsample_blockr9r9r: enable_freeuvs    zI2VGenXLUNet.enable_freeucCsPhd}t|jD]\}}|D]}t||st||ddur$t||dqq dS)zDisables the FreeU mechanism.>rrrrN)rrrgetattrr)r6 freeu_keysrrkr9r9r: disable_freeus zI2VGenXLUNet.disable_freeucCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsritemsrIr8rEr|modulesr}r fuse_projectionsrr)r6_attn_processorrr9r9r:fuse_qkv_projectionss   z!I2VGenXLUNet.fuse_qkv_projectionscCs|jdur ||jdSdS)unDisables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)rr)r6r9r9r:unfuse_qkv_projectionss z#I2VGenXLUNet.unfuse_qkv_projectionsTsampletimestepfps image_latentsimage_embeddingsr= timestep_condcross_attention_kwargs return_dictc & s |j\} } } } }d|jd}d}tfdd|jddDr(tdd}|}t|s`|jjd k}|jjd k}t |t rK|sD|rGtj ntj }n |sO|rRtj ntj}tj|g||jd }nt|jd kro|d|j}||jd }||}|j|jd }|||}||jd }|||j|jd }||}|j| d |jd | d}|| d |jj}tj||gdd}|ddddddddf}|d dddd|jd |jd|jd|jd|jd}||}|j\}}}}|d ddd||||}tj||gdd}||}| d|jj!|jj}tj||gdd}|j| d |jd | d}|d dddd|jd |jd|jd|jd|jd}|"|}|dddf| | | | |d dddd| | || | }|#|}|| | || | d dddd}tj||gdd}|d dddd|jd | df|jdd}|$|}|j%|| |ddd }|f} |j&D]&}!t'|!dr|!j(r|!|||| |d\}}"n |!||| d\}}"| |"7} q|j)dur|j)|||| |d}t*|j+D]Q\}#}$|#t|j+dk}%| t|$j, d}"| dt|$j, } |%s9|r9| djdd}t'|$drP|$j(rP|$|||"||| |d}q |$|||"|| d}q |-|}|.|}|/|}|dddfd| f|jddd dddd}| s|fSt0|dS)a The [`I2VGenXLUNet`] forward method. Args: sample (`torch.Tensor`): The noisy input tensor with the following shape `(batch, num_frames, channel, height, width`. timestep (`torch.Tensor` or `float` or `int`): The number of timesteps to denoise an input. fps (`torch.Tensor`): Frames per second for the video being generated. Used as a "micro-condition". image_latents (`torch.Tensor`): Image encodings from the VAE. image_embeddings (`torch.Tensor`): Projection embeddings of the conditioning image computed with a vision encoder. encoder_hidden_states (`torch.Tensor`): The encoder hidden states with shape `(batch, sequence_length, feature_dim)`. cross_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] instead of a plain tuple. Returns: [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] or `tuple`: If `return_dict` is True, an [`~models.unets.unet_3d_condition.UNet3DConditionOutput`] is returned, otherwise a `tuple` is returned where the first element is the sample tensor. rFNc3s|] }|dkVqdS)rNr9)rsdefault_overall_up_factorr9r:rsz'I2VGenXLUNet.forward..z9Forward upsample size to force interpolation output size.Tmpsnpu)dtypedevicer)r)r output_sizer)rrr>rt) num_framesrrhas_cross_attention)r;tembr=rr)r;rr)r=rr)r;rres_hidden_states_tupler= upsample_sizerr)r;rrrr)r)1shaperanyloggerinforK is_tensorrtyper}floatfloat32float64int32int64tensorr{toexpandrrrrrepeat_interleave new_zerosconfigrbcatpermutereshaperrviewr[rrrrrrrrrrresnetsrrrr)&r6rrrrrr=rrr batch_sizechannelsrheightwidthforward_upsample_sizer timestepsis_mpsis_npurt_embfps_embemb context_embimage_latents_for_context_embdsimage_latents_context_embs _batch_size _channels_height_width image_embdown_block_res_samplesdownsample_block res_samplesrrrr9rr:rDs&                4          6 zI2VGenXLUNet.forward) Nr>r>rOrRrUrrWrXrYN)Nr)NNNNT)rErFrG__doc__ _supports_gradient_checkpointingrrHtuplerIr0rrrrrrrrKrLrdictrrJrrDrMr9r9r7r:rN_s    @      rN)+typingrrKtorch.nnr1configuration_utilsrrloadersrutilsr activationsr attentionr r r attention_processorr rrrr embeddingsrrmodeling_utilsr!transformers.transformer_temporalrunet_3d_blocksrrrunet_3d_conditionr get_loggerrErrrrNr9r9r9r:s"         1