o GiU@sddlZddlmmZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZdd lmZmZmZdd lmZdd lmZdd lmZmZmZe eZGd ddejZGdddejZ GdddZ!e GdddejZ"GdddeeZ#dS)N)nn) ConfigMixinregister_to_config)logging)maybe_allow_in_graph) Attention FeedForward)TimestepEmbedding Timestepsget_3d_rotary_pos_embed)Transformer2DModelOutput) ModelMixin) AdaLayerNorm FP32LayerNormRMSNormcsxeZdZ    ddededededed ed d ffd d Zdej dej dej d e ej ej ej ej ffddZ Z S)EasyAnimateLayerNormZeroTh㈵>fp32_layer_normconditioning_dim embedding_dimelementwise_affineepsbias norm_typereturnNcsttt|_tj|d||d|_|dkr$tj|||d|_dS|dkr2t |||d|_dSt d|d)N)r layer_normrrrzUnsupported `norm_type` (z@) provided. Supported ones are: 'layer_norm', 'fp32_layer_norm'.) super__init__rSiLUsiluLinearlinear LayerNormnormr ValueError)selfrrrrrr __class__i/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_easyanimate.pyr!"s  z!EasyAnimateLayerNormZero.__init__ hidden_statesencoder_hidden_statestembc Csv|||jddd\}}}}}} ||d|d|d}||d|d|d}|||| fS)Nrdim)r%r#chunkr' unsqueeze) r)r.r/r0shiftscalegate enc_shift enc_scaleenc_gater,r,r-forward9s &" z EasyAnimateLayerNormZero.forward)TrTr) __name__ __module__ __qualname__intboolfloatstrr!torchTensortupler< __classcell__r,r,r*r-r!s:rcsLeZdZdedeeddffdd ZddZd ejdejfd d Z Z S) EasyAnimateRotaryPosEmbed patch_sizerope_dimrNcst||_||_dSN)r r!rIrJ)r)rIrJr*r,r-r!Es  z"EasyAnimateRotaryPosEmbed.__init__c Cs|}|}|\}}||}|||kr|} tt|||} n |} tt|||} tt|| d} tt|| d} | | f| | | | ffS)Ng@)r@round) r)src tgt_width tgt_heighttwthhwr resize_height resize_widthcrop_top crop_leftr,r,r-get_resize_crop_region_for_gridKs z9EasyAnimateRotaryPosEmbed.get_resize_crop_region_for_gridr.c Csn|\}}}}}||j}||j}d|j}d|j}|||f||} t|j| ||f|ddd} | S)NZ<rT) grid_size temporal_sizeuse_real)sizerIrYr rJ) r)r.bsc num_frames grid_height grid_widthbase_size_widthbase_size_heightgrid_crops_coordsimage_rotary_embr,r,r-r<\s      z!EasyAnimateRotaryPosEmbed.forward) r=r>r?r@listr!rYrDrEr<rGr,r,r*r-rHDsrHc @sReZdZdZddZ  d dedejdejdejdBd ejdBd ejf d d ZdS)EasyAnimateAttnProcessor2_0z Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). This is used in the EasyAnimateTransformer3DModel model. cCsttds tddS)Nscaled_dot_product_attentionzaEasyAnimateAttnProcessor2_0 requires PyTorch 2.0 or above. To use it, please install PyTorch 2.0.)hasattrF ImportError)r)r,r,r-r!vs z$EasyAnimateAttnProcessor2_0.__init__Nattnr.r/attention_maskrhrc Cs4|jdur|durtj||gdd}||}||}||}|d|jdfdd}|d|jdfdd}|d|jdfdd}|j durR| |}|j dur\| |}|jdur|dur||} | |} | |} | d|jdfdd} | d|jdfdd} | d|jdfdd} |j dur| | } |jdur|| } tj| |gdd}tj| |gdd}tj| |gdd}|durddlm} | |dddd|jddf||dddd|jddf<|js| |dddd|jddf||dddd|jddf<tj||||ddd}|dddd }||j}|dur|ddd|jdf|dd|jddf}}t|d ddurm|jd |}|jd|}t|d ddur{||}||fSt|d ddur|jd |}|jd|}||fS) Nr1r2r)apply_rotary_embF) attn_mask dropout_p is_causalrto_outr to_add_out) add_q_projrDcatto_qto_kto_v unflattenheads transposenorm_qnorm_k add_k_proj add_v_proj norm_added_q norm_added_k embeddingsrrshapeis_cross_attentionrmrkflattentodtypegetattrrwrx) r)ror.r/rprhquerykeyvalue encoder_query encoder_key encoder_valuerrr,r,r-__call__|sl                        z$EasyAnimateAttnProcessor2_0.__call__)NN) r=r>r?__doc__r!r rDrErr,r,r,r-rjps$ rjcseZdZ           d ded ed ed ed ed edededededBdededededeffdd Z d!dej dej dej de ej ej fdBde ej ej ff ddZ Z S)"EasyAnimateTransformerBlockrsgelu-approximateTư>NFrr3num_attention_headsattention_head_dimtime_embed_dimdropout activation_fnnorm_elementwise_affinenorm_eps final_dropout ff_inner_dimff_biasqk_norm after_normris_mmdit_blockc stt|||||dd|_t|||| rdndddd|r |nd|r%dndtd |_t|||||dd|_t|||| | | d|_ d|_ |rRt|||| | | d|_ d|_ | rat |||d|_ dSdS) NT)rrrrF) query_dimdim_headrrrradded_proj_biasadded_kv_proj_dimcontext_pre_only processor)rrr inner_dimrr) r r!rnorm1r rjattn1norm2r fftxt_ffnorm3r)r)r3rrrrrrrrrrrrrrr*r,r-r!sR        z$EasyAnimateTransformerBlock.__init__r.r/r0rhrc Cs||||\}}}}|j|||d\} } ||d| }||d| }||||\}}} } |jdurV|||}|jdurM|||}n|||}n||}|jdurf||}n||}|| d|}|| d|}||fS)N)r.r/rhr1)rrr5rrrr) r)r.r/r0rhnorm_hidden_statesnorm_encoder_hidden_statesgate_msa enc_gate_msaattn_hidden_statesattn_encoder_hidden_statesgate_ff enc_gate_ffr,r,r-r<s2        z#EasyAnimateTransformerBlock.forward) rsrTrTNTTFrTrK) r=r>r?r@rBrCrAr!rDrErFr<rGr,r,r*r-rsn     Grc2s@eZdZdZdZdgZgdZe                    d8dedededBdedBdedBdedede de dededede d ed!e d"ed#ed$e d%e d&e d'e d(e d)e d*e f0fd+d, Z      d9d-e jd.e jd/e jdBd0e jdBd1e jdBd2e jdBd3e jdBd4e d5ee jeBfd6d7ZZS):EasyAnimateTransformer3DModela A Transformer model for video-like data in [EasyAnimate](https://github.com/aigc-apps/EasyAnimate). Parameters: num_attention_heads (`int`, defaults to `48`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `64`): The number of channels in each head. in_channels (`int`, defaults to `16`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `16`): The number of channels in the output. patch_size (`int`, defaults to `2`): The size of the patches to use in the patch embedding layer. sample_width (`int`, defaults to `90`): The width of the input latents. sample_height (`int`, defaults to `60`): The height of the input latents. activation_fn (`str`, defaults to `"gelu-approximate"`): Activation function to use in feed-forward. timestep_activation_fn (`str`, defaults to `"silu"`): Activation function to use when generating the timestep embeddings. num_layers (`int`, defaults to `30`): The number of layers of Transformer blocks to use. mmdit_layers (`int`, defaults to `1000`): The number of layers of Multi Modal Transformer blocks to use. dropout (`float`, defaults to `0.0`): The dropout probability to use. time_embed_dim (`int`, defaults to `512`): Output dimension of timestep embeddings. text_embed_dim (`int`, defaults to `4096`): Input dimension of text embeddings from the text encoder. norm_eps (`float`, defaults to `1e-5`): The epsilon value to use in normalization layers. norm_elementwise_affine (`bool`, defaults to `True`): Whether to use elementwise affine in normalization layers. flip_sin_to_cos (`bool`, defaults to `True`): Whether to flip the sin to cos in the time embedding. time_position_encoding_type (`str`, defaults to `3d_rope`): Type of time position encoding. after_norm (`bool`, defaults to `False`): Flag to apply normalization after. resize_inpaint_mask_directly (`bool`, defaults to `True`): Flag to resize inpaint mask directly. enable_text_attention_mask (`bool`, defaults to `True`): Flag to enable text attention mask. add_noise_in_inpaint_model (`bool`, defaults to `False`): Flag to add noise in inpaint model. Tr)z^proj$r'z ^proj_out$0@NrZr[rr#rrsFr3d_roperr in_channels out_channelsrI sample_width sample_heightrtimestep_activation_fn freq_shift num_layers mmdit_layersrradd_norm_text_encodertext_embed_dimtext_embed_dim_t5rrflip_sin_to_costime_position_encoding_typeresize_inpaint_mask_directlyenable_text_attention_maskadd_noise_in_inpaint_modelc sJtt|| |_t | |_t||_tj |||f|dd|_ d|_ d|_ |sEt ||_ |durDt ||_ n$tt|dddt ||_ |duritt|dddt ||_ t f ddt| D|_t|_t ddd |_t ||||_d |_dS) NT) kernel_sizestriderr)rcs4g|]}t |krdndd qS)TF) r3rrrrrrrrr)r).0_ rrrrrrrrrrr,r- s z:EasyAnimateTransformer3DModel.__init__..rr1)r output_dimrr chunk_dimF)r r!r time_projr time_embeddingrHrope_embeddingrConv2dproj text_proj text_proj_t5r$ Sequentialr ModuleListrangetransformer_blocksr& norm_finalrnorm_outproj_outgradient_checkpointing)r)rrrrrIrrrrrrrrrrrrrrrrrrrrr*rr-r!ssJ    z&EasyAnimateTransformer3DModel.__init__r.timestep timestep_condr/encoder_hidden_states_t5inpaint_latentscontrol_latents return_dictrc  Cs|\} } } } } |jj}| |}| |}||j|jd}|||}||}|dur6t ||gd}|durBt ||gd}| ddddd dd}| |}| d| df ddddd}| dddd}||}|dur||}tj||gdd}|jD]}tr|jr||||||\}}q|||||\}}q||}|j||d }||}|jj}|| | ||| ||}| ddddd dd  d d  dd}|s|fSt|d S) N)rr1rrrrqr2)r0r)sample)r_configrIrrrrrrDconcatpermuterrr~rrrrz contiguousris_grad_enabledr_gradient_checkpointing_funcrrrreshaper)r)r.rrr/rrrr batch_sizechannels video_lengthheightwidthppost_patch_heightpost_patch_widthr0rhblockoutputr,r,r-r<sL            & z%EasyAnimateTransformer3DModel.forward)rrNNNrZr[rr#rrrrsrFrNrTTrFTTT)NNNNNT)r=r>r?r _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patternsrr@rCrBrAr!rDrErFrr<rGr,r,r*r-r<s2     ]  r)$rDtorch.nn.functionalr functionalrmconfiguration_utilsrrutilsrutils.torch_utilsr attentionr r rr r r modeling_outputsrmodeling_utilsr normalizationrrr get_loggerr=loggerModulerrHrjrrr,r,r,r-s$      #,`k