o Gi3E @sLddlmZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZddlmZd d lmZd d lmZmZd d lmZd d lmZmZd dlmZd dlmZd dlm Z m!Z!e"e#Z$Gdddej%Z&Gdddej%Z'Gdddej%Z(eGdddej%Z)Gdddej%Z*eGdddeee e eZ+dS))AnyN) ConfigMixinregister_to_config)PeftAdapterMixin)FromOriginalModelMixin)apply_lora_scalelogging)maybe_allow_in_graph) FeedForward)MochiAttentionMochiAttnProcessor2_0) CacheMixin)%MochiCombinedTimestepCaptionEmbedding PatchEmbed)Transformer2DModelOutput) ModelMixin)AdaLayerNormContinuousRMSNormcs,eZdZdeffdd ZdddZZS)MochiModulatedRMSNormepscs"t||_td|d|_dSNrF)super__init__rrnorm)selfr __class__c/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_mochi.pyr's zMochiModulatedRMSNorm.__init__NcCs:|j}|tj}||}|dur||}||}|SN)dtypetotorchfloat32r)r hidden_statesscalehidden_states_dtyperrr forward-s   zMochiModulatedRMSNorm.forwardr!)__name__ __module__ __qualname__floatrr) __classcell__rrrr r&srcsHeZdZ  d dedeffdd Zdejdejd ejfd d ZZS) MochiLayerNormContinuoush㈵>T embedding_dimconditioning_embedding_dimcs6tt|_tj|||d|_t|d|_dS)N)biasr) rrnnSiLUsiluLinearlinear_1rr)rr1r2rr3rrr r<s  z!MochiLayerNormContinuous.__init__xconditioning_embeddingreturncCsF|j}||||j}||d|dtj}||S)N)r"r9r7r#r unsqueezer$r%)rr:r; input_dtyper'rrr r)Js z MochiLayerNormContinuous.forward)r0T) r*r+r,intrr$Tensorr)r.rrrr r/;sr/c sheZdZdZ ddededededd f fd d Zd ej d ej de ej ej ej ej ffddZ Z S)MochiRMSNormZeroz} Adaptive RMS Norm used in Mochi. Parameters: embedding_dim (`int`): The size of each embedding vector. r0Fr1 hidden_dimrelementwise_affiner<Ncs4tt|_t|||_td|d|_dSr) rrr5r6r7r8linearrr)rr1rCrrDrrr r`s  zMochiRMSNormZero.__init__r&embcCsp|j}|||}|jddd\}}}}||tjd|dddftj}||}||||fS)Nr=dim)r"rEr7chunkrr#r$r%)rr&rFr( scale_msagate_msa scale_mlpgate_mlprrr r)is .  zMochiRMSNormZero.forward)r0F) r*r+r,__doc__r@r-boolrr$rAtupler)r.rrrr rBXs* rBcseZdZdZ    ddededed ed ed ed ed eddffdd Z dde j de j de j de j de j dBde e j e j ff ddZ Z S)MochiTransformerBlocka/ Transformer block used in [Mochi](https://huggingface.co/genmo/mochi-1-preview). Args: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. qk_norm (`str`, defaults to `"rms_norm"`): The normalization layer to use. activation_fn (`str`, defaults to `"swiglu"`): Activation function to use in feed-forward. context_pre_only (`bool`, defaults to `False`): Whether or not to process context-related conditions with additional layers. eps (`float`, defaults to `1e-6`): Epsilon value for normalization layers. rms_normswigluFư>rInum_attention_headsattention_head_dimpooled_projection_dimqk_norm activation_fncontext_pre_onlyrr<Nc s(t||_d|dd|_d|dd|_t|d||dd|_|s3t|d||dd|_nt|||d|_t |||d|d|||t dd |_ t |d |_ |jsZt |d nd|_t ||_|jsjt |d nd|_t||j|dd |_d|_|st||j|dd |_t |d |_t |d |_dS) NrGr rF)rrD)r1r2rr0) query_dimheadsdim_headr3added_kv_proj_dimadded_proj_biasout_dimout_context_dimr[ processorrr4) inner_dimrZr3)rrr[ ff_inner_dimff_context_inner_dimrBnorm1 norm1_contextr/r rattn1rnorm2 norm2_contextnorm3 norm3_contextr ff ff_contextnorm4 norm4_context) rrIrVrWrXrYrZr[rrrr rsP    zMochiTransformerBlock.__init__r&encoder_hidden_statestembencoder_attention_maskimage_rotary_embcCs |||\}}}} |js|||\} } } } n|||} |j|| ||d\}}|||t|d}||d|d tj }| |}|| |t| d}|js|| |t| d}||d| d tj } || }|||t| d}||fS)N)r&rrruattention_maskr=)rgr[rhrirjr$tanhr>rlr#r%rnrprkrmrorq)rr&rrrsrtrunorm_hidden_statesrLrMrNnorm_encoder_hidden_states enc_gate_msa enc_scale_mlp enc_gate_mlpattn_hidden_statescontext_attn_hidden_states ff_outputcontext_ff_outputrrr r)s8    zMochiTransformerBlock.forward)rSrTFrUr!)r*r+r,rOr@strrPr-rr$rArQr)r.rrrr rRvsN  DrRcseZdZdZddededdffdd Zdejfd d Z  dd ed ed edej dBdej dBdejf ddZ dejdejdejfddZ  ddejd ed ed edej dBdej dBde ejejffddZZS) MochiRoPEa RoPE implementation used in [Mochi](https://huggingface.co/genmo/mochi-1-preview). Args: base_height (`int`, defaults to `192`): Base height used to compute interpolation scale for rotary positional embeddings. base_width (`int`, defaults to `192`): Base width used to compute interpolation scale for rotary positional embeddings. base_height base_widthr<Ncst|||_dSr!)rr target_area)rrrrrr rs zMochiRoPE.__init__cCs4tj|||d||d}|dd|dddS)Nr=devicer"r )r$linspace)rstartstopnumrr"edgesrrr _centersszMochiRoPE._centers num_framesheightwidthrr"cCs|j||d}tj|||d}|| |d||d|||}|| |d||d|||} tj||| dd\} } } tj| | | gdddd} | S) Ng?rr ij)indexingrrHr)rr$arangermeshgridstackview)rrrrrr"r'thwgrid_tgrid_hgrid_w positionsrrr _get_positions s$$zMochiRoPE._get_positionsfreqsposcCslt|jjtjtd|tj|tj}Wdn1s#wYt|}t|}||fS)Nz nd,dhf->nhf) r$autocastrtyper%einsumr#cossin)rrr freqs_cos freqs_sinrrr _create_ropes    zMochiRoPE._create_ropepos_frequenciesc Cs*||||||}|||\}} || fSr!)rr) rrrrrrr"rrope_cosrope_sinrrr r)&s zMochiRoPE.forward)rr)NN)r*r+r,rOr@rr$rArrr"rrrQr)r.rrrr rsJ   rcseZdZdZdZdgZddgZe       d(dededededededed Bde dedede dedd ffdd Z e d  d)d!e j d"e j d#e jd$e j d ee efd Bd%ede j fd&d'ZZS)*MochiTransformer3DModelaB A Transformer model for video-like data introduced in [Mochi](https://huggingface.co/genmo/mochi-1-preview). Args: patch_size (`int`, defaults to `2`): The size of the patches to use in the patch embedding layer. num_attention_heads (`int`, defaults to `24`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `128`): The number of channels in each head. num_layers (`int`, defaults to `48`): The number of layers of Transformer blocks to use. in_channels (`int`, defaults to `12`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `None`): The number of channels in the output. qk_norm (`str`, defaults to `"rms_norm"`): The normalization layer to use. text_embed_dim (`int`, defaults to `4096`): Input dimension of text embeddings from the text encoder. time_embed_dim (`int`, defaults to `256`): Output dimension of timestep embeddings. activation_fn (`str`, defaults to `"swiglu"`): Activation function to use in feed-forward. max_sequence_length (`int`, defaults to `256`): The maximum sequence length of text embeddings supported. TrR patch_embedrr 0 NrSrT patch_sizerVrW num_layersrX in_channels out_channelsrYtext_embed_dimtime_embed_dimrZmax_sequence_lengthr<c st|p |}t||dd|_t| | dd|_tt ddfd|_ t |_ t fddtD|_td d d d |_t||||_d |_dS) N)rr embed_dimpos_embed_type)r1rXrrrVrr gc s*g|]}t|dkdqS)r=)rIrVrWrXrYrZr[)rR).0irZrWrdrVrrXrYrr ~s  z4MochiTransformer3DModel.__init__..FrU layer_norm)rDr norm_type)rrrrr time_embedr5 Parameterr$fullrrrope ModuleListrangetransformer_blocksrnorm_outr8proj_outgradient_checkpointing) rrrVrWrrXrrrYrrrZrrrr rVs@   z MochiTransformer3DModel.__init__attention_kwargsr&rrtimesteprt return_dictc CsR|j\}}} } } |jj} | | } | | }|j||||jd\}}|ddddddd}||}|d|dfdd}|j |j | | ||j t j d}t|jD]#\}}t ri|jri|||||||\}}qQ||||||d \}}qQ|||}||}||| | || | d}|dd ddddd }||d| | | }|s|fSt|d S) N) hidden_dtyperr r=rrGrr)r&rrrsrtru)sample)shapeconfigrrr"permuteflattenr unflattenrrrr$r% enumerateris_grad_enabledr_gradient_checkpointing_funcrrreshaper)rr&rrrrtrr batch_size num_channelsrrrppost_patch_heightpost_patch_widthrsrurblockoutputrrr r)s\         zMochiTransformer3DModel.forward) r rrrrrNrSrrrTr)NT)r*r+r,rO _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patternsrr@rrrr$rA LongTensordictrrPr)r.rrrr r4s~     @r),typingrr$torch.nnr5configuration_utilsrrloadersrloaders.single_file_modelrutilsrr utils.torch_utilsr attentionr attention_processorr r cache_utilsr embeddingsrrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerr*loggerModulerr/rBrRrrrrrr s0          >