o pi98@sddlmZmZmZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZmZdd lmZmZdd lmZdd lmZdd lmZmZmZe eZGd ddejZ GdddeeZ!dS))AnyDictOptionalN) ConfigMixinregister_to_config)logging)LuminaFeedForward) AttentionLuminaAttnProcessor2_0)&LuminaCombinedTimestepCaptionEmbeddingLuminaPatchEmbed)Transformer2DModelOutput) ModelMixin)LuminaLayerNormContinuousLuminaRMSNormZeroRMSNormcseZdZdZ ddedededededed ed ed ed d ffdd Z ddej dej dej dej dej dej de e e e ffddZZS)LuminaNextDiTBlocka A LuminaNextDiTBlock for LuminaNextDiT2DModel. Parameters: dim (`int`): Embedding dimension of the input features. num_attention_heads (`int`): Number of attention heads. num_kv_heads (`int`): Number of attention heads in key and value features (if using GQA), or set to None for the same as query. multiple_of (`int`): The number of multiple of ffn layer. ffn_dim_multiplier (`float`): The multipier factor of ffn layer dimension. norm_eps (`float`): The eps for norm layer. qk_norm (`bool`): normalization for query and key. cross_attention_dim (`int`): Cross attention embedding dimension of the input text prompt hidden_states. norm_elementwise_affine (`bool`, *optional*, defaults to True), Tdimnum_attention_heads num_kv_heads multiple_offfn_dim_multipliernorm_epsqk_normcross_attention_dimnorm_elementwise_affinereturnNc st|||_tt|g|_t|d|||rdnd||dddt d |_ t |j _ t|||||r:dnd||dddt d |_ t|d|||d|_t||| d|_t||| d|_t||| d|_t||| d|_t||| d|_dS) Nlayer_norm_across_headsh㈵>F) query_dimrdim_headrheadskv_headsepsbiasout_bias processor)r inner_dimrr) embedding_dimrr)r%elementwise_affine)super__init__head_dimnn Parametertorchzerosgater r attn1Identityto_outattn2r feed_forwardrnorm1r ffn_norm1norm2 ffn_norm2 norm1_context) selfrrrrrrrrr __class__l/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/transformers/lumina_nextdit2d.pyr.5sV     zLuminaNextDiTBlock.__init__ hidden_statesattention_maskimage_rotary_embencoder_hidden_states encoder_masktembcross_attention_kwargscCs|}|||\} } } } |jd| | |||d|} ||}|jd| |||dd|}||jdddd}| |}|d}|jjd|}|| d| |}| | |d| d}|| d| |}|S)a Perform a forward pass through the LuminaNextDiTBlock. Parameters: hidden_states (`torch.Tensor`): The input of hidden_states for LuminaNextDiTBlock. attention_mask (`torch.Tensor): The input of hidden_states corresponse attention mask. image_rotary_emb (`torch.Tensor`): Precomputed cosine and sine frequencies. encoder_hidden_states: (`torch.Tensor`): The hidden_states of text prompt are processed by Gemma encoder. encoder_mask (`torch.Tensor`): The hidden_states of text prompt attention mask. temb (`torch.Tensor`): Timestep embedding with text prompt embedding. cross_attention_kwargs (`Dict[str, Any]`): kwargs for cross attention. )rDrGrEquery_rotary_embkey_rotary_embNrrB)r:r5r>r8r4tanhviewflattenr7 unsqueezer<r9r;r=)r?rDrErFrGrHrIrJresidualnorm_hidden_statesgate_msa scale_mlpgate_mlpself_attn_outputnorm_encoder_hidden_statescross_attn_outputmixed_attn_output mlp_outputrBrBrCforwardvs:  zLuminaNextDiTBlock.forward)T)N)__name__ __module__ __qualname____doc__intfloatboolr.r2Tensorrrstrrr^ __classcell__rBrBr@rCr$sP   Irc seZdZdZe          d&d edeedeedeedeedeedeedeedeedeedeedeedeedeeddffdd Z  d'de j de j d e j d!e j d"e j d#e e efde j fd$d%ZZS)(LuminaNextDiT2DModelaa LuminaNextDiT: Diffusion model with a Transformer backbone. Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers. Parameters: sample_size (`int`): The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, *optional*, (`int`, *optional*, defaults to 2): The size of each patch in the image. This parameter defines the resolution of patches fed into the model. in_channels (`int`, *optional*, defaults to 4): The number of input channels for the model. Typically, this matches the number of channels in the input images. hidden_size (`int`, *optional*, defaults to 4096): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. num_layers (`int`, *optional*, default to 32): The number of layers in the model. This defines the depth of the neural network. num_attention_heads (`int`, *optional*, defaults to 32): The number of attention heads in each attention layer. This parameter specifies how many separate attention mechanisms are used. num_kv_heads (`int`, *optional*, defaults to 8): The number of key-value heads in the attention mechanism, if different from the number of attention heads. If None, it defaults to num_attention_heads. multiple_of (`int`, *optional*, defaults to 256): A factor that the hidden size should be a multiple of. This can help optimize certain hardware configurations. ffn_dim_multiplier (`float`, *optional*): A multiplier for the dimensionality of the feed-forward network. If None, it uses a default value based on the model configuration. norm_eps (`float`, *optional*, defaults to 1e-5): A small value added to the denominator for numerical stability in normalization layers. learn_sigma (`bool`, *optional*, defaults to True): Whether the model should learn the sigma parameter, which might be related to uncertainty or variance in predictions. qk_norm (`bool`, *optional*, defaults to True): Indicates if the queries and keys in the attention mechanism should be normalized. cross_attention_dim (`int`, *optional*, defaults to 2048): The dimensionality of the text embeddings. This parameter defines the size of the text representations used in the model. scaling_factor (`float`, *optional*, defaults to 1.0): A scaling factor applied to certain parameters or layers in the model. This can be used for adjusting the overall scale of the model's operations. r r) Nr T? sample_size patch_size in_channels hidden_size num_layersrrrrr learn_sigmarrscaling_factorrc st||_||_||_| r|dn||_|_|_|_||_ t ||dd|_ t t|_ttdd|_t fddt|D|_ttddd d|||jd |_d d ksvJd dS)Nr T)rqrr embed_dimr&i)rsrc s"g|] }tqSrB)r).0_rrrsrrrrrrBrC s z1LuminaNextDiT2DModel.__init__..Fgư>)r+conditioning_embedding_dimr,r%r&out_dimr)rz+2d rope needs head dim to be divisible by 4)r-r.rprqrr out_channelsrsrr/rvrpatch_embedderr0r1r2empty pad_tokenr mintime_caption_embed ModuleListrangelayersrnorm_out)r?rprqrrrsrtrrrrrrurrrvr@rzrCr.s<      zLuminaNextDiT2DModel.__init__rDtimesteprGrHrFrJc Cs|||\}}} }||j}||||} |}|jD] } | |||||| |d}q||| }|j} } | d\}}|d}|| || }|ddd|f ||| || | | |j }| dddddd dd dd}|sx|fSt |d S) a Forward pass of LuminaNextDiT. Parameters: hidden_states (torch.Tensor): Input tensor of shape (N, C, H, W). timestep (torch.Tensor): Tensor of diffusion timesteps of shape (N,). encoder_hidden_states (torch.Tensor): Tensor of caption features of shape (N, D). encoder_mask (torch.Tensor): Tensor of caption masks of shape (N, L). )rIrJrNrMrr r))sample)rtodevicerrerrrqsizerQr~permuterRr)r?rDrrGrHrFrJ return_dictmaskimg_sizerIlayer height_tokens width_tokensheightwidth batch_sizesequence_lengthoutputrBrBrCr^!s4     $ zLuminaNextDiT2DModel.forward)rjr r)rkrlrlNrmNr TTrnro)NT)r_r`rarbrrcrrdrer.r2rfrrgrr^rhrBrBr@rCris-     G  ri)"typingrrrr2torch.nnr0configuration_utilsrrutilsr attentionr attention_processorr r embeddingsr rmodeling_outputsrmodeling_utilsr normalizationrrr get_loggerr_loggerModulerrirBrBrBrCs