o ۷i8@sddlmZddlZddlmZddlmZmZddlm Z ddl m Z ddl m Z mZdd lmZmZdd lmZdd lmZdd lmZmZmZe eZGd ddejZGdddeeZdS))AnyN) ConfigMixinregister_to_config)logging)LuminaFeedForward) AttentionLuminaAttnProcessor2_0)&LuminaCombinedTimestepCaptionEmbeddingLuminaPatchEmbed)Transformer2DModelOutput) ModelMixin)LuminaLayerNormContinuousLuminaRMSNormZeroRMSNormcseZdZdZ ddedededededed ed ed ed d ffdd Z ddej dej dej dej dej dej de e e fd Bd ej fddZ ZS)LuminaNextDiTBlocka A LuminaNextDiTBlock for LuminaNextDiT2DModel. Parameters: dim (`int`): Embedding dimension of the input features. num_attention_heads (`int`): Number of attention heads. num_kv_heads (`int`): Number of attention heads in key and value features (if using GQA), or set to None for the same as query. multiple_of (`int`): The number of multiple of ffn layer. ffn_dim_multiplier (`float`): The multiplier factor of ffn layer dimension. norm_eps (`float`): The eps for norm layer. qk_norm (`bool`): normalization for query and key. cross_attention_dim (`int`): Cross attention embedding dimension of the input text prompt hidden_states. norm_elementwise_affine (`bool`, *optional*, defaults to True), Tdimnum_attention_heads num_kv_heads multiple_offfn_dim_multipliernorm_epsqk_normcross_attention_dimnorm_elementwise_affinereturnNc st|||_tt|g|_t|d|||rdnd||dddt d |_ t |j _ t|||||r:dnd||dddt d |_ t|td|d||d|_t||| d|_t||| d |_t||| d |_t||| d |_t||| d |_dS) Nlayer_norm_across_headsh㈵>F) query_dimrdim_headrheadskv_headsepsbiasout_bias processorr)r inner_dimrr) embedding_dimrr)r#elementwise_affine)super__init__head_dimnn Parametertorchzerosgater r attn1Identityto_outattn2rint feed_forwardrnorm1r ffn_norm1norm2 ffn_norm2 norm1_context) selfrrrrrrrrr __class__d/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/transformers/lumina_nextdit2d.pyr,5sV     zLuminaNextDiTBlock.__init__ hidden_statesattention_maskimage_rotary_embencoder_hidden_states encoder_masktembcross_attention_kwargscCs|}|||\} } } } |jd| | |||d|} ||}|jd| |||dd|}||jdddd}| |}|d}|jjd|}|| d| |}| | |d| d}|| d| |}|S)a Perform a forward pass through the LuminaNextDiTBlock. Parameters: hidden_states (`torch.Tensor`): The input of hidden_states for LuminaNextDiTBlock. attention_mask (`torch.Tensor): The input of hidden_states corresponse attention mask. image_rotary_emb (`torch.Tensor`): Precomputed cosine and sine frequencies. encoder_hidden_states: (`torch.Tensor`): The hidden_states of text prompt are processed by Gemma encoder. encoder_mask (`torch.Tensor`): The hidden_states of text prompt attention mask. temb (`torch.Tensor`): Timestep embedding with text prompt embedding. cross_attention_kwargs (`dict[str, Any]`): kwargs for cross attention. )rCrFrDquery_rotary_embkey_rotary_embNrrA)r9r3r=r6r2tanhviewflattenr5 unsqueezer;r8r:r<)r>rCrDrErFrGrHrIresidualnorm_hidden_statesgate_msa scale_mlpgate_mlpself_attn_outputnorm_encoder_hidden_statescross_attn_outputmixed_attn_output mlp_outputrArArBforwardvs:  zLuminaNextDiTBlock.forward)T)N)__name__ __module__ __qualname____doc__r7floatboolr,r0Tensordictstrrr] __classcell__rArAr?rBr$sT   I rc seZdZdZgdZe         d'dededBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBdedBddffdd Z  d(de j d e j d!e j d"e j d#e j d$e e efdee j eBfd%d&ZZS))LuminaNextDiT2DModelaa LuminaNextDiT: Diffusion model with a Transformer backbone. Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers. Parameters: sample_size (`int`): The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings. patch_size (`int`, *optional*, (`int`, *optional*, defaults to 2): The size of each patch in the image. This parameter defines the resolution of patches fed into the model. in_channels (`int`, *optional*, defaults to 4): The number of input channels for the model. Typically, this matches the number of channels in the input images. hidden_size (`int`, *optional*, defaults to 4096): The dimensionality of the hidden layers in the model. This parameter determines the width of the model's hidden representations. num_layers (`int`, *optional*, default to 32): The number of layers in the model. This defines the depth of the neural network. num_attention_heads (`int`, *optional*, defaults to 32): The number of attention heads in each attention layer. This parameter specifies how many separate attention mechanisms are used. num_kv_heads (`int`, *optional*, defaults to 8): The number of key-value heads in the attention mechanism, if different from the number of attention heads. If None, it defaults to num_attention_heads. multiple_of (`int`, *optional*, defaults to 256): A factor that the hidden size should be a multiple of. This can help optimize certain hardware configurations. ffn_dim_multiplier (`float`, *optional*): A multiplier for the dimensionality of the feed-forward network. If None, it uses a default value based on the model configuration. norm_eps (`float`, *optional*, defaults to 1e-5): A small value added to the denominator for numerical stability in normalization layers. learn_sigma (`bool`, *optional*, defaults to True): Whether the model should learn the sigma parameter, which might be related to uncertainty or variance in predictions. qk_norm (`bool`, *optional*, defaults to True): Indicates if the queries and keys in the attention mechanism should be normalized. cross_attention_dim (`int`, *optional*, defaults to 2048): The dimensionality of the text embeddings. This parameter defines the size of the text representations used in the model. scaling_factor (`float`, *optional*, defaults to 1.0): A scaling factor applied to certain parameters or layers in the model. This can be used for adjusting the overall scale of the model's operations. )patch_embeddernormffn_normr NrT? sample_size patch_size in_channels hidden_size num_layersrrrrr learn_sigmarrscaling_factorrc st||_||_||_| r|dn||_|_|_|_||_ t ||dd|_ t t|_ttdd|_t fddt|D|_ttddd d|||jd |_d d ksvJd dS)NrT)rtru embed_dimr$i)rvrc s"g|] }tqSrA)r).0_rrrvrrrrrrArB s z1LuminaNextDiT2DModel.__init__..Fgư>)r)conditioning_embedding_dimr*r#r$out_dimrmrz+2d rope needs head dim to be divisible by 4)r+r,rsrtru out_channelsrvrr-ryr rir.r/r0empty pad_tokenr mintime_caption_embed ModuleListrangelayersrnorm_out)r>rsrtrurvrwrrrrrrxrrryr?r}rBr,s<      zLuminaNextDiT2DModel.__init__rCtimesteprFrGrErIc Cs|||\}}} }||j}||||} |}|jD] } | |||||| |d}q||| }|j} } | d\}}|d}|| || }|ddd|f ||| || | | |j }| dddddd dd dd}|sx|fSt |d S) a Forward pass of LuminaNextDiT. Parameters: hidden_states (torch.Tensor): Input tensor of shape (N, C, H, W). timestep (torch.Tensor): Tensor of diffusion timesteps of shape (N,). encoder_hidden_states (torch.Tensor): Tensor of caption features of shape (N, D). encoder_mask (torch.Tensor): Tensor of caption masks of shape (N, L). )rHrIrNrLrrrm)sample)ritodevicerrcrrrtsizerPrpermuterQr )r>rCrrFrGrErI return_dictmaskimg_sizerHlayer height_tokens width_tokensheightwidth batch_sizesequence_lengthoutputrArArBr]#s4     $ zLuminaNextDiT2DModel.forward)rlrrmrnroroNrpNrTTrqrr)NT)r^r_r`ra _skip_layerwise_casting_patternsrr7rbrcr,r0rdrerfrtupler r]rgrArAr?rBrhs-     G  rh) typingrr0torch.nnr.configuration_utilsrrutilsr attentionrattention_processorr r embeddingsr r modeling_outputsr modeling_utilsr normalizationrrr get_loggerr^loggerModulerrhrArArArBs