o Gi2@sddlZddlmZddlmZmZddlmZddlm Z m Z ddl m Z m Z ddlmZmZdd lmZdd lmZdd lmZmZeeZGd d d ejZGdddee eZdS)N) ConfigMixinregister_to_config)logging)AttentionMixin FeedForward) AttentionCogVideoXAttnProcessor2_0)&CogView3CombinedTimestepSizeEmbeddingsCogView3PlusPatchEmbed)Transformer2DModelOutput) ModelMixin)AdaLayerNormContinuous%CogView3PlusAdaLayerNormZeroTextImagec sheZdZdZ    ddededed effd d Zd ejd ejdejdeejejffddZ Z S)CogView3PlusTransformerBlocka Transformer block used in [CogView](https://github.com/THUDM/CogView3) model. Args: dim (`int`): The number of channels in the input and output. num_attention_heads (`int`): The number of heads to use for multi-head attention. attention_head_dim (`int`): The number of channels in each head. time_embed_dim (`int`): The number of channels in timestep embedding. @(dimnum_attention_headsattention_head_dimtime_embed_dimc sntt||d|_t||||ddddtd |_tj|ddd|_ tj|ddd|_ t ||d d |_ dS) N) embedding_dimrT layer_normFư>) query_dimheadsdim_headout_dimbiasqk_normelementwise_affineeps processorgh㈵>)r#r$zgelu-approximate)rdim_out activation_fn) super__init__rnorm1r r attn1nn LayerNormnorm2 norm2_contextrff)selfrrrr __class__j/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_cogview3plus.pyr)/s  z%CogView3PlusTransformerBlock.__init__ hidden_statesencoder_hidden_statesembreturnc Cs^|d}||||\ }}}}} } } } } }|j|| d\}}||d|}|| d|}||}|d|dddf|dddf}||} | d| dddf| dddf} tj| |gdd}||}|| d|dd|df}||d|ddd|f}|j tj kr| dd}|j tj kr| dd}||fS)N)r6r7)ri i) sizer*r+ unsqueezer.r/torchcatr0dtypefloat16clip)r1r6r7r8text_seq_lengthnorm_hidden_statesgate_msa shift_mlp scale_mlpgate_mlpnorm_encoder_hidden_states c_gate_msa c_shift_mlp c_scale_mlp c_gate_mlpattn_hidden_statesattn_encoder_hidden_states ff_outputr4r4r5forwardKs>    ( ( ""    z$CogView3PlusTransformerBlock.forward)rrrr) __name__ __module__ __qualname____doc__intr)r=TensortuplerP __classcell__r4r4r2r5r s0rcseZdZdZdZddgZddgZe      d'dededededededededededeffdd Z d(de j de j de j d e j d!e j d"e j d#e d$ee j eBfd%d&ZZS))CogView3PlusTransformer2DModela The Transformer model introduced in [CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion](https://huggingface.co/papers/2403.05121). Args: patch_size (`int`, defaults to `2`): The size of the patches to use in the patch embedding layer. in_channels (`int`, defaults to `16`): The number of channels in the input. num_layers (`int`, defaults to `30`): The number of layers of Transformer blocks to use. attention_head_dim (`int`, defaults to `40`): The number of channels in each head. num_attention_heads (`int`, defaults to `64`): The number of heads to use for multi-head attention. out_channels (`int`, defaults to `16`): The number of channels in the output. text_embed_dim (`int`, defaults to `4096`): Input dimension of text embeddings from the text encoder. time_embed_dim (`int`, defaults to `512`): Output dimension of timestep embeddings. condition_dim (`int`, defaults to `256`): The embedding dimension of the input SDXL-style resolution conditions (original_size, target_size, crop_coords). pos_embed_max_size (`int`, defaults to `128`): The maximum resolution of the positional embeddings, from which slices of shape `H x W` are taken and added to input patched latents, where `H` and `W` are the latent height and width respectively. A value of 128 means that the maximum supported height and width for image generation is `128 * vae_scale_factor * patch_size => 128 * 8 * 2 => 2048`. sample_size (`int`, defaults to `128`): The base resolution of input latents. If height/width is not provided during generation, this value is used to determine the resolution as `sample_size * vae_scale_factor => 128 * 8 => 1024` T patch_embednormrr rrrr patch_size in_channels num_layersrr out_channelstext_embed_dimr condition_dimpos_embed_max_size sample_sizec st|__d| _t|j||| d_t| jjd_t fddt |D_ t jddd_t jj||jd d _d_dS) N)rb hidden_sizeratext_hidden_sizerg)rrfpooled_projection_dim timesteps_dimcsg|] }tjdqS))rrrr)r inner_dim).0_rrr1rr4r5 sz;CogView3PlusTransformer2DModel.__init__..Fr)rconditioning_embedding_dimr#r$T)r!)r(r)rdrnrlr rZr time_condition_embedr, ModuleListrangetransformer_blocksrnorm_outLinearproj_outgradient_checkpointing) r1rarbrcrrrdrerrfrgrhr2rqr5r)s<     z'CogView3PlusTransformer2DModel.__init__r6r7timestep original_size target_size crop_coords return_dictr9cCs@|jdd\}} |jd} |||}||||||j} |ddd| f}|dd| df}t|jD]\} } trM|jrM| | ||| \}}q7| ||| d\}}q7| || }| |}|j j }||}| |} |j|jd|| |j||fd}td|}|j|jd|j||| |fd}|s|fSt|dS) a The [`CogView3PlusTransformer2DModel`] forward method. Args: hidden_states (`torch.Tensor`): Input `hidden_states` of shape `(batch size, channel, height, width)`. encoder_hidden_states (`torch.Tensor`): Conditional embeddings (embeddings computed from the input conditions such as prompts) of shape `(batch_size, sequence_len, text_embed_dim)` timestep (`torch.LongTensor`): Used to indicate denoising step. original_size (`torch.Tensor`): CogView3 uses SDXL-like micro-conditioning for original image size as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). target_size (`torch.Tensor`): CogView3 uses SDXL-like micro-conditioning for target image size as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). crop_coords (`torch.Tensor`): CogView3 uses SDXL-like micro-conditioning for crop coordinates as explained in section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: `torch.Tensor` or [`~models.transformer_2d.Transformer2DModelOutput`]: The denoised latents using provided inputs as conditioning. Nr:)r6r7r8r)shapeznhwcpq->nchpwq)sample)rrZrtr? enumeraterwr=is_grad_enabledr{_gradient_checkpointing_funcrxrzconfigrareshaperdeinsumr )r1r6r7r|r}r~rrheightwidthrBr8 index_blockblockraoutputr4r4r5rPsH&      z&CogView3PlusTransformer2DModel.forward) rr\r]rrr\r^rr_r`r`)T)rQrRrSrT _supports_gradient_checkpointing _skip_layerwise_casting_patterns_no_split_modulesrrUr)r=rV LongTensorboolrWr rPrXr4r4r2r5rY~st"    C rY)r=torch.nnr,configuration_utilsrrutilsr attentionrrattention_processorr r embeddingsr r modeling_outputsr modeling_utilsr normalizationrr get_loggerrQloggerModulerrYr4r4r4r5s     ^