o GiZn@sddlmZddlZddlmmZddlmZddlmZm Z ddl m Z m Z ddl mZdd lmZmZmZdd lmZdd lmZdd lmZe eZGd ddeZGdddeeZdS))AnyN)nn)LegacyConfigMixinregister_to_config) deprecatelogging)BasicTransformerBlock)ImagePositionalEmbeddings PatchEmbedPixArtAlphaTextProjection)Transformer2DModelOutput)LegacyModelMixin)AdaLayerNormSinglecseZdZfddZZS)rcs&d}tdd|tj|i|dS)NzImporting `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead.r1.0.0)rsuper__init__)selfargskwargsdeprecation_message __class__`/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/transformer_2d.pyr!s z!Transformer2DModelOutput.__init__)__name__ __module__ __qualname__r __classcell__rrrrr src5seZdZdZdZdgZddgZe                dGdedededBdedBdede dededBde dedBdedBdedBde dedBde d e d!e d"e d#e d$e d%e d&e d'ed(e d)e dBf2fd*d+ Z d,d-Z d.d/Zd0d1Z        dHd2ejd3ejdBd4ejdBd5ee ejfd6ejdBd7ee efd8ejdBd9ejdBd:e fd;d<Zd=d>Zd?d@ZdAdBZdCdDZ dIdEdFZZS)JTransformer2DModela+ A 2D Transformer model for image-like data. Parameters: num_attention_heads (`int`, *optional*, defaults to 16): The number of heads to use for multi-head attention. attention_head_dim (`int`, *optional*, defaults to 88): The number of channels in each head. in_channels (`int`, *optional*): The number of channels in the input and output (specify if the input is **continuous**). num_layers (`int`, *optional*, defaults to 1): The number of layers of Transformer blocks to use. dropout (`float`, *optional*, defaults to 0.0): The dropout probability to use. cross_attention_dim (`int`, *optional*): The number of `encoder_hidden_states` dimensions to use. sample_size (`int`, *optional*): The width of the latent images (specify if the input is **discrete**). This is fixed during training since it is used to learn a number of position embeddings. num_vector_embeds (`int`, *optional*): The number of classes of the vector embeddings of the latent pixels (specify if the input is **discrete**). Includes the class for the masked latent pixel. activation_fn (`str`, *optional*, defaults to `"geglu"`): Activation function to use in feed-forward. num_embeds_ada_norm ( `int`, *optional*): The number of diffusion steps used during training. Pass if at least one of the norm_layers is `AdaLayerNorm`. This is fixed during training since it is used to learn a number of embeddings that are added to the hidden states. During inference, you can denoise for up to but not more steps than `num_embeds_ada_norm`. attention_bias (`bool`, *optional*): Configure if the `TransformerBlocks` attention should contain a bias parameter. Tr latent_image_embeddingnormXN Fgeglu layer_normh㈵>defaultnum_attention_headsattention_head_dim in_channels out_channels num_layersdropoutnorm_num_groupscross_attention_dimattention_bias sample_sizenum_vector_embeds patch_size activation_fnnum_embeds_ada_normuse_linear_projectiononly_cross_attentiondouble_self_attentionupcast_attention norm_typenorm_elementwise_affinenorm_epsattention_typecaption_channelsinterpolation_scaleuse_additional_conditionscst| dur%|dvrtd|d|dvr%|dur%td|d|duo,| du|_| du|_|duo:| du|_|jrM|jrMtd|d| d |jr^|jr^td | d | d |jsu|jsu|jsutd |d| d| d|dkr|durd|jd}tdd|ddd}||_ ||_ ||_ ||_ ||_ |jj |jj |_||_|dur|n||_d|_|dur|dkr| dkrd}nd}||_|jr|j|ddS|jr|j|ddS|jr|j|ddSdS)N)ada_norm ada_norm_zeroada_norm_singlezRForward pass is not implemented when `patch_size` is not None and `norm_type` is 'z'.)rErFz0When using a `patch_size` and this `norm_type` (z(), `num_embeds_ada_norm` cannot be None.z"Cannot define both `in_channels`: z and `num_vector_embeds`: zE. Make sure that either `in_channels` or `num_vector_embeds` is None.z(Cannot define both `num_vector_embeds`: z and `patch_size`: zE. Make sure that either `num_vector_embeds` or `num_patches` is None.zHas to define `in_channels`: z, `num_vector_embeds`: z, or patch_size: zQ. Make sure that `in_channels`, `num_vector_embeds` or `num_patches` is not None.r)z&The configuration file of this model: a is outdated. `norm_type` is either not set or incorrectly set to `'layer_norm'`. Make sure to set `norm_type` to `'ada_norm'` in the config. Please make sure to update the config accordingly as leaving `norm_type` might led to incorrect results in future versions. If you have downloaded this checkpoint from the Hugging Face Hub, it would be very nice if you could open a Pull request for the `transformer/config.json` fileznorm_type!=num_embeds_ada_normrF) standard_warnrErGT)r>)rrNotImplementedError ValueErroris_input_continuousis_input_vectorizedis_input_patchesrrr:rCrBr,r-config inner_dimr.r/gradient_checkpointingrD_init_continuous_input_init_vectorized_inputs_init_patched_inputs)rr,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrrrrrGsj       zTransformer2DModel.__init__cstjjjjjddd_jrtjjj _ ntjj jj dddd_ t fddt jjD_jrKtjj j_dStjj j jdddd_dS) Nư>T) num_groups num_channelsepsaffiner%r) kernel_sizestridepaddingchg|]0}tjjjjjjjjjjjjjjj jj jj jj jj jjjjdqS) r1r3r8r9r4r;r<r=r>r?r@rAr rPrOr,r-r1r3r8r9r4r;r<r=r?r@rA.0_r>rrr (z=Transformer2DModel._init_continuous_input..)torchr GroupNormrOr2r.r"r:LinearrPproj_inConv2d ModuleListranger0transformer_blocksr/proj_outrr>rrcrrRs   z)Transformer2DModel._init_continuous_inputcsjjdus JdjjdusJdjj_jj_jj_tjjjjjd_t fddt jj D_ t j_t jjjd_dS)Nz?Transformer2DModel over discrete input must provide sample_sizez=Transformer2DModel over discrete input must provide num_embed) num_embed embed_dimheightwidthcr]r^r_r`rcrrrdrez>Transformer2DModel._init_vectorized_inputs..r%)rOr5r6rrrsnum_latent_pixelsr rPr!rrkrlr0rm LayerNormnorm_outrhoutrorrcrrSs"    z*Transformer2DModel._init_vectorized_inputscsjjdus Jdjj_jj_jj_jjdur#jjntjjdd}tjjjjjjjj |d_ t fddt jjD_jjdkr~t jj dd d _t j d j _t j jjjjj_n1jjdkrt jj dd d _t td j j d _t j jjjjj_d_jjdkrtj jd _d_j durt!j j d_dSdS)Nz>Transformer2DModel over patched input must provide sample_size@r%)rrrsr7r.rqrCcr]r^r_r`rcrrrdrez;Transformer2DModel._init_patched_inputs..rGFrU)elementwise_affinerXr ?)rD) in_features hidden_size)"rOr5rrrsr7rCmaxr r.rP pos_embedrrkrlr0rmr>rurvrh proj_out_1r/ proj_out_2 Parameterrfrandnscale_shift_tablern adaln_singlerrDcaption_projectionrBr )rr>rCrrcrrTsX            z'Transformer2DModel._init_patched_inputs hidden_statesencoder_hidden_statestimestepadded_cond_kwargs class_labelscross_attention_kwargsattention_maskencoder_attention_mask return_dictc  Cs|dur|dddurtd|dur)|jdkr)d||jd}|d}|durA|jdkrAd||jd}|d}|jrU|j\} } } } |}| |\}}n)|j r^| |}n |j r~|jd|j |jd|j } } |||||\}}}}|jD]!}tr|jr|||||||||}q||||||||d }q|jr|j||| | | |d }n|j r||}n|j r|j||||| | d }| s|fSt|d S) ar The [`Transformer2DModel`] forward method. Args: hidden_states (`torch.LongTensor` of shape `(batch size, num latent pixels)` if discrete, `torch.Tensor` of shape `(batch size, channel, height, width)` if continuous): Input `hidden_states`. encoder_hidden_states ( `torch.Tensor` of shape `(batch size, sequence len, embed dims)`, *optional*): Conditional embeddings for cross attention layer. If not given, cross-attention defaults to self-attention. timestep ( `torch.LongTensor`, *optional*): Used to indicate denoising step. Optional timestep to be applied as an embedding in `AdaLayerNorm`. class_labels ( `torch.LongTensor` of shape `(batch size, num classes)`, *optional*): Used to indicate class labels conditioning. Optional class labels to be applied as an embedding in `AdaLayerZeroNorm`. cross_attention_kwargs ( `dict[str, Any]`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). attention_mask ( `torch.Tensor`, *optional*): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. encoder_attention_mask ( `torch.Tensor`, *optional*): Cross-attention mask applied to `encoder_hidden_states`. Two formats supported: * Mask `(batch, sequence_length)` True = keep, False = discard. * Bias `(batch, 1, sequence_length)` 0 = keep, -10000 = discard. If `ndim == 2`: will be interpreted as a mask, then converted into a bias consistent with the format above. This bias will be added to the cross-attention scores. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.unets.unet_2d_condition.UNet2DConditionOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformers.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. NscalezSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r r%g)rrrrrr)rresidual batch_sizerrrsrP)rrrembedded_timesteprrrs)sample)getloggerwarningndimtodtype unsqueezerLshape_operate_on_continuous_inputsrMr!rNr7_operate_on_patched_inputsrmrfis_grad_enabledrQ_gradient_checkpointing_func!_get_output_for_continuous_inputs!_get_output_for_vectorized_inputs_get_output_for_patched_inputsr)rrrrrrrrrrrrbrrrsrrPrblockoutputrrrforwardDs2     "      zTransformer2DModel.forwardcCs|j\}}}}||}|js,||}|jd}|dddd||||}||fS|jd}|dddd||||}||}||fS)Nr%rr r)rr"r:ripermutereshape)rrbatchrbrrrsrPrrrrs     z0Transformer2DModel._operate_on_continuous_inputscCs|jd}||}d}|jdur(|jr|durtd|j||||jd\}}|jdur<||}||d|jd}||||fS)NrzW`added_cond_kwargs` cannot be None when using additional conditions for `adaln_single`.)r hidden_dtyper)rr~rrDrKrrview)rrrrrrrrrrrs       z-Transformer2DModel._operate_on_patched_inputscCsh|js|||||dddd}||}n||}|||||dddd}||}|S)Nrrr%r )r:rr contiguousrn)rrrrrrrsrPrrrrrs  z4Transformer2DModel._get_output_for_continuous_inputscCs<||}||}|ddd}tj|dd}|S)Nrr r%dim)rvrwrF log_softmaxdoublefloat)rrlogitsrrrrrs  z4Transformer2DModel._get_output_for_vectorized_inputsc CsP|jjdkr?|jdjj|||jd}|t|j ddd\}} | |d| dddf|dddf}| |}n1|jjdkrp|j d|dddfj ddd\}} | |}|d| |}| |}|d}|jdurt|jdd}}|jd|||j|j|jfd }td |}|jd|j||j||jfd } | S) NrGr)rr r%rrzr)rznhwpqc->nchpwq)rOr>rmnorm1embrrrsiluchunkrvrrrnsqueezerintrrr7r/rfeinsum) rrrrrrrrs conditioningshiftrrrrrr s.  .  (     z1Transformer2DModel._get_output_for_patched_inputs)r#r$NNr%r&r'NFNNNr(NFFFFr)Tr*r+NNN)NNNNNNNT)NN)rrr__doc__ _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patternsrrrboolstrrrRrSrTrfTensor LongTensordictrrrrrrrrrrrrr 's     g%(K      r )typingrrftorch.nn.functionalr functionalrconfiguration_utilsrrutilsrr attentionr embeddingsr r r modeling_outputsrmodeling_utilsr normalizationr get_loggerrrr rrrrs