o GiT@s:ddlmZddlZddlmmZddlmZddlmZm Z ddl m Z m Z ddl mZmZdd lmZdd lmZmZdd lmZmZmZmZdd lmZdd lmZddlmZm Z e!e"Z#Gdddej$Z%Gdddej$Z&Gdddej$Z'GdddZ(Gdddej$Z)Gdddeeee e Z*dS))AnyN)nn) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)apply_lora_scalelogging)AttentionMixin) AttentionSanaLinearAttnProcessor2_0) PatchEmbedPixArtAlphaTextProjectionTimestepEmbedding Timesteps)Transformer2DModelOutput) ModelMixin)AdaLayerNormSingleRMSNormc sXeZdZ   ddededededBded df fd d Zd ej d ej fd dZ Z S) GLUMBConvNT in_channels out_channels expand_ratio norm_typeresidual_connectionreturncstt||}||_||_t|_t||dddd|_ tj|d|dddd|dd|_ tj||ddddd|_ d|_ |dkrRt |d d d d |_ dSdS) Nr rr)groupsFbiasrms_normh㈵>T)epselementwise_affiner")super__init__intrrrSiLU nonlinearityConv2d conv_inverted conv_depth conv_pointnormr)selfrrrrrhidden_channels __class__b/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/transformers/sana_transformer.pyr('s   $zGLUMBConv.__init__ hidden_statescCs|jr|}||}||}||}tj|ddd\}}|||}||}|jdkr<|| dd dd}|jrC||}|S)Nr rdimr#) rr-r+r.torchchunkr/rr0movedim)r1r7residualgater5r5r6forward>s     zGLUMBConv.forward)rNT) __name__ __module__ __qualname__r)floatstrboolr(r;Tensorr@ __classcell__r5r5r3r6r&s$rcsNeZdZddededeffdd Zdejd ejd ejd ejfd d Z Z S)SanaModulatedNormFư>r9r&r%cs ttj|||d|_dS)Nr&r%)r'r(r LayerNormr0)r1r9r&r%r3r5r6r(Vs zSanaModulatedNorm.__init__r7tembscale_shift_tablercCsL||}|d|dddf|jjddd\}}|d||}|S)Nr rr8)r0todevicer<)r1r7rMrNshiftscaler5r5r6r@Zs .zSanaModulatedNorm.forward)FrJ) rArBrCr)rFrDr(r;rGr@rHr5r5r3r6rIUsrIcs:eZdZfddZd dejdejdejfddZZS) &SanaCombinedTimestepGuidanceEmbeddingscsjttdddd|_td|d|_tdddd|_td|d|_t |_ tj |d|dd|_ dS)NTr) num_channelsflip_sin_to_cosdownscale_freq_shift)rtime_embed_dimr!) r'r(r time_projrtimestep_embedderguidance_condition_projguidance_embedderrr*siluLinearlinear)r1 embedding_dimr3r5r6r(ds  z/SanaCombinedTimestepGuidanceEmbeddings.__init__Ntimestepguidance hidden_dtypec CsT||}||j|d}||}||j|d}||}||||fS)N)dtype)rZr[rOr\r]r`r^) r1rbrcrdtimesteps_proj timesteps_emb guidance_proj guidance_emb conditioningr5r5r6r@os  z.SanaCombinedTimestepGuidanceEmbeddings.forwardNN) rArBrCr(r;rGrer@rHr5r5r3r6rScs & rSc @sLeZdZdZddZ  d dedejdejdBdejdBd ejf d d ZdS) SanaAttnProcessor2_0zs Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0). cCsttds tddS)Nscaled_dot_product_attentionzTSanaAttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.)hasattrF ImportError)r1r5r5r6r(s zSanaAttnProcessor2_0.__init__Nattnr7encoder_hidden_statesattention_maskrc Csd|dur|jn|j\}}}|dur$||||}|||jd|jd}||}|dur/|}||} ||} |jdurC||}|jdurM|| } | jd} | |j} ||d|j|  dd}| |d|j|  dd} | |d|j|  dd} t j || | |ddd}| dd |d|j| }| |j}|jd|}|jd|}||j}|S)Nr:rr F) attn_mask dropout_p is_causalr)shapeprepare_attention_maskviewheadsto_qto_kto_vnorm_qnorm_k transposerormreshaperOreto_outrescale_output_factor) r1rqr7rrrs batch_sizesequence_length_querykeyvalue inner_dimhead_dimr5r5r6__call__s8            zSanaAttnProcessor2_0.__call__rk) rArBrC__doc__r(r r;rGrr5r5r5r6rlzs rlcseZdZdZ          d&d ededededed Bded Bded Bdedededededed Bdd ffdd Z d'de j de j d Bde j d Bd e j d Bd!e j d Bd"ed#ede j fd$d%Z Z S)(SanaTransformerBlockz[ Transformer block introduced in [Sana](https://huggingface.co/papers/2410.10629). F rtpTFrJ@Nr9num_attention_headsattention_head_dimdropoutnum_cross_attention_headscross_attention_head_dimcross_attention_dimattention_biasnorm_elementwise_affinenorm_epsattention_out_bias mlp_ratioqk_normrc sttj|d| d|_t|||| dur|nd| ||dtd |_|durFtj|| | d|_t|| | dur9|nd||||d| t d |_ t ||| ddd|_ t td||d|_dS) NFrK) query_dimr{dim_headkv_headsrrr"r processorT) rrrrr{rrr"out_biasr)rrrY?)r'r(rrLnorm1r rattn1norm2rlattn2rff Parameterr;randnrN)r1r9rrrrrrrrrrrrr3r5r6r(s:   zSanaTransformerBlock.__init__r7rsrrencoder_attention_maskrbheightwidthcCs|jd}|jd||ddjddd\} } } } } }||}|d| | }||j}||}|| |}|jdurK|j|||d}||}| |}|d| | }| d||f dddd}| |}| dd ddd}|||}|S) NrrYr:rr8)rrrsrr )rxrNrr<rrOrerrr unflattenpermuterflatten)r1r7rsrrrrbrrr shift_msa scale_msagate_msa shift_mlp scale_mlpgate_mlpnorm_hidden_states attn_output ff_outputr5r5r6r@s.         zSanaTransformerBlock.forward) rrrrtrrrTFrJTrN)NNNNNN)rArBrCrr)rDrFrEr(r;rG LongTensorr@rHr5r5r3r6rs     8 rc.sZeZdZdZdZgdZddgZe                d8dededBdededededBdedBdedBdede de de d ed!ed"e d#e d$edBd%e d&e d'e dBd(e d)df,fd*d+ Z e d,      d9d-ejd.ejd/ejd0ejdBd1ejdBd2ejdBd,ee efdBd3eejdBd4e d)eejd5feBfd6d7ZZS):SanaTransformer2DModelax A 2D Transformer model introduced in [Sana](https://huggingface.co/papers/2410.10629) family of models. Args: in_channels (`int`, defaults to `32`): The number of channels in the input. out_channels (`int`, *optional*, defaults to `32`): The number of channels in the output. num_attention_heads (`int`, defaults to `70`): The number of heads to use for multi-head attention. attention_head_dim (`int`, defaults to `32`): The number of channels in each head. num_layers (`int`, defaults to `20`): The number of layers of Transformer blocks to use. num_cross_attention_heads (`int`, *optional*, defaults to `20`): The number of heads to use for cross-attention. cross_attention_head_dim (`int`, *optional*, defaults to `112`): The number of channels in each head for cross-attention. cross_attention_dim (`int`, *optional*, defaults to `2240`): The number of channels in the cross-attention output. caption_channels (`int`, defaults to `2304`): The number of channels in the caption embeddings. mlp_ratio (`float`, defaults to `2.5`): The expansion ratio to use in the GLUMBConv layer. dropout (`float`, defaults to `0.0`): The dropout probability. attention_bias (`bool`, defaults to `False`): Whether to use bias in the attention layer. sample_size (`int`, defaults to `32`): The base size of the input latent. patch_size (`int`, defaults to `1`): The size of the patches to use in the patch embedding layer. norm_elementwise_affine (`bool`, defaults to `False`): Whether to use elementwise affinity in the normalization layer. norm_eps (`float`, defaults to `1e-6`): The epsilon value for the normalization layer. qk_norm (`str`, *optional*, defaults to `None`): The normalization to use for the query and key. timestep_scale (`float`, defaults to `1.0`): The scale to use for the timesteps. T)rrrI patch_embedr0rrrrr rrtFrrJN皙??rrrr num_layersrrrcaption_channelsrrr sample_size patch_sizerrinterpolation_scaleguidance_embedsguidance_embeds_scalertimestep_scalercst|p|} t| | ||||durdndd|_|r't|_nt|_t| d|_t ddd|_ t  f ddt |D|_t td d |_td d d |_t ||||_d |_dS)Nsincos)rrrr embed_dimrpos_embed_type) in_features hidden_sizer$T)r%r&cs,g|]}t   d qS)) rrrrrrrrr)r).0r rrrrrrrrrrrrr5r6 s"z3SanaTransformer2DModel.__init__..r rFrJrK)r'r(rrrS time_embedrrcaption_projectionr caption_normr ModuleListrangetransformer_blocksrr;rrNrInorm_outr_proj_outgradient_checkpointing)r1rrrrrrrrrrrrrrrrrrrrrr3rr6r(Ss4     zSanaTransformer2DModel.__init__attention_kwargsr7rrrbrcrrscontrolnet_block_samples return_dict.c  Cs4|dur|jdkrd||jd}|d}|dur0|jdkr0d||jd}|d}|j\} } } } |jj}| || |}}||}|durY|j|||jd\}}n |j|| |jd\}}| |}| | d|jd}| |}t r|jrt|jD]*\}}|||||||||}|durd|krt|krnq|||d}qn.t|jD](\}}||||||||}|durd|krt|krnq|||d}q||||j}||}|| |||jj|jjd}|dddd dd }|| d||||}| s|fSt|d S) Nr rg)rcrd)rrdr:rrr)sample)ndimrOre unsqueezerxconfigrrrrrzrr;is_grad_enabledr enumerater_gradient_checkpointing_funclenrrNrrrr)r1r7rrrbrcrrsrrrrrUrrppost_patch_heightpost_patch_widthembedded_timestep index_blockblockoutputr5r5r6r@sv      $ $   zSanaTransformer2DModel.forward)rrrrrrrrrrrtFrrFrJNFrNr)NNNNNT)rArBrCr _supports_gradient_checkpointing_no_split_modules _skip_layerwise_casting_patternsrr)rDrFrEr(r r;rGdictrtuplerr@rHr5r5r3r6r$s*     M   r)+typingrr;torch.nn.functionalr functionalroconfiguration_utilsrrloadersrrutilsr r attentionr attention_processorr r embeddingsrrrrmodeling_outputsrmodeling_utilsr normalizationrr get_loggerrAloggerModulerrIrSrlrrr5r5r5r6s(      /Bh