o piC@s&ddlmZmZddlZddlmmZddlmZddlm Z ddl m Z m Z ddl mZdd lmZmZdd lmZmZmZmZmZdd lmZmZdd lmZdd lmZmZddl m!Z!m"Z"Gdddee eZ#Gdddej$Z%Gdddej$Z&Gdddej$Z'Gdddej$Z(dS))DictUnionN)nn checkpoint) ConfigMixinregister_to_config)PeftAdapterMixin)BasicTransformerBlockSkipFFTransformerBlock)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORSAttentionProcessorAttnAddedKVProcessor AttnProcessor)TimestepEmbeddingget_timestep_embedding) ModelMixin)GlobalResponseNormRMSNorm) Downsample2D Upsample2Dc.s eZdZdZe                 d7dededededededededededededededed ed!ed"ed#ed$ed%ed&ef,fd'd( Zd8d)ed*d+fd,d-Z d9d.d/Z e d*e e effd0d1Zd2eee e efffd3d4Zd5d6ZZS): UVit2DModelTF@ r  ư>@ hidden_sizeuse_biashidden_dropoutcond_embed_dimmicro_cond_encode_dimmicro_cond_embed_dimencoder_hidden_size vocab_size codebook_size in_channelsblock_out_channelsnum_res_blocks downsampleupsampleblock_num_headsnum_hidden_layersnum_attention_headsattention_dropoutintermediate_sizelayer_norm_epsln_elementwise_affine sample_sizec sttj|d|_t|_t| | ||_t ||d|_ t | | ||| d |_ t| |_ tj| d|_tfddt|D|_t|_tj| d|_t | | ||d|d |_t| | | |_d|_dS)Nbias)sample_proj_biasFcs4g|]}tddqS)ada_norm_continuous)dimr8attention_head_dimdropoutcross_attention_dimattention_bias norm_type-ada_norm_continous_conditioning_embedding_dimnorm_elementwise_affinenorm_eps ada_norm_bias ff_inner_dimff_biasattention_out_bias)r .0_r*r(r:r;r<r8r)\/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/unets/uvit_2d.py ks&z(UVit2DModel.__init__..)r4r5)super__init__rLinear encoder_projrencoder_proj_layer_normUVit2DConvEmbedembedr cond_embed UVitBlock down_blockproject_to_hidden_normproject_to_hidden ModuleListrangetransformer_layersproject_from_hidden_normproject_from_hiddenup_block ConvMlmLayer mlm_layergradient_checkpointing)selfr(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r= __class__rRrTrW*s` "    zUVit2DModel.__init__valuereturnNcCsdSNrS)rkmodulernrSrSrT_set_gradient_checkpointingsz'UVit2DModel._set_gradient_checkpointingcs\||}||}t||jjddd}||jddf}tj ||gdd}|j |j d}| | |j }| |}|j||||d}|j\}} } } |dd d d|| | | }||}||}|jD]|jry|jryfd d } n} | |||d |id}qj||}||}||| | | dd dd }|j||||d}||} | S)NTr)flip_sin_to_cosdownscale_freq_shiftrB)dtype)pooled_text_embencoder_hidden_statescross_attention_kwargsr rcstg|RSrpr)argslayerrSrTlayer_sz#UVit2DModel.forward..layer_ry)rzr{added_cond_kwargs)rYrZrflattenconfigr,reshapeshapetorchcattorxr]r\r_permuter`rardtrainingrjrerfrgri)rk input_idsrzry micro_condsr{micro_cond_embeds hidden_states batch_sizechannelsheightwidthrlogitsrSr}rTforwardsR          zUVit2DModel.forwardcsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namerq processorscsHt|dr|||d<|D]\}}|d|||q|S)N get_processor .processor.)hasattrrnamed_children)rrqrsub_namechildfn_recursive_add_processorsrSrTrs z@UVit2DModel.attn_processors..fn_recursive_add_processors)strrrModulerrr)rkrrrqrSrrTattn_processorss  & zUVit2DModel.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rrqcsbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorrr)r isinstancedictrpopr)rrqrrrfn_recursive_attn_processorrSrTrs   zCUVit2DModel.set_attn_processor..fn_recursive_attn_processorN) lenrkeysrr ValueErrorrrrrr)rkrcountrrqrSrrTset_attn_processors   zUVit2DModel.set_attn_processorcCsjtdd|jDrt}ntdd|jDr t}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. cs|]}|jtvVqdSrp)rmrrPprocrSrSrT z9UVit2DModel.set_default_attn_processor..csrrp)rmrrrSrSrTrrzOCannot call `set_default_attn_processor` when attention processors are of type N) allrvaluesrrrnextiterr)rkrrSrSrTset_default_attn_processorsz&UVit2DModel.set_default_attn_processor)rFrrrrrr r!rrrFFr"r#r$rr%r&Tr')Frp)__name__ __module__ __qualname__ _supports_gradient_checkpointingr intboolfloatrWrrrpropertyrrrrrrr __classcell__rSrSrlrTr's    l =#rcs$eZdZfddZddZZS)r[cs>tt|||_t||||_tj||d|d|_dSNrv) kernel_sizer?) rVrWr Embedding embeddingsr layer_normConv2dconv)rkr1r2r/elementwise_affineepsr?rlrSrTrW%s zUVit2DConvEmbed.__init__cCs2||}||}|dddd}||}|S)Nrrrvr )rrrr)rkrrrSrSrTr+s   zUVit2DConvEmbed.forwardrrrrWrrrSrSrlrTr[$s r[cs2eZdZdededeffdd ZddZZS)r^r3r4r5c st| rtdddddd |_nd|_tfddt|D|_tfd dt|D|_| rSt dddd dd d |_ dSd|_ dS) NTrConv2d_0r rms_norm)use_convpaddingrrrGrrr?c sg|] }tqSrS) ConvNextBlock)rPi)rr*r(r;r<r)rSrTrUTs z&UVitBlock.__init__..c s*g|]}td qS))rFrN)r rO)r9r6rr(r)rSrTrUbs rF) use_conv_transposerrrrGrrr? interpolate) rVrWrr4rrbrc res_blocksattention_blocksrr5) rkrr3r(r*r<r;r)r6r9r4r5rl)r9r6rr*r(r;r<r)rTrW4sL      zUVitBlock.__init__c Cs|jdur ||}t|j|jD]2\}}|||}|j\}}} } |||| | ddd}||||d}|ddd||| | }q|jdurN||}|S)Nrr rv)rzr{)r4ziprrrviewrr5) rkxryrzr{ res_blockattention_blockrrrrrSrSrTrs     zUVitBlock.forward)rrrrrrWrrrSrSrlrTr^3s  Nr^cs(eZdZ dfdd ZddZZS)rcsttj||dd||d|_t||||_tj|t|||d|_ t |_ t t|||_ tjt||||d|_t||_t||d||_dS)Nrrv)rrgroupsr?r>r )rVrWrr depthwisernormrXrchannelwise_linear_1GELUchannelwise_actrchannelwise_normchannelwise_linear_2Dropoutchannelwise_dropoutcond_embeds_mapper)rkrr;r<r)r*r(res_ffn_factorrlrSrTrWs   zConvNextBlock.__init__cCs|}||}|dddd}||}||}||}||}||}||}|dddd}||}|t |j ddd\}}|d|ddddddf|ddddddf}|S)Nrr rrvrw) rrrrrrrrrFsiluchunk)rkr cond_embedsx_resscaleshiftrSrSrTrs       8zConvNextBlock.forward)rrrSrSrlrTrsrc s>eZdZdedededededef fdd Zd d ZZS) rhr2r1r)r<r;r0csDttj||d|d|_t||||_tj||d|d|_dSr)rVrWrrconv1rrconv2)rkr2r1r)r<r;r0rlrSrTrWs zConvMlmLayer.__init__cCs:||}||dddddddd}||}|S)Nrr rrv)rrrr)rkrrrSrSrTrs " zConvMlmLayer.forward) rrrrrrrWrrrSrSrlrTrhsrh))typingrrrtorch.nn.functionalr functionalrtorch.utils.checkpointrconfiguration_utilsrr loadersr attentionr r attention_processorrrrrrrrrmodeling_utilsr normalizationrrresnetrrrrr[r^rrhrSrSrSrTs&    ~c-