o ۷i"j@sddlZddlmZddlZddlZddlmZddlm Z m Z ddl m Z ddl mZddlmZdd lmZGd d d ejZGd d d ejZGdddejZGdddejZGdddejZGdddejZeGdddeZGdddee e ZdS)N) dataclass) ConfigMixinregister_to_config)FromOriginalModelMixin) BaseOutput) Attention) ModelMixincs(eZdZfddZfddZZS)SDCascadeLayerNormcstj|i|dSN)super__init__)selfargskwargs __class__`/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/unets/unet_stable_cascade.pyrszSDCascadeLayerNorm.__init__cs,|dddd}t|}|ddddS)Nrrr)permuter forward)rxrrrr"s zSDCascadeLayerNorm.forward__name__ __module__ __qualname__rr __classcell__rrrrr s r cs(eZdZgffdd ZddZZS)SDCascadeTimestepBlockc sNtt||d|_||_|D]}t|d|t||dqdS)Nrmapper_)r rnnLinearmappercondssetattr)rc c_timestepr$cnamerrrr)s  zSDCascadeTimestepBlock.__init__c Cs|jt|jddd}||dddddddfjddd\}}t|jD],\}}t|d|||dddddddfjddd\}}||||}}q)|d||S)Nrdimrrr )chunklenr$r# enumerategetattr) rrtabir&acbcrrrr1s 0>zSDCascadeTimestepBlock.forwardrrrrrr(srcs(eZdZd fdd Zd ddZZS) SDCascadeResBlockrrc szttj||||d|d|_t|ddd|_tt|||dt t |dt |t|d||_ dS)Nr) kernel_sizepaddinggroupsFư>elementwise_affineeps) r rr!Conv2d depthwiser norm Sequentialr"GELUGlobalResponseNormDropout channelwise)rr&c_skipr7dropoutrrrr;s   zSDCascadeResBlock.__init__NcCsX|}|||}|durtj||gdd}||dddddddd}||S)Nrr)rrr)rAr@torchcatrFr)rrx_skipx_resrrrrGs "zSDCascadeResBlock.forward)rrr6r rrrrrr5:s r5cs$eZdZfddZddZZS)rDcs>tttddd||_ttddd||_dS)Nr)r rr! ParameterrIzerosgammabeta)rr*rrrrRs zGlobalResponseNorm.__init__cCs@tj|dddd}||jdddd}|j|||j|S)Nr)rrT)pr*keepdim)r*rRr:)rIrAmeanrOrP)rragg_normstand_div_normrrrrWszGlobalResponseNorm.forwardrrrrrrDQs rDcs&eZdZdfdd ZddZZS)SDCascadeAttnBlockTr6csVt||_t|ddd|_t|||||dd|_tt t |||_ dS)NFr:r;T) query_dimheadsdim_headrHbias) r r self_attnr rAr attentionr!rBSiLUr" kv_mapper)rr&c_condnheadr\rHrrrr^s zSDCascadeAttnBlock.__init__cCsb||}||}|jr&|j\}}}}tj|||ddd|gdd}||j||d}|S)NrSrrr))encoder_hidden_states) r_rAr\shaperIrJview transposer])rrkvnorm_x batch_sizechannel_rrrrfs  $zSDCascadeAttnBlock.forward)Tr6rrrrrrW]srWcs&eZdZdfdd ZddZZS) UpDownBlock2dTcs|t|dvrt|d|r tj|dkrdnddddnt}tj||d d }t|dkr6||gn||g|_dS) N)updown not supportedrlrg?bilinearT) scale_factormode align_cornersrr7) r r ValueErrorr!UpsampleIdentityr? ModuleListblocks)r in_channels out_channelsrqenabled interpolationmappingrrrrqs $zUpDownBlock2d.__init__cCs|jD]}||}q|Sr )rx)rrblockrrrr}s  zUpDownBlock2d.forward)Trrrrrrkps rkc@seZdZUdZejed<dS)StableCascadeUNetOutputNsample)rrrrrITensor__annotations__rrrrrs rc+sReZdZdZe              d6dedededededeedfdeedfdeedfdeedfdeed Bdeed Bdeeeded Bd ed Bd!ed Bd"ed Bd#eeeBd$e ee Bd%eedfd&ee d Bf(fd'd( Z d)d*Z d7d,d-Z d8d.d/Z d0d1Zd2d3Z d9d4d5ZZS):StableCascadeUNetT@rrr rrrrrr5rrWrNr>r皙?rscacrpryrztimestep_ratio_embedding_dim patch_sizeconditioning_dimblock_out_channels.num_attention_headsdown_num_layers_per_blockup_num_layers_per_blockdown_blocks_repeat_mappersup_blocks_repeat_mappersblock_types_per_layerclip_text_in_channelsclip_image_in_channelseffnet_in_channelspixel_mapper_in_channelsrHr\timestep_conditioning_type switch_levelc# stt|t|krtdt|t|t| kr'tdt|t|t| kr8tdt|t|t| krItdt|t|t| krZtdt|t|trf|ft|}t|trr|ft|}|durttj ||ddd d t tj |dd|dd d t |dd d d |_ |durttj ||ddd d t tj |dd|dd d t |dd d d |_ t|||_| durt| |_|durt|||_tjd d d |_tt|tj ||d|dd d t |dd d d |_dfdd }t|_t|_t|_tt|D]}|dkrp|jtt ||d d d d |dur^t||d ||d||d dntj ||d ||dddn|jtt}t||D]!}| |D]}||||||||||d}||qq|j|| durt}t| |d D]}|tj ||||d d q|j|q1t|_t|_t|_ t!tt|D]}|dkr*|jtt ||d d d |durt||||d d||d dntj"||||d dddn|jtt}t| ddd|D]C} t#| |D]9\}!}|t|d krh| |!krbdkrhnn||nd}"|||||||"||||d}||qIqA|j|| durt}t| ddd|d D]}|tj ||||d d q|j |qtt |dd d d tj |d||dd d t$||_%d |_&dS)a Parameters: in_channels (`int`, defaults to 16): Number of channels in the input sample. out_channels (`int`, defaults to 16): Number of channels in the output sample. timestep_ratio_embedding_dim (`int`, defaults to 64): Dimension of the projected time embedding. patch_size (`int`, defaults to 1): Patch size to use for pixel unshuffling layer conditioning_dim (`int`, defaults to 2048): Dimension of the image and text conditional embedding. block_out_channels (tuple[int], defaults to (2048, 2048)): tuple of output channels for each block. num_attention_heads (tuple[int], defaults to (32, 32)): Number of attention heads in each attention block. Set to -1 to if block types in a layer do not have attention. down_num_layers_per_block (tuple[int], defaults to [8, 24]): Number of layers in each down block. up_num_layers_per_block (tuple[int], defaults to [24, 8]): Number of layers in each up block. down_blocks_repeat_mappers (tuple[int], optional, defaults to [1, 1]): Number of 1x1 Convolutional layers to repeat in each down block. up_blocks_repeat_mappers (tuple[int], optional, defaults to [1, 1]): Number of 1x1 Convolutional layers to repeat in each up block. block_types_per_layer (tuple[tuple[str]], optional, defaults to ( ("SDCascadeResBlock", "SDCascadeTimestepBlock", "SDCascadeAttnBlock"), ("SDCascadeResBlock", "SDCascadeTimestepBlock", "SDCascadeAttnBlock") ): Block types used in each layer of the up/down blocks. clip_text_in_channels (`int`, *optional*, defaults to `None`): Number of input channels for CLIP based text conditioning. clip_text_pooled_in_channels (`int`, *optional*, defaults to 1280): Number of input channels for pooled CLIP text embeddings. clip_image_in_channels (`int`, *optional*): Number of input channels for CLIP based image conditioning. clip_seq (`int`, *optional*, defaults to 4): effnet_in_channels (`int`, *optional*, defaults to `None`): Number of input channels for effnet conditioning. pixel_mapper_in_channels (`int`, defaults to `None`): Number of input channels for pixel mapper conditioning. kernel_size (`int`, *optional*, defaults to 3): Kernel size to use in the block convolutional layers. dropout (tuple[float], *optional*, defaults to (0.1, 0.1)): Dropout to use per block. self_attn (bool | tuple[bool]): tuple of booleans that determine whether to use self attention in a block or not. timestep_conditioning_type (tuple[str], defaults to ("sca", "crp")): Timestep conditioning type. switch_level (tuple[bool] | None, *optional*, defaults to `None`): tuple that indicates whether upsampling or downsampling should be applied in a block zaNumber of elements in `down_num_layers_per_block` must match the length of `block_out_channels`: z_Number of elements in `up_num_layers_per_block` must match the length of `block_out_channels`: zbNumber of elements in `down_blocks_repeat_mappers` must match the length of `block_out_channels`: z`Number of elements in `up_blocks_repeat_mappers` must match the length of `block_out_channels`: z]Number of elements in `block_types_per_layer` must match the length of `block_out_channels`: Nrr>rrsFr:r;rTcsX|dkr t|||dS|dkrt||||dS|dkr$t|dStd|d) Nr5)r7rHrW)r\rHr)r$z Block type rn)r5rWrrt) block_typeryrarGrHr\rr7rrrr get_blocksz-StableCascadeUNet.__init__..get_blockrm)rqr{)r7stride)rHr\rlrS)rGrHr\)rrT)'r rr,rt isinstancefloatboolr!rBr?rCr effnet_mapper pixels_mapperr"clip_txt_pooled_mapperclip_txt_mapperclip_img_mapper LayerNorm clip_normPixelUnshuffle embeddingrw down_blocksdown_downscalersdown_repeat_mappersrangeappendrkrv up_blocks up_upscalersup_repeat_mappersreversedConvTranspose2dr- PixelShuffleclfgradient_checkpointing)#rryrzrrrrrrrrrrrclip_text_pooled_in_channelsrclip_seqrrr7rHr\rrrr2 down_blockrjrr~block_repeat_mappersup_blockjkrGrrrrs W               "       : "  zStableCascadeUNet.__init__c Cst|tjtjfrtjj|j|jdurtj |jdtjj |j jddt |dr7tjj |j jddndt |drHtjj |jjddndt |drgtjj |jdjddtjj |jdjddt |drtjj |jdjddtjj |jdjddtjj|jd jdtj |jd jd|j|jD]1}|D],}t|tr|jd jjtd t|jjd9_qt|trtj |jjdqqdS) Nrg{Gz?)stdrrrrrrrS)rr!r?r"rIinitxavier_uniform_weightr[ constant_normal_rhasattrrrrrrrrrr5rFdatanpsqrtsumconfigrxrr#)rm level_blockr~rrr _init_weightss4      . zStableCascadeUNet._init_weights'cCs||}|jjd}t||d}tj||jd|  }|dddf|dddf}tj | | gdd}|jjddkrQt jj|ddd}|j|jdS) Nrr)devicer))rrconstant)rq)dtype)rrmathlogrIarangerrmulexprJsincosr! functionalpadtor)rtimestep_ratio max_positionsrhalf_dimembrrrget_timestep_ratio_embeddings   z.StableCascadeUNet.get_timestep_ratio_embeddingcCst|jdkr |d}|||d|d|jjd}|durX|durX||}t|jdkr9|d}| ||d|d|jjd}t j |||gdd}n|}| |S)NrrrrSr)) r,rc unsqueezerrdsizerrrrrIrJr)rclip_txt_pooledclip_txtclip_img clip_txt_poolcliprrrget_clip_embeddingss      z%StableCascadeUNet.get_clip_embeddingsc Csg}t|j|j|j}trm|jrm|D]V\}}}||}tt|dD]>} |D]-} t | t r7| | |}q)t | t rD| | ||}q)t | t rQ| | ||}q)| | }q)| t|krc|| |}q%|d|q|S|D]O\}}}||}tt|dD]7} |D]&} t | t r| |}qt | t r| ||}qt | t r| ||}q| |}q| t|kr|| |}q|d|qo|S)Nrr)ziprrrrIis_grad_enabledrrr,rr5_gradient_checkpointing_funcrWrinsert) rrr_embedr level_outputs block_groupr downscalerrepmapr2r~rrr _down_encodesF               zStableCascadeUNet._down_encodec Cs|d}t|j|j|j}tr|jrt|D]\}\}}} tt | dD]} t|D]q\} } t | t r~| dkrB|dkrB||nd} | durv| d| dks\| d| dkrv|j }tjjj|| jddddd}||}|| || }q-t | tr|| ||}q-t | tr|| ||}q-|| |}q-| t | kr| | |}q'||}q|St|D]\}\}}} tt | dD]} t|D]n\} } t | t r| dkr|dkr||nd} | dur| d| dks| d| dkr|j }tjjj|| jddddd}||}| || }qt | tr*| ||}qt | tr6| ||}q| |}q| t | krH| | |}q||}q|S)NrrrSroT)rqrr)rrrrrIrrr-rr,rr5rrr!r interpolaterrcrrrWr)rrrrrrr2rupscalerrrrr~skip orig_typerrr _up_decodes^ 0       4         zStableCascadeUNet._up_decodec  Cs<|dur||dddd}||} |jjD]%} | dkr!|} n | dkr(| } nd} | p0t|}tj| ||gdd} q|j|||d}| |}t |d rh|durh|| t j j||jd dd d d }t |dr|t j j|||jd dd d d }||| |}||| |}||}| s|fSt|dS)Nrrrrrrr))rrrrrroT)rrqrrr)r) new_zerosrrrrrI zeros_likerJrrrrr!rrrcrrrrr)rrrclip_text_pooled clip_textreffnetpixelsrr return_dicttimestep_ratio_embedr&condt_condrrrrrrrs6       zStableCascadeUNet.forward)rrrrrrrrrrrrNrNr>NNrrTrN)r)NN)NNNNNNT)rrr _supports_gradient_checkpointingrinttuplestrrrrrrrrrrrrrrrrs           z  &7r)r dataclassesrnumpyrrItorch.nnr!configuration_utilsrrloadersrutilsrattention_processorr modeling_utilsr rr Modulerr5rDrWrkrrrrrrs&