o piQS@sddlmZddlmZddlmZmZmZmZm Z m Z ddl Z ddl Z ddl m Z mZddlmZmZddlmZmZmZdd lmZd d lmZmZmZmZmZmZmZd d l m!Z!d d l"m#Z#m$Z$d dl%m&Z&d dl'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.d dl/m0Z0e1e2Z3eGdddeZ4Gdddej5Z6Gdddej5Z7Gdddej5Z8      dDde9de9de9d e9d!e9d"ee9d#ee e9e e9fd$ee9d%ee9d&e:d'ee:d(ee:fd)d*Z;     dEd+e9d,e9d!ee9d"ee9d#e9d$ee9d%ee9d'e:d(e:fd-d.ZGd6d7d7e&eZ?Gd8d9d9ej5Z@Gd:d;d;ej5ZAGdd?ZCd@dAZDdBdCZEdS)G) dataclass)gcd)AnyDictListOptionalTupleUnionN)Tensornn) ConfigMixinregister_to_config) BaseOutputis_torch_versionlogging) apply_freeu)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttentionProcessorAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)ControlNetConditioningEmbedding)TimestepEmbedding Timesteps) ModelMixin)CrossAttnDownBlock2DCrossAttnUpBlock2D Downsample2D ResnetBlock2DTransformer2DModelUNetMidBlock2DCrossAttn Upsample2D)UNet2DConditionModelc@seZdZUdZdZeed<dS)ControlNetXSOutputa= The output of [`UNetControlNetXSModel`]. Args: sample (`Tensor` of shape `(batch_size, num_channels, height, width)`): The output of the `UNetControlNetXSModel`. Unlike `ControlNetOutput` this is NOT to be added to the base model output, but is already the final output. Nsample)__name__ __module__ __qualname____doc__r(r __annotations__r.r.\/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/controlnet_xs.pyr'4s  r'c sNeZdZdZ  d dejdejdejdeejdeejf fdd ZZ S) DownBlockControlNetXSAdapterz}Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnDownBlock2D`Nresnets base_to_ctrl ctrl_to_base attentions downsamplercs,t||_||_||_||_||_dSN)super__init__r1r2r3r4 downsamplers)selfr1r2r3r4r5 __class__r.r/r8Fs  z%DownBlockControlNetXSAdapter.__init__)NN) r)r*r+r,r ModuleListrConv2dr8 __classcell__r.r.r;r/r0Bsr0cs2eZdZdZdedejdejffdd ZZS)MidBlockControlNetXSAdapterz|Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnMidBlock2D`midblockr2r3cs t||_||_||_dSr6)r7r8rAr2r3)r:rAr2r3r;r.r/r8Zs  z$MidBlockControlNetXSAdapter.__init__) r)r*r+r,r$r r=r8r?r.r.r;r/r@Vs&r@cs(eZdZdZdejffdd ZZS)UpBlockControlNetXSAdapterzwComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnUpBlock2D`r3cst||_dSr6)r7r8r3)r:r3r;r.r/r8ds  z#UpBlockControlNetXSAdapter.__init__)r)r*r+r,r r=r8r?r.r.r;r/rBasrB TFbase_in_channelsbase_out_channelsctrl_in_channelsctrl_out_channels temb_channelsmax_norm_num_groupstransformer_layers_per_blocknum_attention_headscross_attention_dimadd_downsampleupcast_attentionuse_linear_projectionc CsZd} g}g}g}g}t|tr|g| }t| D]T}|dkr |n|}|dkr(|n|}|t|||t||||t|||dt||ddd|rd|t||||||| | | t||dd|t||q| r|t||t||d|dd }|t||nd}t t |t |t |d }|rt ||_ |dur||_ |S) Nr r max_factorh㈵> in_channels out_channelsrIgroups groups_outepsrU num_layersrMrPrOnorm_num_groupsTopuse_convrVname)r1r2r3) isinstanceintrangeappendmake_zero_convr"find_largest_factorr#r!r0r r=r4r9)rErFrGrHrIrJ has_crossattnrKrLrMrNrOrPr[r1r4r3r2ir9down_block_componentsr.r.r/get_down_block_adapterish        rj base_channels ctrl_channelsc CsPt||} t|||||tt||||||||d } t||} t| | | dS)N rKrUrVrI resnet_groupsrMrLrPrO)r2rAr3)rer$rfrr@) rkrlrIrJrKrLrMrOrPr2rAr3r.r.r/get_mid_block_adapters  rorVprev_output_channelctrl_skip_channelscCsJg}d}t|D]}|dkr|n|}|t|||qtt|dS)Nr)r3)rcrdrerBr r=)rVrprqr3r[rhresnet_in_channelsr.r.r/get_up_block_adapters  rtc seZdZdZe          d(dededeedede de eeefdeedeededeede ede eeefde dede ffdd Z e ! ! !     d)d"ed#e ede eede eede dedededeefd$d%Zd&d'ZZS)*ControlNetXSAdaptera A `ControlNetXSAdapter` model. To use it, pass it into a `UNetControlNetXSModel` (together with a `UNet2DConditionModel` base model). This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Like `UNetControlNetXSModel`, `ControlNetXSAdapter` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. Parameters: conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channels for each block in the `controlnet_cond_embedding` layer. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapters's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. learn_time_embedding (`bool`, defaults to `False`): Whether a time embedding should be learned. If yes, `UNetControlNetXSModel` will combine the time embeddings of the base model and the control adapter. If no, `UNetControlNetXSModel` will use the base model's time embedding. num_attention_heads (`list[int]`, defaults to `[4]`): The number of attention heads. block_out_channels (`list[int]`, defaults to `[4, 8, 16, 16]`): The tuple of output channels for each block. base_block_out_channels (`list[int]`, defaults to `[320, 640, 1280, 1280]`): The tuple of output channels for each block in the base unet. cross_attention_dim (`int`, defaults to 1024): The dimension of the cross attention features. down_block_types (`list[str]`, defaults to `["CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"]`): The tuple of downsample blocks to use. sample_size (`int`, defaults to 96): Height and width of input/output sample. transformer_layers_per_block (`Union[int, Tuple[int]]`, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. upcast_attention (`bool`, defaults to `True`): Whether the attention computation should always be upcasted. max_norm_num_groups (`int`, defaults to 32): Maximum number of groups in group normal. The actual number will be the largest divisor of the respective channels, that is <= max_norm_num_groups. rrrgbrC`?Fr|rxrxi@irrDrrr DownBlock2DryrTrCconditioning_channelsconditioning_channel_order#conditioning_embedding_out_channelstime_embedding_mixlearn_time_embeddingrLblock_out_channelsbase_block_out_channelsrMdown_block_types sample_sizerKrOrJrPcst|d}|dd}|dvrtd|t|t| kr-td|d| dt| ttfs;| gt| } t| ttfsI| gt| } t|ttfsW|gt| }t|t| krjtd|d| dt|d||d |_|r}t |||_ nd|_ t g|_ t g|_t jd|dd d d |_t|d|d|_|d}|d}t| D]8\}}|}||}|}||}d |v}|t| d k}|j t|||||||| |||| || | |d qt|d|d|| d|d| d| |d|_|dgt|D]\}}|t|d krd nd}|g|qtt|}|d}tt| D]}|}||}fddtd D}|jt|||dq2dS)Nrr|)rvbgrz&unknown `conditioning_channel_order`: zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: conditioning_embedding_channelsrrrrr kernel_sizepadding CrossAttn) rErFrGrHrIrJrgrKrLrMrNrOrP)rkrlrIrKrLrMrOrPr cg|]}qSr.pop.0_rqr.r/ z0ControlNetXSAdapter.__init__..)rVrprq)r7r8 ValueErrorlenralisttuplercontrolnet_cond_embeddingrtime_embeddingr r= down_blocksup_connectionsr>conv_inrecontrol_to_base_for_conv_in enumeraterdrjro mid_blockextendreversedrcrt)r:rrrrrrLrrrMrrrKrOrJrPtime_embedding_input_dimtime_embedding_dimrFrHrhdown_block_typerErGrgis_final_blockrVnumber_of_subblocks reversed_base_block_out_channelsprev_base_output_channelctrl_skip_channels_r;rr/r8$s      zControlNetXSAdapter.__init__Nunet size_ratioc s|du} du} | | Astd|pfdd|jjD}|dur%|jj}|||| |||||jj|jj|jj|jj|jj|jj|jj |jj d} | |j | S)a8 Instantiate a [`ControlNetXSAdapter`] from a [`UNet2DConditionModel`]. Parameters: unet (`UNet2DConditionModel`): The UNet model we want to control. The dimensions of the ControlNetXSAdapter will be adapted to it. size_ratio (float, *optional*, defaults to `None`): When given, block_out_channels is set to a fraction of the base model's block_out_channels. Either this or `block_out_channels` must be given. block_out_channels (`List[int]`, *optional*, defaults to `None`): Down blocks output channels in control model. Either this or `size_ratio` must be given. num_attention_heads (`List[int]`, *optional*, defaults to `None`): The dimension of the attention heads. The naming seems a bit confusing and it is, see https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131 for why. learn_time_embedding (`bool`, defaults to `False`): Whether the `ControlNetXSAdapter` should learn a time embedding. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapter's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`Tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channel for each block in the `controlnet_cond_embedding` layer. NzePass exactly one of `block_out_channels` (for absolute sizing) or `size_ratio` (for relative sizing).csg|]}t|qSr.)rb)rbrr.r/rsz1ControlNetXSAdapter.from_unet..)rrrrrrLrrrMrrrKrOrJrP) rconfigrattention_head_dimrMrrrKrOr\rPtodtype) clsrrrrLrrrrr fixed_size relative_sizemodelr.rr/ from_unets8) zControlNetXSAdapter.from_unetcOstd)NzA ControlNetXSAdapter cannot be run by itself. Use it together with a UNet2DConditionModel to instantiate a UNetControlNetXSModel.)r)r:argskwargsr.r.r/forwardszControlNetXSAdapter.forward)rrrvrwr{Fr|r}rrDrryrTrCT)NNNFr{rrrvrw)r)r*r+r,rrbstrrfloatboolr rr8 classmethodr&rrrr?r.r.r;r/rus.        Lruc/szeZdZdZdZe              dWdeedee dee deedeede eeefde eeefde eeefdee deede de deed eed!e d"ed#eed$e d%e d&eed'e eeefd(ef,fd)d* Z e dXd+ed,eed-ee d&eee d!ee d.eef d/d0ZdYd2d3ZdZd4d5Zed1ee effd6d7Zd8e eee efffd9d:Zd;d<Zd=e d>e d?e d@e fdAdBZdCdDZdEdFZdGdHZ  d[dIedJe e je efdKe jdLee jdMee dNee jdOee jdPee jdQeee e!fdReee e jfdSe dTe d1e e"effdUdVZ#Z$S)\UNetControlNetXSModela9 A UNet fused with a ControlNet-XS adapter model This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). `UNetControlNetXSModel` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. It's parameters are either passed to the underlying `UNet2DConditionModel` or used exactly like in `ControlNetXSAdapter` . See their documentation for details. Tryr UpBlock2Dr r r rrCrDrr~Nr{rrrwrvFr}r|rrup_block_typesrr\rMrKrLaddition_embed_typeaddition_time_embed_dimrOrPtime_cond_proj_dim%projection_class_embeddings_input_dimrctrl_conditioning_channels(ctrl_conditioning_embedding_out_channelsctrl_conditioning_channel_orderctrl_learn_time_embeddingctrl_block_out_channelsctrl_num_attention_headsctrl_max_norm_num_groupsc.st|dks |dkrtd|dkr|std| dur'| dkr'tdt|ttfs5|gt|}t|ttfsC|gt|}t|ttfsQ|gt|}t|ttfs_|gt|}|}d|_tj d|dddd |_ t |d||d |_ tj d|dddd |_ t|d|d|_|d}|dd}t|dd dd |_t||| d |_|rt||d|_nd|_| durd|_d|_nt| d dd |_t|||_g}|d}|d}t|D];\}}|}||}|} ||}d|v}!|t|dk}"|t||| |||||!|||||||||" | | dqt|d|d||||d|d|d|d| | d |_g}#tt|}$tt|}%tt|}&|dgt|D]\}}'|t|dkrddnd}(|'g|(qUtt|})|)d}'t|D]K\}}*|'}+|)|}'|)t|dt|d},fddtdD}-d|*v}!|t|dk}"|#t |,|'|+|-|||!|$||%||&||" | || dq~t!||_"t!|#|_#tj$|d|d|_%t&|_'tj |ddddd |_(dS)Nrrz1`time_embedding_mix` needs to be between 0 and 1.zKTo use `time_embedding_mix` < 1, `ctrl_learn_time_embedding` must be `True` text_timezAs `UNetControlNetXSModel` currently only supports StableDiffusion and StableDiffusion-XL, `addition_embed_type` must be `None` or `'text_time'`.r|rrrrT)flip_sin_to_cosdownscale_freq_shift) cond_proj_dim)rUtime_embed_dimrrErFrGrHrIr\rrgrKbase_num_attention_headsrrMrNrOrPr rkrlrIr\rrKrrrMrOrPr crr.rrrr.r/rrz2UNetControlNetXSModel.__init__..)rUrVrprqrIresolution_idxrgrKrLrM add_upsamplerOr\rP) num_channels num_groups))r7r8rrarrrrUr r> base_conv_inrr ctrl_conv_inrerrbase_time_projrbase_time_embeddingctrl_time_embeddingbase_add_time_projbase_add_embeddingrrd ControlNetXSCrossAttnDownBlock2DControlNetXSCrossAttnMidBlock2DrrrminrcControlNetXSCrossAttnUpBlock2Dr=r up_blocks GroupNormbase_conv_norm_outSiLU base_conv_act base_conv_out).r:rrrrr\rMrKrLrrrOrPrrrrrrrrrrrtime_embed_input_dimrrrFrHrhrrErGrgrr rev_transformer_layers_per_blockrev_num_attention_headsrev_cross_attention_dimrVrreversed_block_out_channels up_block_typerprUrr;rr/r8s !            zUNetControlNetXSModel.__init__r controlnetrctrl_optional_kwargsc s|durtj|||fi|}ntdd||||fDr!tdgdfdd|jD|jjd<gd fd d|jD|jjd <|i}gd }|D]} t |d |  t ||  q\ddg} | D]} t || rt || durt |d |  t ||  qt|j |j |j |j |jdur|j |j |j |j tddt|j|jD|_t|j|j|_tddt|j|jD|_||j|S)a Instantiate a [`UNetControlNetXSModel`] from a [`UNet2DConditionModel`] and an optional [`ControlNetXSAdapter`] . Parameters: unet (`UNet2DConditionModel`): The UNet model we want to control. controlnet (`ControlNetXSAdapter`): The ConntrolNet-XS adapter with which the UNet will be fused. If none is given, a new ConntrolNet-XS adapter will be created. size_ratio (float, *optional*, defaults to `None`): Used to contruct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details. ctrl_block_out_channels (`List[int]`, *optional*, defaults to `None`): Used to contruct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details, where this parameter is called `block_out_channels`. time_embedding_mix (`float`, *optional*, defaults to None): Used to contruct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details. ctrl_optional_kwargs (`Dict`, *optional*, defaults to `None`): Passed to the `init` of the new controlent if no controlent was given. Ncss|]}|duVqdSr6r.)ror.r.r/ s z2UNetControlNetXSModel.from_unet..zWhen a controlnet is passed, none of these parameters should be passed: size_ratio, ctrl_block_out_channels, time_embedding_mix, ctrl_optional_kwargs.) rrrrr\rMrKrrrOrPrrcsi|] \}}|vr||qSr.r.rkv)params_for_unetr.r/ sz3UNetControlNetXSModel.from_unet..rL)rrrrrrLrJcs"i|] \}}|vrd||qS)ctrl_r.r)params_for_controlnetr.r/rs"r)rr conv_norm_outconv_outbase_ add_time_proj add_embeddingcs |] \}}t||VqdSr6)r from_modulesrrcr.r.r/r7  csr r6)rr r r.r.r/r<r)ruranyrritemsrr from_configgetattrload_state_dict state_dicthasattrrrrrrrr r=ziprrr rrrrr) rrrrrrrrmodules_from_unetmoptional_modules_from_unetr.)rrr/rsV           zUNetControlNetXSModel.from_unetreturncsD]}d|_qgd}fdd|D}|D] }|D]}d|_qqjD]}|q)jjD]}|q8dS)Freeze the weights of the parts belonging to the base UNet2DConditionModel, and leave everything else unfrozen for fine tuning.T)rrrrrrrrcs$g|]}t|durt|qSr6)r)rpartr:r.r/rXs$z.FN) parameters requires_gradrfreeze_base_paramsrr)r:param base_partsrdur.rr/freeze_unet_paramsFs        z(UNetControlNetXSModel.freeze_unet_paramscCst|dr ||_dSdS)Ngradient_checkpointing)rr&)r:modulevaluer.r.r/_set_gradient_checkpointingcs  z1UNetControlNetXSModel._set_gradient_checkpointingcsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. r`r' processorscsHt|dr|||d<|D]\}}|d|||q|S)N get_processor .processorr)rr+named_children)r`r'r*sub_namechildfn_recursive_add_processorsr.r/r1rs zJUNetControlNetXSModel.attn_processors..fn_recursive_add_processors)rtorchr Modulerrr-)r:r*r`r'r.r0r/attn_processorsgs  & z%UNetControlNetXSModel.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.r`r'csbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorr,r)rradictr6rr-)r`r'r5r.r/fn_recursive_attn_processorr.r/r9s   zMUNetControlNetXSModel.set_attn_processor..fn_recursive_attn_processorN) rr4keysrar7rrr2r r3r-)r:r5countr`r'r.r8r/set_attn_processors   z(UNetControlNetXSModel.set_attn_processorcCsjtdd|jDrt}ntdd|jDr t}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. cs|]}|jtvVqdSr6)r<rrprocr.r.r/rzCUNetControlNetXSModel.set_default_attn_processor..csr=r6)r<rr>r.r.r/rr@zOCannot call `set_default_attn_processor` when attention processors are of type N) allr4valuesrrrnextiterr<)r:r5r.r.r/set_default_attn_processorsz0UNetControlNetXSModel.set_default_attn_processors1s2b1b2cCsHt|jD]\}}t|d|t|d|t|d|t|d|qdS)a>Enables the FreeU mechanism from https://arxiv.org/abs/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. rFrGrHrIN)rrsetattr)r:rFrGrHrIrhupsample_blockr.r.r/ enable_freeus    z"UNetControlNetXSModel.enable_freeucCsPhd}t|jD]\}}|D]}t||st||ddur$t||dqq dS)zDisables the FreeU mechanism.>rHrIrFrGN)rrrrrJ)r: freeu_keysrhrKrr.r.r/ disable_freeus z#UNetControlNetXSModel.disable_freeucCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsr4rrr<r)rmodulesrarfuse_projectionsr<r)r:rattn_processorr'r.r.r/fuse_qkv_projectionss    z*UNetControlNetXSModel.fuse_qkv_projectionscCs|jdur ||jdSdS)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rQr<rr.r.r/unfuse_qkv_projectionss z,UNetControlNetXSModel.unfuse_qkv_projectionsr(timestepencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskcross_attention_kwargsadded_cond_kwargs return_dict apply_controlc ' Cs||jjdkrtj|dgd}|dur!d||jd}|d}|} t| sO|jj dk}t |t r<|r8tj ntj }n|rAtjntj}tj| g||jd} nt| jdkr^| d|j} | |jd} || }|j|jd }|jjr| r|||}|||}|jjd }|||d|}n||}d}|jjdurnX|jjd krd | vrt|jd | d }d| vrt|jd| d}||}||jddf}tj ||gdd}||j}|!|}n td|jjd|dur||n|}|}|}}gg}}|"|}|#|}|$|}|dur)||7}| r5||%||}|&||&||j'D]} | |||||| || d\}}}!}"|(|!|(|"qB|j)|||||| || d\}}|j*D]1}#t|#j+}$||$ d}%||$ d}&|d|$ }|d|$ }|#||%|&|||| || d }qr|,|}|-|}|.|}| s|fSt/|dS)ay The [`ControlNetXSModel`] forward method. Args: sample (`Tensor`): The noisy input tensor. timestep (`Union[torch.Tensor, float, int]`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): How much the control model affects the base model outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnet.ControlNetOutput`] instead of a plain tuple. apply_control (`bool`, defaults to `True`): If `False`, the input is run only through the base model. Returns: [`~models.controlnetxs.ControlNetXSOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnetxs.ControlNetXSOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rr)dimsNgmps)rdevicer)rg333333?r text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rdimzgControlNet-XS currently only supports StableDiffusion and StableDiffusion-XL, so addition_embed_type = z is currently not supported.)hidden_states_basehidden_states_ctrltembrXrZr^r]ra) hidden_statesres_hidden_states_tuple_baseres_hidden_states_tuple_ctrlrkrXrZr^r]ra)r()0rrr2fliprr unsqueeze is_tensorrdtyperarfloat32float64int32int64tensorrshapeexpandrrrrrrrr<getrflattenreshapeconcatrrrrrrdrrrrr1rrrr')'r:r(rWrXrYrZr[r\r]r^r_r`ra timestepsis_mpsrt_emb ctrl_temb base_tembinterpolation_paramrkaug_embrerf time_embeds add_embedscembh_ctrlh_basehs_basehs_ctrl guided_hintdown residual_hb residual_hcup n_resnetsskips_hbskips_hcr.r.r/rs 7                               zUNetControlNetXSModel.forward)ryrrrrCrDrr~NNTTNNr{rrrwrvFr}r|rC)NNNNNrN)F) Nr{NNNNNTT)%r)r*r+r, _supports_gradient_checkpointingrrrbrrr rrr8rr&rurrrr%r)propertyrr4r<rErLrNrUrVr r2rr'rr?r.r.r;r/rs     ?  v #       rcs.eZdZ          d+dededed ed ed ed ed eeeeefdeedeedeededeedeeffdd Ze de de fddZ d,ddZ       d-ded ed!eed"eed#eed$eed%eeeefd&eed'edeeeeed(feed(fffd)d*ZZS).rrCTrrDFrErFrGrHrIr\rrKrrrMrNrOrPcstg}g}g}g}g}g}d}t| tr| g|} t|D]r}|dkr)|n|}|dkr1|n|}|t|||t||||d|t||||t|||dt||ddd|r|t | || || || |||d|t | || || || ||t||dd|t||q!| r|t||t |d|d d |_ t ||d|d d |_ |t||nd|_ d|_ t ||_t ||_|rt |ndg||_|rt |ndg||_t ||_t ||_d |_dS) Nr rrUrVrIrWrQrSrTrZTr]r^F)r7r8rarbrcrdrer"rfr#r!base_downsamplersctrl_downsamplersr r= base_resnets ctrl_resnetsbase_attentionsctrl_attentionsr2r3r&)r:rErFrGrHrIr\rrgrKrrrMrNrOrPrrrrr3r2r[rhr;r.r/r8s              z)ControlNetXSCrossAttnDownBlock2D.__init__base_downblockctrl_downblockcCsdd}|jdj}|jdj}|jdj|}|jdj}|jdjj}|jdjj} |jdjj} t|dr]d} t|j dj } ||j } ||j }||j }||j }|j dj}nd} d} d} d}d}d}d}|jdu}||||||| | | | | |||||d}|j|j|j|j| r|j|j |j|j |r|j|jd|j|j|j|j|j|j|S)NcS|jdjdjSNrr4transformer_blocksattn2blockr.r.r/get_first_cross_attentionBzPControlNetXSCrossAttnDownBlock2D.from_modules..get_first_cross_attentionrr4TFr)r1rUrV time_emb_proj in_featuresnorm1rrrr4rheadsrMrOrPr9rrrrrrrrr2r3)rrrrrErFrGrHrIrctrl_num_groupsrgrKrrrMrOrPrNrr.r.r/r ?sj         z-ControlNetXSCrossAttnDownBlock2D.from_modulesrNcCn|D]}d|_q|jg}t|jtjr||j|jdur&||j|D] }|D]}d|_q.q(dSrTNF) rrrrarr r=rdrr:r!r"rr.r.r/r      z3ControlNetXSCrossAttnDownBlock2D.freeze_base_paramsr{rirkrXrjrZr]r^encoder_attention_maskra.c  Cs.|dur|dddurtd|} |} d} d} tt|j|j}tt|j|j}ddd}t|||j |j D]\\}}\}}}}| rPt j | || gdd} |j rq|jrqtdd r_d d ini}t jjj||| |fi|} n|| |} |dur|| ||||d d d } | r|j r|jrtdd rd d ini}t jjj||| |fi|} n|| |} |dur|| ||||d d d } | r| || |} | | f} | | f} q9|jdur|j d}|j d}| rt j | || gdd} || } | r|| } | r| || |} | | f} | | f} | | | | fS)NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r.cfdd}|S)Ncdur |diS|SNr`r.inputsr'r`r.r/custom_forwardz_ControlNetXSCrossAttnDownBlock2D.forward..create_custom_forward..custom_forwardr.r'r`rr.rr/create_custom_forwardzGControlNetXSCrossAttnDownBlock2D.forward..create_custom_forwardrrg>=1.11.0 use_reentrantFrXr^r]rr`rrr6)rzloggerwarningrrrrrrr2r3r2cattrainingr&rutils checkpointrr)r:rirkrXrjrZr]r^rrarrbase_output_statesctrl_output_states base_blocks ctrl_blocksrb_resb_attnc_resc_attnb2cc2b ckpt_kwargsr.r.r/rs                   z(ControlNetXSCrossAttnDownBlock2D.forward) rCrCTrrrrDTFTr)NNr{NNNT)r)r*r+rbrr rrr8rrr0r r r rrrrrr?r.r.r;r/rs    v A   rcseZdZ         d'deded eed ed ed ed eedeedeededeeffdd Zedede fddZ d(ddZ      d)de de de dee d ee d!eeeefd"ee d#ee d$edee e ffd%d&ZZS)*rNrCrrDFTrkrlrIr\rrKrrrMrOrPc svtt|||_t||||| || | d|_t|||||tt||||| || | d |_t|||_ d|_ dS)N)rKrUrIrnrMrLrPrOrmF) r7r8rer2r$ base_midblockrfr ctrl_midblockr3r&) r:rkrlrIr\rrKrrrMrOrPr;r.r/r8s6     z(ControlNetXSCrossAttnMidBlock2D.__init__rrc Cs|j}|j}|j}dd}|j}|j}t|jdj}|jdj j } |jdj j } |jdj j } ||j } ||j } ||j}||j}|jdj}|||| | | || | |||d }|j||j||j||j||S)NcSrrr)rAr.r.r/rDrzOControlNetXSCrossAttnMidBlock2D.from_modules..get_first_cross_attentionrr)r2r3rArVrUrr4rr1rrrrrrMrOrPrrrr)rrrr2r3rrkrlrKrIrrrrrMrOrPrr.r.r/r 9sB     z,ControlNetXSCrossAttnMidBlock2D.from_modulesrcCs.|D]}d|_q|jD]}d|_qdS)rTFN)rrr)r:r!r.r.r/r js z2ControlNetXSCrossAttnMidBlock2D.freeze_base_paramsr{rirkrXrjrZr^r]rrac Cs|dur|dddurtd|} |} |||||d} | r+tj| || gdd} |j| fi| } | rH|j| fi| } | || |} | | fS)Nrr)rkrXr]r^rrrg) rzrrr2rr2rrr3) r:rirkrXrjrZr^r]rrarr joint_argsr.r.r/rus$  z'ControlNetXSCrossAttnMidBlock2D.forward) NrCrCrrrrDFTr)Nr{NNNT)r)r*r+rbrrr8rr$r@r r r rrrrrrr?r.r.r;r/rs    4 0   rcs eZdZ         d,deded ed eed ed ed eededededededeeffdd Zede de fddZ d-ddZ       d.de dee d fd!ee d fd"e d#ee d$eed%eeeefd&ee d'eed(ee d)ede fd*d+ZZS)/rrCNTrrDFrUrVrprqrIr\rrKrLrMrrOrPc s"tg}g}g}d}||_| |_t| tr| g|} t|D]@}||dkr+|n|}|dkr3|n|}|t||||t |||||d|ra|t | || || || || |dq!t ||_ |rot |ndg||_t ||_| rt|d|d|_nd|_d|_||_dS) NrrrrrrZT)r_rVF)r7r8has_cross_attentionrLrarbrcrdrer"r#r r=r1r4r3r% upsamplersr&r)r:rUrVrprqrIr\rrgrKrLrMrrOrPr1r4r3r[rhres_skip_channelsrsr;r.r/r8sV        z'ControlNetXSCrossAttnUpBlock2D.__init__ base_upblock ctrl_upblockcCsP|j}dd}|jdj}|jdj|}|jdj|}dd|D}|jdjj} |jdjj} |j} t |drZd} t |j dj } ||j }||j}||j}|j dj}n d } d} d}d}d}d}|jdu}|||||| | | | | |||||d }|j|j| r|j |j |r|j|jd|j||S) NcSrrrrr.r.r/rrzNControlNetXSCrossAttnUpBlock2D.from_modules..get_first_cross_attentionrrcSsg|]}|jqSr.)rU)rr r.r.r/rsz?ControlNetXSCrossAttnUpBlock2D.from_modules..r4TF)rUrVrprqrIr\rrgrKrLrMrrOrP)r3r1rVrUrrrrrrrr4rrrMrOrPrrr)rrrctrl_to_base_skip_connectionsrrVrUprev_output_channelsctrl_skip_channelssrIrrrgrKrLrMrOrPrrr.r.r/r s\      z+ControlNetXSCrossAttnUpBlock2D.from_modulesrcCrr) rrr1rar4r r=rdrrr.r.r/r rz1ControlNetXSCrossAttnUpBlock2D.freeze_base_paramsr{rlrm.rnrkrXrZr^r] upsample_sizerrac  sR|dur|dddurtdtddo(tddo(tddo(tddddd} fd d } tjjjt|t|D]W\}}}}}| rU||||7}| ||\}}t j ||gd d }j rj rt d drtddini}t jjj| |||fi|}n|||}|dur|||||| ddd}qDjdur|| }|S)NrrrFrGrHrIcr)Ncrrr.rrr.r/rHrz]ControlNetXSCrossAttnUpBlock2D.forward..create_custom_forward..custom_forwardr.rr.rr/rGrzEControlNetXSCrossAttnUpBlock2D.forward..create_custom_forwardc s,rtj||jjjjdS||fS)N)rFrGrHrI)rrrFrGrHrI)rl res_h_baseis_freeu_enabledr:r.r/maybe_apply_freeu_to_subblockPs zMControlNetXSCrossAttnUpBlock2D.forward..maybe_apply_freeu_to_subblockrrgrrrFrrr6)rzrrrrr1r4r3rr2rrr&rrrr)r:rlrmrnrkrXrZr^r]rrrarrresnetattnrr res_h_ctrlrr.rr/r.s`          z&ControlNetXSCrossAttnUpBlock2D.forward) rCNTrrrDTFTr)Nr{NNNNT)r)r*r+rbrrrr8rr rBr r r rrrrrrr?r.r.r;r/rs    G :       rcCsttj||dddS)Nrr)r) zero_moduler r>)rUrVr.r.r/resrecCs|D]}tj|q|Sr6)rr initzeros_)r'pr.r.r/rs rcCsD|}||kr|S|dkr ||}|dkr|S|d8}|dks dSdS)Nrrr.)numberrRfactorresidualr.r.r/rfsrf)rCTrrrDTFT)NrCrrrDFTr6)F dataclassesrmathrtypingrrrrrr r2torch.utils.checkpointr r configuration_utilsr rrrrrutils.torch_utilsrattention_processorrrrrrrrrr embeddingsrrmodeling_utilsrunets.unet_2d_blocksrr r!r"r#r$r%unets.unet_2d_conditionr& get_loggerr)rr'r3r0r@rBrbrrjrortrurrrrrerrfr.r.r.r/s   $  $        ^ #  M> n