o GiA@sddlmZddlmZddlmZddlZddlmZmZddl m Z m Z ddl m Z mZdd lmZd d lmZd d lmZmZmZmZmZmZd d lmZmZd dlmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(ddl)m*Z*e+e,Z-eGddde Z.Gdddej/Z0Gdddej/Z1Gdddej/Z2        dFde3d e3d!e3d"e3d#e3d$e3dBd%e3e4e3BdBd&e3dBd'e3dBd(e5d)e5dBd*e5dBfd+d,Z6       dGd-e3d.e3d#e3dBd$e3dBd%e3d&e3dBd'e3dBd)e5d*e5fd/d0Z7d1e3d2e3d3e8e3fd4d5Z9Gd6d7d7eee Z:Gd8d9d9eee Z;Gd:d;d;ej/Zd?d?ej/Z>dHd@dAZ?dBdCZ@dDdEZAdS)I) dataclass)gcd)AnyN)Tensornn) ConfigMixinregister_to_config) BaseOutputlogging) apply_freeu)AttentionMixin)ADDED_KV_ATTENTION_PROCESSORSCROSS_ATTENTION_PROCESSORS AttentionAttnAddedKVProcessor AttnProcessorFusedAttnProcessor2_0)TimestepEmbedding Timesteps) ModelMixin)CrossAttnDownBlock2DCrossAttnUpBlock2D Downsample2D ResnetBlock2DTransformer2DModelUNetMidBlock2DCrossAttn Upsample2D)UNet2DConditionModel)ControlNetConditioningEmbeddingc@seZdZUdZdZeed<dS)ControlNetXSOutputa= The output of [`UNetControlNetXSModel`]. Args: sample (`Tensor` of shape `(batch_size, num_channels, height, width)`): The output of the `UNetControlNetXSModel`. Unlike `ControlNetOutput` this is NOT to be added to the base model output, but is already the final output. Nsample)__name__ __module__ __qualname____doc__r#r__annotations__r)r)^/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/controlnets/controlnet_xs.pyr"3s  r"c sNeZdZdZ  d dejdejdejdejdBdejdBf fdd ZZS) DownBlockControlNetXSAdapterz}Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnDownBlock2D`Nresnets base_to_ctrl ctrl_to_base attentions downsamplercs,t||_||_||_||_||_dSN)super__init__r,r-r.r/ downsamplers)selfr,r-r.r/r0 __class__r)r*r3Es  z%DownBlockControlNetXSAdapter.__init__)NN) r$r%r&r'r ModuleListConv2dr3 __classcell__r)r)r6r*r+Asr+cs2eZdZdZdedejdejffdd ZZS)MidBlockControlNetXSAdapterz|Components that together with corresponding components from the base model will form a `ControlNetXSCrossAttnMidBlock2D`midblockr-r.cs t||_||_||_dSr1)r2r3r<r-r.)r5r<r-r.r6r)r*r3Ys  z$MidBlockControlNetXSAdapter.__init__) r$r%r&r'rrr8r3r:r)r)r6r*r;Us&r;cs(eZdZdZdejffdd ZZS)UpBlockControlNetXSAdapterzwComponents that together with corresponding components from the base model will form a `ControlNetXSCrossAttnUpBlock2D`r.cst||_dSr1)r2r3r.)r5r.r6r)r*r3cs  z#UpBlockControlNetXSAdapter.__init__)r$r%r&r'rr8r3r:r)r)r6r*r=`sr= TFbase_in_channelsbase_out_channelsctrl_in_channelsctrl_out_channels temb_channelsmax_norm_num_groupstransformer_layers_per_blocknum_attention_headscross_attention_dimadd_downsampleupcast_attentionuse_linear_projectionc CsZd} g}g}g}g}t|tr|g| }t| D]T}|dkr |n|}|dkr(|n|}|t|||t||||t|||dt||ddd|rd|t||||||| | | t||dd|t||q| r|t||t||d|dd }|t||nd}t t |t |t |d }|rt ||_ |dur||_ |S) Nr r max_factorh㈵> in_channels out_channelsrDgroups groups_outepsrP num_layersrHrKrJnorm_num_groupsTopuse_convrQname)r,r-r.) isinstanceintrangeappendmake_zero_convrfind_largest_factorrrr+rr8r/r4)r@rArBrCrDrE has_crossattnrFrGrHrIrJrKrVr,r/r.r-ir4down_block_componentsr)r)r*get_down_block_adapterhsh        re base_channels ctrl_channelsc CsPt||} t|||||tt||||||||d } t||} t| | | dS)N rFrPrQrD resnet_groupsrHrGrKrJ)r-r<r.)r`rrarr;) rfrgrDrErFrGrHrJrKr-r<r.r)r)r*get_mid_block_adapters  rjrQprev_output_channelctrl_skip_channelscCsJg}d}t|D]}|dkr|n|}|t|||qtt|dS)Nrr)r.)r^r_r`r=rr8)rQrkrlr.rVrcresnet_in_channelsr)r)r*get_up_block_adapters  rnc s eZdZdZe          d(dededeedede deeeBdeedeededeededBdeeeBde dede ffd d! Z e        d)d"e d#edBde edBde edBde dedededeefd$d%Zd&d'ZZS)*ControlNetXSAdaptera A `ControlNetXSAdapter` model. To use it, pass it into a `UNetControlNetXSModel` (together with a `UNet2DConditionModel` base model). This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). Like `UNetControlNetXSModel`, `ControlNetXSAdapter` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. Parameters: conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channels for each block in the `controlnet_cond_embedding` layer. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapters's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. learn_time_embedding (`bool`, defaults to `False`): Whether a time embedding should be learned. If yes, `UNetControlNetXSModel` will combine the time embeddings of the base model and the control adapter. If no, `UNetControlNetXSModel` will use the base model's time embedding. num_attention_heads (`list[int]`, defaults to `[4]`): The number of attention heads. block_out_channels (`list[int]`, defaults to `[4, 8, 16, 16]`): The tuple of output channels for each block. base_block_out_channels (`list[int]`, defaults to `[320, 640, 1280, 1280]`): The tuple of output channels for each block in the base unet. cross_attention_dim (`int`, defaults to 1024): The dimension of the cross attention features. down_block_types (`list[str]`, defaults to `["CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "CrossAttnDownBlock2D", "DownBlock2D"]`): The tuple of downsample blocks to use. sample_size (`int`, defaults to 96): Height and width of input/output sample. transformer_layers_per_block (`int | tuple[int]`, defaults to 1): The number of transformer blocks of type [`~models.attention.BasicTransformerBlock`]. Only relevant for [`~models.unet_2d_blocks.CrossAttnDownBlock2D`], [`~models.unet_2d_blocks.UNetMidBlock2DCrossAttn`]. upcast_attention (`bool`, defaults to `True`): Whether the attention computation should always be upcasted. max_norm_num_groups (`int`, defaults to 32): Maximum number of groups in group normal. The actual number will be the largest divisor of the respective channels, that is <= max_norm_num_groups. rrgbr>`?Frvrrrri@irzr?rrr DownBlock2Drsr Tr>conditioning_channelsconditioning_channel_order#conditioning_embedding_out_channelstime_embedding_mixlearn_time_embeddingrGblock_out_channelsbase_block_out_channelsrHdown_block_types sample_sizeNrFrJrErKcst|d}|dd}|dvrtd|t|t| kr-td|d| dt| ttfs;| gt| } t| ttfsI| gt| } t|ttfsW|gt| }t|t| krjtd|d| dt|d||d |_|r}t |||_ nd|_ t g|_ t g|_t jd|dd d d |_t|d|d|_|d}|d}t| D]8\}}|}||}|}||}d |v}|t| d k}|j t|||||||| |||| || | |d qt|d|d|| d|d| d| |d|_|dgt|D]\}}|t|d krd nd}|g|qtt|}|d}tt| D]}|}||}fddtd D}|jt|||dq2dS)Nrrv)rpbgrz&unknown `conditioning_channel_order`: zbMust provide the same number of `block_out_channels` as `down_block_types`. `block_out_channels`: z. `down_block_types`: .zdMust provide the same number of `num_attention_heads` as `down_block_types`. `num_attention_heads`: conditioning_embedding_channelsrr}rr  kernel_sizepadding CrossAttn) r@rArBrCrDrErbrFrGrHrIrJrK)rfrgrDrFrGrHrJrKr cg|]}qSr)pop.0_rlr)r* z0ControlNetXSAdapter.__init__..)rQrkrl)r2r3 ValueErrorlenr\listtupler!controlnet_cond_embeddingrtime_embeddingrr8 down_blocksup_connectionsr9conv_inr`control_to_base_for_conv_in enumerater_rerj mid_blockextendreversedr^rn)r5r}r~rrrrGrrrHrrrFrJrErKtime_embedding_input_dimtime_embedding_dimrArCrcdown_block_typer@rBrbis_final_blockrQnumber_of_subblocks reversed_base_block_out_channelsprev_base_output_channelctrl_skip_channels_r6rr*r3#s      zControlNetXSAdapter.__init__unet size_ratioc s|du} du} | | Astd|pfdd|jjD}|dur%|jj}|||| |||||jj|jj|jj|jj|jj|jj|jj |jj d} | |j | S)a8 Instantiate a [`ControlNetXSAdapter`] from a [`UNet2DConditionModel`]. Parameters: unet (`UNet2DConditionModel`): The UNet model we want to control. The dimensions of the ControlNetXSAdapter will be adapted to it. size_ratio (float, *optional*, defaults to `None`): When given, block_out_channels is set to a fraction of the base model's block_out_channels. Either this or `block_out_channels` must be given. block_out_channels (`list[int]`, *optional*, defaults to `None`): Down blocks output channels in control model. Either this or `size_ratio` must be given. num_attention_heads (`list[int]`, *optional*, defaults to `None`): The dimension of the attention heads. The naming seems a bit confusing and it is, see https://github.com/huggingface/diffusers/issues/2011#issuecomment-1547958131 for why. learn_time_embedding (`bool`, defaults to `False`): Whether the `ControlNetXSAdapter` should learn a time embedding. time_embedding_mix (`float`, defaults to 1.0): If 0, then only the control adapter's time embedding is used. If 1, then only the base unet's time embedding is used. Otherwise, both are combined. conditioning_channels (`int`, defaults to 3): Number of channels of conditioning input (e.g. an image) conditioning_channel_order (`str`, defaults to `"rgb"`): The channel order of conditional image. Will convert to `rgb` if it's `bgr`. conditioning_embedding_out_channels (`tuple[int]`, defaults to `(16, 32, 96, 256)`): The tuple of output channel for each block in the `controlnet_cond_embedding` layer. NzePass exactly one of `block_out_channels` (for absolute sizing) or `size_ratio` (for relative sizing).csg|]}t|qSr))r])rbrr)r*rsz1ControlNetXSAdapter.from_unet..)r}r~rrrrGrrrHrrrFrJrErK) rconfigrattention_head_dimrHrrrFrJrWrKtodtype) clsrrrrGrrr}r~r fixed_size relative_sizemodelr)rr* from_unets8) zControlNetXSAdapter.from_unetcOstd)NzA ControlNetXSAdapter cannot be run by itself. Use it together with a UNet2DConditionModel to instantiate a UNetControlNetXSModel.)r)r5argskwargsr)r)r*forwardszControlNetXSAdapter.forward)rrprqruFrvrwryr?r{rsr Tr>T)NNNFrurrprq)r$r%r&r'r r]strrfloatboolr3 classmethodrrrrr:r)r)r6r*ros.          Lroc.s"eZdZdZdZe              dPded Bdeedeedeeded BdeeeBdeeeBdeeeBded Bded Bde de ded Bd ed Bd!e d"ed#eed$ed%e d&eed'eeeBd(ef,fd)d* Z e dQd+e d,ed Bd-e d Bd&ee d Bd!e d Bd.ed Bf d/d0ZdRd2d3Zd4d5Zd6e d7e d8e d9e fd:d;Zdd?Zd@dAZ  dSdBedCeje BeBdDejdEejd BdFe d BdGejd BdHejd BdIejd BdJeeefd BdKeeejfd BdLe dMe d1eeBfdNdOZZS)TUNetControlNetXSModela9 A UNet fused with a ControlNet-XS adapter model This model inherits from [`ModelMixin`] and [`ConfigMixin`]. Check the superclass documentation for it's generic methods implemented for all models (such as downloading or saving). `UNetControlNetXSModel` is compatible with StableDiffusion and StableDiffusion-XL. It's default parameters are compatible with StableDiffusion. It's parameters are either passed to the underlying `UNet2DConditionModel` or used exactly like in `ControlNetXSAdapter` . See their documentation for details. Trsr{ UpBlock2Drrrryr>r?r rxNrurrqrpFrwrvrrup_block_typesrrWrHrFrGaddition_embed_typeaddition_time_embed_dimrJrKtime_cond_proj_dim%projection_class_embeddings_input_dimrctrl_conditioning_channels(ctrl_conditioning_embedding_out_channelsctrl_conditioning_channel_orderctrl_learn_time_embeddingctrl_block_out_channelsctrl_num_attention_headsctrl_max_norm_num_groupsc.st|dks |dkrtd|dkr|std| dur'| dkr'tdt|ttfs5|gt|}t|ttfsC|gt|}t|ttfsQ|gt|}t|ttfs_|gt|}|}d|_tj d|dddd |_ t |d||d |_ tj d|dddd |_ t|d|d|_|d}|dd}t|dd dd |_t||| d |_|rt||d|_nd|_| durd|_d|_nt| d dd |_t|||_g}|d}|d}t|D];\}}|}||}|} ||}d|v}!|t|dk}"|t||| |||||!|||||||||" | | dqt|d|d||||d|d|d|d| | d |_g}#tt|}$tt|}%tt|}&|dgt|D]\}}'|t|dkrddnd}(|'g|(qUtt|})|)d}'t|D]K\}}*|'}+|)|}'|)t|dt|d},fddtdD}-d|*v}!|t|dk}"|#t |,|'|+|-|||!|$||%||&||" | || dq~t!||_"t!|#|_#tj$|d|d|_%t&|_'tj |ddddd |_(dS)Nrr z1`time_embedding_mix` needs to be between 0 and 1.zKTo use `time_embedding_mix` < 1, `ctrl_learn_time_embedding` must be `True` text_timezAs `UNetControlNetXSModel` currently only supports StableDiffusion and StableDiffusion-XL, `addition_embed_type` must be `None` or `'text_time'`.rvrrrT)flip_sin_to_cosdownscale_freq_shift) cond_proj_dim)rPtime_embed_dimrr@rArBrCrDrWrrbrFbase_num_attention_headsrrHrIrJrKr rfrgrDrWrrFrrrHrJrKr crr)rrrr)r*rrz2UNetControlNetXSModel.__init__..)rPrQrkrlrDresolution_idxrbrFrGrH add_upsamplerJrWrK) num_channels num_groups))r2r3rr\rrrrPrr9 base_conv_inr!r ctrl_conv_inr`rrbase_time_projrbase_time_embeddingctrl_time_embeddingbase_add_time_projbase_add_embeddingrr_ ControlNetXSCrossAttnDownBlock2DControlNetXSCrossAttnMidBlock2Drrrminr^ControlNetXSCrossAttnUpBlock2Dr8r up_blocks GroupNormbase_conv_norm_outSiLU base_conv_act base_conv_out).r5rrrrrWrHrFrGrrrJrKrrrrrrrrrrrtime_embed_input_dimrrrArCrcrr@rBrbrr rev_transformer_layers_per_blockrev_num_attention_headsrev_cross_attention_dimrQrreversed_block_out_channels up_block_typerkrPrr6rr*r3s !            zUNetControlNetXSModel.__init__r controlnetrctrl_optional_kwargsc s|durtj|||fi|}ntdd||||fDr!tdgdfdd|jD|jjd<gd fd d|jD|jjd <|i}gd }|D]} t |d |  t ||  q\ddg} | D]} t || rt || durt |d |  t ||  qt|j |j |j |j |jdur|j |j |j |j tddt|j|jD|_t|j|j|_tddt|j|jD|_||j|S)a Instantiate a [`UNetControlNetXSModel`] from a [`UNet2DConditionModel`] and an optional [`ControlNetXSAdapter`] . Parameters: unet (`UNet2DConditionModel`): The UNet model we want to control. controlnet (`ControlNetXSAdapter`): The ControlNet-XS adapter with which the UNet will be fused. If none is given, a new ControlNet-XS adapter will be created. size_ratio (float, *optional*, defaults to `None`): Used to construct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details. ctrl_block_out_channels (`list[int]`, *optional*, defaults to `None`): Used to construct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details, where this parameter is called `block_out_channels`. time_embedding_mix (`float`, *optional*, defaults to None): Used to construct the controlnet if none is given. See [`ControlNetXSAdapter.from_unet`] for details. ctrl_optional_kwargs (`Dict`, *optional*, defaults to `None`): Passed to the `init` of the new controlnet if no controlnet was given. Ncss|]}|duVqdSr1r))ror)r)r* s z2UNetControlNetXSModel.from_unet..zWhen a controlnet is passed, none of these parameters should be passed: size_ratio, ctrl_block_out_channels, time_embedding_mix, ctrl_optional_kwargs.) rrrrrWrHrFrrrJrKrrcsi|] \}}|vr||qSr)r)rkv)params_for_unetr)r* sz3UNetControlNetXSModel.from_unet..rG)r}rr~rrrGrEcs"i|] \}}|vrd||qS)ctrl_r)r)params_for_controlnetr)r*rs"r)rr conv_norm_outconv_outbase_ add_time_proj add_embeddingcs |] \}}t||VqdSr1)r from_modulesrrcr)r)r*r6  csrr1)rrrr)r)r*r;r)roranyrritemsrr from_configgetattrload_state_dict state_dicthasattrrrrrrrrr8ziprrrrrrrr) rrrrrrrrmodules_from_unetmoptional_modules_from_unetr))rrr*rsV           zUNetControlNetXSModel.from_unetreturncsD]}d|_qgd}fdd|D}|D] }|D]}d|_qqjD]}|q)jjD]}|q8dS)Freeze the weights of the parts belonging to the base UNet2DConditionModel, and leave everything else unfrozen for fine tuning.T)rrrrrrrrcs$g|]}t|durt|qSr1)r )rpartr5r)r*rWs$z.FN) parameters requires_gradrfreeze_base_paramsrr)r5param base_partsrdur)rr*freeze_unet_paramsEs        z(UNetControlNetXSModel.freeze_unet_paramscCsjtdd|jDrt}ntdd|jDr t}ntdtt|j||dS)ze Disables custom attention processors and sets the default attention implementation. cs|]}|jtvVqdSr1)r7rrprocr)r)r*rgzCUNetControlNetXSModel.set_default_attn_processor..csr r1)r7rr!r)r)r*rir#zOCannot call `set_default_attn_processor` when attention processors are of type N) allattn_processorsvaluesrrrnextiterset_attn_processor)r5 processorr)r)r*set_default_attn_processorcsz0UNetControlNetXSModel.set_default_attn_processors1s2b1b2cCsHt|jD]\}}t|d|t|d|t|d|t|d|qdS)aFEnables the FreeU mechanism from https://huggingface.co/papers/2309.11497. The suffixes after the scaling factors represent the stage blocks where they are being applied. Please refer to the [official repository](https://github.com/ChenyangSi/FreeU) for combinations of values that are known to work well for different pipelines such as Stable Diffusion v1, v2, and Stable Diffusion XL. Args: s1 (`float`): Scaling factor for stage 1 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. s2 (`float`): Scaling factor for stage 2 to attenuate the contributions of the skip features. This is done to mitigate the "oversmoothing effect" in the enhanced denoising process. b1 (`float`): Scaling factor for stage 1 to amplify the contributions of backbone features. b2 (`float`): Scaling factor for stage 2 to amplify the contributions of backbone features. r,r-r.r/N)rrsetattr)r5r,r-r.r/rcupsample_blockr)r)r* enable_freeuss    z"UNetControlNetXSModel.enable_freeucCsPhd}t|jD]\}}|D]}t||st||ddur$t||dqq dS)zDisables the FreeU mechanism.>r.r/r,r-N)rrrr r0)r5 freeu_keysrcr1rr)r)r* disable_freeus z#UNetControlNetXSModel.disable_freeucCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsr%r rr7r$rmodulesr\rfuse_projectionsr)r)r5rattn_processormoduler)r)r*fuse_qkv_projectionss   z*UNetControlNetXSModel.fuse_qkv_projectionscCs|jdur ||jdSdS)unDisables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)r7r)rr)r)r*unfuse_qkv_projectionss z,UNetControlNetXSModel.unfuse_qkv_projectionsr#timestepencoder_hidden_statescontrolnet_condconditioning_scale class_labels timestep_condattention_maskcross_attention_kwargsadded_cond_kwargs return_dict apply_controlc ( Cs|jjdkrtj|dgd}|dur!d||jd}|d}|} t| sY|jj dk}|jj dk}t |t rD|s=|r@tj ntj }n |sH|rKtjntj}tj| g||jd} nt| jd krh| d|j} | |jd } || }|j|jd }|jjr| r|||}|||}|jjd }|||d|}n||}d}|jjdurnX|jjd krd | vrt|jd| d }d| vrt|jd| d}||}||jd df}tj ||gdd}||j}|!|}n td|jjd|dur||n|}|}|}}gg}}|"|} |#|}|$|}| dur3|| 7}| r?||%||}|&||&||j'D]}!|!|||||| || d\}}}"}#|(|"|(|#qL|j)|||||| || d\}}|j*D]1}$t|$j+}%||% d}&||% d}'|d|% }|d|% }|$||&|'|||| || d }q||,|}|-|}|.|}| s|fSt/|dS)a The [`ControlNetXSModel`] forward method. Args: sample (`Tensor`): The noisy input tensor. timestep (`torch.Tensor | float | int`): The number of timesteps to denoise an input. encoder_hidden_states (`torch.Tensor`): The encoder hidden states. controlnet_cond (`Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): How much the control model affects the base model outputs. class_labels (`torch.Tensor`, *optional*, defaults to `None`): Optional class labels for conditioning. Their embeddings will be summed with the timestep embeddings. timestep_cond (`torch.Tensor`, *optional*, defaults to `None`): Additional conditional embeddings for timestep. If provided, the embeddings will be summed with the timestep_embedding passed through the `self.time_embedding` layer to obtain the final timestep embeddings. attention_mask (`torch.Tensor`, *optional*, defaults to `None`): An attention mask of shape `(batch, key_tokens)` is applied to `encoder_hidden_states`. If `1` the mask is kept, otherwise if `0` it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens. cross_attention_kwargs (`dict[str]`, *optional*, defaults to `None`): A kwargs dictionary that if specified is passed along to the `AttnProcessor`. added_cond_kwargs (`dict`): Additional conditions for the Stable Diffusion XL UNet. return_dict (`bool`, defaults to `True`): Whether or not to return a [`~models.controlnets.controlnet.ControlNetOutput`] instead of a plain tuple. apply_control (`bool`, defaults to `True`): If `False`, the input is run only through the base model. Returns: [`~models.controlnetxs.ControlNetXSOutput`] **or** `tuple`: If `return_dict` is `True`, a [`~models.controlnetxs.ControlNetXSOutput`] is returned, otherwise a tuple is returned where the first element is the sample tensor. rr )dimsNgmpsnpu)rdevicer)rg333333?r text_embedsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `text_embeds` to be passed in `added_cond_kwargs`time_idsz has the config param `addition_embed_type` set to 'text_time' which requires the keyword argument `time_ids` to be passed in `added_cond_kwargs`rdimzgControlNet-XS currently only supports StableDiffusion and StableDiffusion-XL, so addition_embed_type = z is currently not supported.)hidden_states_basehidden_states_ctrltembr?rArErDrH) hidden_statesres_hidden_states_tuple_baseres_hidden_states_tuple_ctrlrSr?rArErDrH)r#)0rrtorchfliprr unsqueeze is_tensorrLtyper\rfloat32float64int32int64tensorrshapeexpandrrrrrrrr7getrflattenreshapeconcatrrrrrr_rrrrr,rrrr")(r5r#r>r?r@rArBrCrDrErFrGrH timestepsis_mpsis_npurt_emb ctrl_temb base_tembinterpolation_paramrSaug_embrMrN time_embeds add_embedscembh_ctrlh_basehs_basehs_ctrl guided_hintdown residual_hb residual_hcup n_resnetsskips_hbskips_hcr)r)r*rs 8                                zUNetControlNetXSModel.forward)rsr{rryr>r?r rxNNTTNNrurrqrpFrwrvr>)NNNNNrN) NruNNNNNTT)r$r%r&r' _supports_gradient_checkpointingr r]rrrrr3rrrordictrrr+r2r4r<r=rrWrr"rr:r)r)r6r*rs         ?  v       rcs*eZdZ          d+dededed ed ed ed ed eeeBdBdedBdedBdedBdededBdedBffdd Zedede fddZ d,ddZ       d-de d e d!e dBd"e dBd#e dBd$e dBd%eeefdBd&e dBd'edee e ee d(fee d(fffd)d*ZZS).rr>Tr r?Fr@rArBrCrDrWrrFNrrrHrIrJrKcstg}g}g}g}g}g}d}t| tr| g|} t|D]r}|dkr)|n|}|dkr1|n|}|t|||t||||d|t||||t|||dt||ddd|r|t | || || || |||d|t | || || || ||t||dd|t||q!| r|t||t |d|d d |_ t ||d|d d |_ |t||nd|_ d|_ t ||_t ||_|rt |ndg||_|rt |ndg||_t ||_t ||_d |_dS) Nr rrPrQrDrRrLrNrOrUTrXrYF)r2r3r\r]r^r_r`rrarrbase_downsamplersctrl_downsamplersrr8 base_resnets ctrl_resnetsbase_attentionsctrl_attentionsr-r.gradient_checkpointing)r5r@rArBrCrDrWrrbrFrrrHrIrJrKrrrrr.r-rVrcr6r)r*r3s              z)ControlNetXSCrossAttnDownBlock2D.__init__base_downblockctrl_downblockcCsdd}|jdj}|jdj}|jdj|}|jdj}|jdjj}|jdjj} |jdjj} t|dr]d} t|j dj } ||j } ||j }||j }||j }|j dj}nd} d} d} d}d}d}d}|jdu}||||||| | | | | |||||d}|j|j|j|j| r|j|j |j|j |r|j|jd|j|j|j|j|j|j|S)NcS|jdjdjSNrr/transformer_blocksattn2blockr)r)r*get_first_cross_attentionzPControlNetXSCrossAttnDownBlock2D.from_modules..get_first_cross_attentionrr/TFr)r,rPrQ time_emb_proj in_featuresnorm1rrrr/rheadsrHrJrKr4rr rrrrrrr-r.)rrrrr@rArBrCrDrctrl_num_groupsrbrFrrrHrJrKrIrr)r)r*rsj         z-ControlNetXSCrossAttnDownBlock2D.from_modulesrcCn|D]}d|_q|jg}t|jtjr||j|jdur&||j|D] }|D]}d|_q.q(dSrTNF) rrrr\rrr8r_rr5rrrr)r)r*r:     z3ControlNetXSCrossAttnDownBlock2D.freeze_base_paramsrurQrSr?rRrArDrEencoder_attention_maskrH.c  Cs|dur|dddurtd|} |} d} d} tt|j|j}tt|j|j}t|||j |j D]u\\}}\}}}}| rKt j | || gdd} t rZ|jrZ||| |} n|| |} |duro|| ||||ddd} | rt r|jr||| |} n|| |} |dur|| ||||ddd} | r| || |} | | f} | | f} q4|jdur|j d }|j d }| rt j | || gdd} || } | r|| } | r| || |} | | f} | | f} | | | | fS) NscaleSPassing `scale` to `cross_attention_kwargs` is deprecated. `scale` will be ignored.r)r rOFr?rErDrrGrr)rcloggerwarningrrrrrrr-r.rWcatis_grad_enabledr_gradient_checkpointing_funcrr)r5rQrSr?rRrArDrErrHrsrrbase_output_statesctrl_output_states base_blocks ctrl_blocksb_resb_attnc_resc_attnb2cc2br)r)r*rKsx                 z(ControlNetXSCrossAttnDownBlock2D.forward) r>r>Tr r r r?TFTr~)NNruNNNT)r$r%r&r]rrr3rrr+rrrrrrrrr:r)r)r6r*rs    v A   rcseZdZ         d'deded edBd ed ed ed edBdedBdedBdededBffdd ZededefddZ d(ddZ      d)de de de de dBd e dBd!e eefdBd"e dBd#e dBd$edee e ffd%d&ZZS)*rNr>r r?FTrfrgrDrWrrFrrrHrJrKc svtt|||_t||||| || | d|_t|||||tt||||| || | d |_t|||_ d|_ dS)N)rFrPrDrirHrGrKrJrhF) r2r3r`r-r base_midblockrar ctrl_midblockr.r) r5rfrgrDrWrrFrrrHrJrKr6r)r*r3s6     z(ControlNetXSCrossAttnMidBlock2D.__init__rrc Cs|j}|j}|j}dd}|j}|j}t|jdj}|jdj j } |jdj j } |jdj j } ||j } ||j } ||j}||j}|jdj}|||| | | || | |||d }|j||j||j||j||S)NcSrrr)r<r)r)r*rrzOControlNetXSCrossAttnMidBlock2D.from_modules..get_first_cross_attentionrr)r-r.r<rQrPrr/rr,rrrrrrHrJrKr rrr)rrrr-r.rrfrgrFrDrrrrrHrJrKrr)r)r*rsB     z,ControlNetXSCrossAttnMidBlock2D.from_modulesrcCs.|D]}d|_q|jD]}d|_qdS)rTFN)rrr)r5rr)r)r*rs z2ControlNetXSCrossAttnMidBlock2D.freeze_base_paramsrurQrSr?rRrArErDrrHc Cs|dur|dddurtd|} |} |||||d} | r+tj| || gdd} |j| fi| } | rH|j| fi| } | || |} | | fS)Nrr)rSr?rDrErr rO) rcrrrWrr-rrr.) r5rQrSr?rRrArErDrrHrsrr joint_argsr)r)r*rs$  z'ControlNetXSCrossAttnMidBlock2D.forward) Nr>r>r r r r?FTr~)NruNNNT)r$r%r&r]rr3rrr;rrrrrrrrrr:r)r)r6r*rs    4 0   rcs eZdZ         d,deded ed eed ed ed edBdededededededBffdd Zedede fddZ d-ddZ       d.de de e d fd!e e d fd"e d#e dBd$edBd%eeefdBd&e dBd'edBd(e dBd)ede fd*d+ZZS)/rr>NTr r?FrPrQrkrlrDrWrrFrGrHrrJrKc s"tg}g}g}d}||_| |_t| tr| g|} t|D]@}||dkr+|n|}|dkr3|n|}|t||||t |||||d|ra|t | || || || || |dq!t ||_ |rot |ndg||_t ||_| rt|d|d|_nd|_d|_||_dS) Nrr rrrUT)rZrQF)r2r3has_cross_attentionrGr\r]r^r_r`rrrr8r,r/r.r upsamplersrr)r5rPrQrkrlrDrWrrbrFrGrHrrJrKr,r/r.rVrcres_skip_channelsrmr6r)r*r3?sV        z'ControlNetXSCrossAttnUpBlock2D.__init__ base_upblock ctrl_upblockcCsP|j}dd}|jdj}|jdj|}|jdj|}dd|D}|jdjj} |jdjj} |j} t |drZd} t |j dj } ||j }||j}||j}|j dj}n d } d} d}d}d}d}|jdu}|||||| | | | | |||||d }|j|j| r|j |j |r|j|jd|j||S) NcSrrrrr)r)r*rrzNControlNetXSCrossAttnUpBlock2D.from_modules..get_first_cross_attentionrrcSsg|]}|jqSr))rP)rrr)r)r*rsz?ControlNetXSCrossAttnUpBlock2D.from_modules..r/TF)rPrQrkrlrDrWrrbrFrGrHrrJrK)r.r,rQrPrrrrrrrr/rrrHrJrKrr r)rrrctrl_to_base_skip_connectionsrrQrPprev_output_channelsctrl_skip_channelssrDrrrbrFrGrHrJrKrrr)r)r*rs\      z+ControlNetXSCrossAttnUpBlock2D.from_modulesrcCrr) rrr,r\r/rr8r_rrr)r)r*rrz1ControlNetXSCrossAttnUpBlock2D.freeze_base_paramsrurTrU.rVrSr?rArErD upsample_sizerrHc  s$|dur|dddurtdtddo(tddo(tddo(tddfdd} tjjjt|t|D]E\} }}}}| rP||||7}| ||\}}t j ||gd d }t roj ro | ||}n| ||}|dur|||||| d d d }q?jdur|| }|S)Nrrr,r-r.r/c s,rtj||jjjjdS||fS)N)r,r-r.r/)r rr,r-r.r/)rT res_h_baseis_freeu_enabledr5r)r*maybe_apply_freeu_to_subblocks zMControlNetXSCrossAttnUpBlock2D.forward..maybe_apply_freeu_to_subblockr rOFrr)rcrrr rr,r/r.rrWrrrrr)r5rTrUrVrSr?rArErDrrrHrresnetattnrr res_h_ctrlr)rr*rsP       z&ControlNetXSCrossAttnUpBlock2D.forward) r>NTr r r?TFTr~)NruNNNNT)r$r%r&r]rrr3rrr=rrrrrrrrrr:r)r)r6r*r>s    G :       rcCsttj||dddS)Nr r)r) zero_modulerr9)rPrQr)r)r*r`sr`cCs|D]}tj|q|Sr1)rrinitzeros_)r;pr)r)r*r s rcCsD|}||kr|S|dkr ||}|dkr|S|d8}|dks dSdS)Nrr r))numberrMfactorresidualr)r)r*ra&sra)r>Tr r r?TFT)Nr>r r r?FTr1)B dataclassesrmathrtypingrrWrrconfiguration_utilsrr utilsr r utils.torch_utilsr attentionrattention_processorrrrrrr embeddingsrrmodeling_utilsrunets.unet_2d_blocksrrrrrrrunets.unet_2d_conditionrrr! get_loggerr$rr"Moduler+r;r=r]rrrerjrrnrorrrrr`rrar)r)r)r*s      $         ^ #  ) _