o ۷ibL@s ddlmZddlmZddlZddlmZddlmZm Z ddl m Z m Z ddl mZmZdd lmZmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZddlm Z m!Z!e"e#Z$eGddde Z%Gdddeeee e Z&GdddeZ'dS)) dataclass)AnyN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)apply_lora_scalelogging)AttentionMixinJointTransformerBlock) AttentionFusedJointAttnProcessor2_0)"CombinedTimestepTextProjEmbeddings PatchEmbed)Transformer2DModelOutput) ModelMixin)SD3SingleTransformerBlock) BaseOutput zero_modulec@seZdZUeejed<dS)SD3ControlNetOutputcontrolnet_block_samplesN)__name__ __module__ __qualname__tupletorchTensor__annotations__r!r!a/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/controlnets/controlnet_sd3.pyr%s rc$sFeZdZdZdZe            d>dededededededededededededeedfdedBdedBd e d!e f"fd"d# Z d?d$edBd%ed&dfd'd(Z d)d*Z d+d,Z d-d.Ze d@d1d2Zed3 4     dAd5ejd6ejd7ed8ejd9ejd:ejd3eeefdBd;e d&ejeBfds z/SD3ControlNetModel.__init__..csg|] }tjdqS))rBr2r1)rrE)rF_)r1r2rIr!r"rJs)r>r?r.r/r@r;F)super__init__r6rEr pos_embedrtime_text_embednnLinearcontext_embedder ModuleListrangetransformer_blockscontrolnet_blockslenrappendpos_embed_inputgradient_checkpointing)rIr-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<r=default_out_channelsrKcontrolnet_blockrY __class__rHr"rMWs\        zSD3ControlNetModel.__init__ chunk_sizerBreturncsZ|dvr td||pd}dtjjdtdtffdd |D]}|||q"d S) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rmoduler_rBcs6t|dr |j||d|D]}|||qdS)Nset_chunk_feed_forward)r_rB)hasattrrbchildren)rar_rBchildfn_recursive_feed_forwardr!r"rgs  zMSD3ControlNetModel.enable_forward_chunking..fn_recursive_feed_forwardN) ValueErrorrrPModuleintrd)rIr_rBrar!rfr"enable_forward_chunkings  z*SD3ControlNetModel.enable_forward_chunkingcCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. > [!WARNING] > This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsattn_processorsitemsstrr^rrhmodules isinstancerfuse_projectionsset_attn_processorr)rIrKattn_processorrar!r!r"fuse_qkv_projectionss   z'SD3ControlNetModel.fuse_qkv_projectionscCs|jdur ||jdSdS)unDisables the fused QKV projection if enabled. > [!WARNING] > This API is 🧪 experimental. N)rnru)rIr!r!r"unfuse_qkv_projectionss z)SD3ControlNetModel.unfuse_qkv_projectionscCsBt|jj|jj|jj|jj|j|jjd}|j|j dd|S)N)r>r?r.r/r@r7Tstrict) rconfigr-r.r/rEr7load_state_dictrN state_dict)rI transformerrNr!r!r"_get_pos_embed_from_transformersz2SD3ControlNetModel._get_pos_embed_from_transformer rcCs|j}|p|j|d<||d<||}|rA|j|j|j|j|j|j|jj|jddt |j |_ |S)Nr0r8Fry) r{r0 from_configrNr|r}rOrRrUrrY)clsr~r0num_extra_conditioning_channelsload_weights_from_transformerr{ controlnetr!r!r"from_transformers  z#SD3ControlNetModel.from_transformerjoint_attention_kwargs? hidden_statescontrolnet_condconditioning_scaleencoder_hidden_statespooled_projectionstimestep return_dictc s|jdur|jdkrtd|jdur|jdkrtd|jdur)|dur)td|jdur6|dur6td|jdur@||}|||} |jdurP||}|||}d} |jD]:} tr}|j r}|jduru| | ||| \}}n| | || }n|jdur| ||| d \}}n| || }| |f} q\d} t | |j D] \} }|| } | | f} qfd d | D} |s| fSt | d S) a The [`SD3Transformer2DModel`] forward method. Args: hidden_states (`torch.Tensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.Tensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.Tensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nz/hidden_states must be 4D when pos_embed is usedrz3hidden_states must be 3D when pos_embed is not usedzDencoder_hidden_states must be provided when context_embedder is usedzNencoder_hidden_states should not be provided when context_embedder is not usedr!)rrtembcsg|]}|qSr!r!)rFsamplerr!r"rJqsz.SD3ControlNetModel.forward..)r)rNndimrhrRrOrYrUris_grad_enabledrZ_gradient_checkpointing_funcziprVr)rIrrrrrrrrrblock_res_samplesblockcontrolnet_block_res_samplesblock_res_sampler\r!rr"forwardsN(             zSD3ControlNetModel.forward)r$r r%r&r'r&r(r)r*r%r+rr!Nr,TT)Nr)rrT)rNNNNT)rrr__doc__ _supports_gradient_checkpointingrrjrrqboolrMrkrwrxr classmethodrr rrfloat LongTensordictrrr __classcell__r!r!r]r"r#*s*      Z     r#cszeZdZdZfddZ    ddejdeejdee d ejd ejd ej d e e e fdBd edeeBfddZZS)SD3MultiControlNetModela `SD3ControlNetModel` wrapper class for Multi-SD3ControlNet This module is a wrapper for multiple instances of the `SD3ControlNetModel`. The `forward()` API is designed to be compatible with `SD3ControlNetModel`. Args: controlnets (`list[SD3ControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `SD3ControlNetModel` as a list. cstt||_dS)N)rLrMrPrSnets)rI controlnetsr]r!r"rMs z SD3MultiControlNetModel.__init__NTrrrrrrrrr`c  Csrtt|||jD]-\} \} } } | ||||| | ||d} | dkr#| }q ddt|d| dD}t|f}q |S)N)rrrrrrrrrcSsg|]\}}||qSr!r!)rFcontrol_block_sample block_sampler!r!r"rJsz3SD3MultiControlNetModel.forward..) enumeraterrr)rIrrrrrrrrrGimagescaler block_samplescontrol_block_samplesr!r!r"rs$   zSD3MultiControlNetModel.forward)NNNT)rrrrrMrrlisttensorrrrrqrrrrrrr!r!r]r"rys4    r)( dataclassesrtypingrrtorch.nnrPconfiguration_utilsrrloadersrrutilsr r attentionr r attention_processorrr embeddingsrrmodeling_outputsrmodeling_utilsrtransformers.transformer_sd3rrrr get_loggerrloggerrr#rr!r!r!r"s*       Q