o pi$F@sddlmZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZmZddlmZddlmZmZmZdd lmZdd lmZdd lmZmZmZm Z m!Z!d d l"m#Z#m$Z$d dl%m&Z&m'Z'e(e)Z*eGddde#Z+Gdddee eeZ,GdddeZ-dS)) dataclass)AnyDictListOptionalTupleUnionN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin)JointTransformerBlock) AttentionAttentionProcessorFusedJointAttnProcessor2_0)Transformer2DModelOutput) ModelMixin)USE_PEFT_BACKENDis_torch_versionloggingscale_lora_layersunscale_lora_layers) BaseOutput zero_module)"CombinedTimestepTextProjEmbeddings PatchEmbedc@seZdZUeejed<dS)SD3ControlNetOutputcontrolnet_block_samplesN)__name__ __module__ __qualname__rtorchTensor__annotations__r&r&]/home/ubuntu/SoloSpeech/.venv/lib/python3.10/site-packages/diffusers/models/controlnet_sd3.pyr$s rcsBeZdZdZe          d9d ed ed ededededededededeffdd Zd:deededdfddZe de e e ffdd Z d!ee e e e fffd"d#Zd$d%Zd&d'Zd;d)d*ZedSD3ControlNetModelTr @` sample_size patch_size in_channels num_layersattention_head_dimnum_attention_headsjoint_attention_dimcaption_projection_dimpooled_projection_dim out_channelspos_embed_max_sizec st|} | dur | n| _|_t||||j| d_tj| d_t ||_ t fddt |D_ t g_t tj D]} t jj}t|}j|qMt||||jdd}t|_d_dS)N)heightwidthr2r3 embed_dimr;) embedding_dimr9cs"g|] }tjjjddqS)F)dimr6r5context_pre_only)r inner_dimconfigr5).0ir6selfr&r' Psz/SD3ControlNetModel.__init__..)r<r=r2r3r>pos_embed_typeF)super__init__r:rBr pos_embedrtime_text_embednnLinearcontext_embedder ModuleListrangetransformer_blockscontrolnet_blockslenrappendpos_embed_inputgradient_checkpointing)rGr1r2r3r4r5r6r7r8r9r:r;default_out_channels_controlnet_blockrW __class__rFr'rK,sH      zSD3ControlNetModel.__init__Nr chunk_sizer@returncsZ|dvr td||pd}dtjjdtdtffdd |D]}|||q"d S) aX Sets the attention processor to use [feed forward chunking](https://huggingface.co/blog/reformer#2-chunked-feed-forward-layers). Parameters: chunk_size (`int`, *optional*): The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=`dim`. dim (`int`, *optional*, defaults to `0`): The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length). )rrz-Make sure to set `dim` to either 0 or 1, not rmoduler^r@cs6t|dr |j||d|D]}|||qdS)Nset_chunk_feed_forward)r^r@)hasattrrachildren)r`r^r@childfn_recursive_feed_forwardr&r'rfs  zMSD3ControlNetModel.enable_forward_chunking..fn_recursive_feed_forwardN) ValueErrorr#rNModuleintrc)rGr^r@r`r&rer'enable_forward_chunkingns  z*SD3ControlNetModel.enable_forward_chunkingcsLi}dtdtjjdtttfffdd |D] \}}|||q|S)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namer` processorscsHt|dr|||d<|D]\}}|d|||q|S)N get_processor .processor.)rbrmnamed_children)rkr`rlsub_namerdfn_recursive_add_processorsr&r'rss zGSD3ControlNetModel.attn_processors..fn_recursive_add_processors)strr#rNrhrrrp)rGrlrkr`r&rrr'attn_processorss  & z"SD3ControlNetModel.attn_processors processorcst|j}t|tr"t||kr"tdt|d|d|ddtdtjj ffdd | D] \}}|||q3d S) a4 Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.rkr`csbt|drt|ts||n |||d|D]\}}|d|||qdS)N set_processorrnro)rb isinstancedictrwpoprp)rkr`rvrqrdfn_recursive_attn_processorr&r'r|s   zJSD3ControlNetModel.set_attn_processor..fn_recursive_attn_processorN) rUrukeysrxryrgrtr#rNrhrp)rGrvcountrkr`r&r{r'set_attn_processors   z%SD3ControlNetModel.set_attn_processorcCsnd|_|jD]\}}dt|jjvrtdq|j|_|D] }t|t r.|j ddq!| t dS)u1 Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. This API is 🧪 experimental. NAddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.T)fuse) original_attn_processorsruitemsrtr]r rgmodulesrxrfuse_projectionsrr)rGrZattn_processorr`r&r&r'fuse_qkv_projectionss    z'SD3ControlNetModel.fuse_qkv_projectionscCs|jdur ||jdSdS)uDisables the fused QKV projection if enabled. This API is 🧪 experimental. N)rr)rGr&r&r'unfuse_qkv_projectionss z)SD3ControlNetModel.unfuse_qkv_projectionsFcCst|dr ||_dSdS)NrX)rbrX)rGr`valuer&r&r'_set_gradient_checkpointings  z.SD3ControlNetModel._set_gradient_checkpointing cCs|j}|p|j|d<|di|}|r?|j|j|j|j|j|j|jj|jddt|j |_ |S)Nr4F)strictr&) rCr4rLload_state_dict state_dictrMrPrSrrW)cls transformerr4load_weights_from_transformerrC controlnetr&r&r'from_transformers z#SD3ControlNetModel.from_transformer? hidden_statescontrolnet_condconditioning_scaleencoder_hidden_statespooled_projectionstimestepjoint_attention_kwargs return_dictc s\|dur|}|dd} nd} trt|| n|dur*|dddur*td||}|||} | |}|| |}d} |j D]7} |j ro|j roddd} tdd r\d d ini}tjjj| | ||| fi|}n | ||| d \}}| |f} qFd}t| |jD] \}}||}||f}qfd d|D}trt|| |s|fSt|dS)a The [`SD3Transformer2DModel`] forward method. Args: hidden_states (`torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.FloatTensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nscalerz\Passing `scale` via `joint_attention_kwargs` when not using the PEFT backend is ineffective.r&csfdd}|S)Ncsdur |diS|S)Nrr&)inputsr`rr&r'custom_forwardFszQSD3ControlNetModel.forward..create_custom_forward..custom_forwardr&)r`rrr&rr'create_custom_forwardEsz9SD3ControlNetModel.forward..create_custom_forwardz>=z1.11.0 use_reentrantF)rrtembcsg|]}|qSr&r&)rDsamplerr&r'rHdsz.SD3ControlNetModel.forward..)rN)copyrzrrgetloggerwarningrLrMrPrWrStrainingrXrr#utils checkpointziprTrr)rGrrrrrrrr lora_scalerblock_res_samplesblockr ckpt_kwargscontrolnet_block_res_samplesblock_res_sampler[r&rr'forwardsR'             zSD3ControlNetModel.forward) r)r r*r+r,r+r-r.r/r*r0)Nr)F)rT)rNNNNT)r r!r" _supports_gradient_checkpointingr rirKrrjpropertyrrtrrurrrrr classmethodrr# FloatTensorr$float LongTensorrboolrr __classcell__r&r&r\r'r()s    A#    r(cs~eZdZdZfddZ    ddejdeejdee d ejd ejd ej d e e e efd edeeeffddZZS)SD3MultiControlNetModela `SD3ControlNetModel` wrapper class for Multi-SD3ControlNet This module is a wrapper for multiple instances of the `SD3ControlNetModel`. The `forward()` API is designed to be compatible with `SD3ControlNetModel`. Args: controlnets (`List[SD3ControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `SD3ControlNetModel` as a list. cstt||_dSr)rJrKrNrQnets)rG controlnetsr\r&r'rK}s z SD3MultiControlNetModel.__init__NTrrrrrrrrr_c  Csrtt|||jD]-\} \} } } | ||||| | ||d} | dkr#| }q ddt|d| dD}t|f}q |S)N)rrrrrrrrrcSsg|]\}}||qSr&r&)rDcontrol_block_sample block_sampler&r&r'rHsz3SD3MultiControlNetModel.forward..) enumeraterrtuple)rGrrrrrrrrrEimagerr block_samplescontrol_block_samplesr&r&r'rs$   zSD3MultiControlNetModel.forward)NNNT)r r!r"__doc__rKr#rrtensorrrrrrtrrrrrrrr&r&r\r'rps4    r). dataclassesrtypingrrrrrrr#torch.nnrNconfiguration_utilsr r loadersr r models.attentionrmodels.attention_processorrrrmodels.modeling_outputsrmodels.modeling_utilsrrrrrrrrrr embeddingsrr get_loggerr rrr(rr&r&r&r's(       I