o GiK@sddlmZddlmZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZdd lmZdd lmZmZdd lmZmZmZdd lmZdd lmZddlmZmZe e!Z"eGddde Z#Gdddeeee Z$GdddeZ%dS)) dataclass)AnyN) ConfigMixinregister_to_config)PeftAdapterMixin) BaseOutputapply_lora_scalelogging)AttentionMixin)ControlNetConditioningEmbedding zero_module)*CombinedTimestepGuidanceTextProjEmbeddings"CombinedTimestepTextProjEmbeddings FluxPosEmbed)Transformer2DModelOutput) ModelMixin)FluxSingleTransformerBlockFluxTransformerBlockc@s*eZdZUeejed<eejed<dS)FluxControlNetOutputcontrolnet_block_samplescontrolnet_single_block_samplesN)__name__ __module__ __qualname__tupletorchTensor__annotations__r r `/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/models/controlnets/controlnet_flux.pyr's rcs"eZdZdZedddddddd d gd d d f d edededededededededeededeffdd Ze     d/dedededefddZ e d d0d!e j d"e j d#e j d$ed%e j d&e j d'e jd(e j d)e j d*e j deeefd Bd+ed,e jeBfd-d.ZZS)1FluxControlNetModelT@&iiF)8r*N patch_size in_channels num_layersnum_single_layersattention_head_dimnum_attention_headsjoint_attention_dimpooled_projection_dimguidance_embedsaxes_dims_ropenum_modeconditioning_embedding_channelsc st|__td| d_| rtnt} | j|d_t |j_ t j |j_ t fddt|D_t fddt|D_t g_ttjD]}jtt jjqbt g_ttjD]}jtt jjq| du_jrt | j_| durt| dd_t j |j_nd_tt j |j_d _dS) Ni')thetaaxes_dim) embedding_dimr2cg|] }tjdqS)dimr0r/)r inner_dim.0ir/r0selfr r! Pz0FluxControlNetModel.__init__..cr:r;)rr=r>rAr r!rC[rD)r)r)r)r))r6block_out_channelsF)super__init__ out_channelsr=r pos_embedrrtime_text_embednnLinearcontext_embedderr x_embedder ModuleListrangetransformer_blockssingle_transformer_blockscontrolnet_blockslenappendrcontrolnet_single_blocksunion Embeddingcontrolnet_mode_embedderr input_hint_blockcontrolnet_x_embeddergradient_checkpointing)rBr+r,r-r.r/r0r1r2r3r4r5r6text_time_guidance_cls_ __class__rAr!rG0sL        zFluxControlNetModel.__init__ c Cst|j}||d<||d<||d<||d<||}|r\|j|j|j|j|j|j|j|j|j j|j dd|j j|j ddt |j |_ |S)Nr-r.r/r0F)strict) dictconfig from_configrIload_state_dict state_dictrJrMrNrQrRrr[) cls transformerr-r.r/r0load_weights_from_transformerre controlnetr r r!from_transformer}s"    z$FluxControlNetModel.from_transformerjoint_attention_kwargs? hidden_statescontrolnet_condcontrolnet_modeconditioning_scaleencoder_hidden_statespooled_projectionstimestepimg_idstxt_idsguidance return_dictreturnc s||}|jdurC||}|j\} }}}||jj}||jj}|| |||jj||jj}|dddddd}|| ||d}|||}||j d }| dur_| |j d } nd} | durk| ||n| || |}| |}| j dkrt d | d} |j dkrt d |d}|jr|durtd ||}tj||gdd }tj| dd| gdd } tj| |fdd }||}d}t|jD]&\}}tr|jr||||||\}}n |||||d\}}||f}qd}t|jD](\}}tr|jr||||||\}}n |||||d\}}||f}qd}t||jD]\}}||}||f}q-d}t||jD]\}}||}||f}qDfdd|D}fdd|D}t|dkrndn|}t|dkrydn|}| s||fSt||dS)a The [`FluxTransformer2DModel`] forward method. Args: hidden_states (`torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. controlnet_mode (`torch.Tensor`): The mode tensor of shape `(batch_size, 1)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.FloatTensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. pooled_projections (`torch.FloatTensor` of shape `(batch_size, projection_dim)`): Embeddings projected from the embeddings of input conditions. timestep ( `torch.LongTensor`): Used to indicate denoising step. block_controlnet_hidden_states: (`list` of `torch.Tensor`): A list of tensors that if specified are added to the residuals of transformer blocks. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.transformer_2d.Transformer2DModelOutput`] instead of a plain tuple. Returns: If `return_dict` is True, an [`~models.transformer_2d.Transformer2DModelOutput`] is returned, otherwise a `tuple` where the first element is the sample tensor. Nrr rar#rizrPassing `txt_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch TensorzrPassing `img_ids` 3d torch.Tensor is deprecated.Please remove the batch dimension and pass it as a 2d torch TensorzA`controlnet_mode` cannot be `None` when applying ControlNet-Union)r<r )rprttembimage_rotary_embcg|]}|qSr r r?samplersr r!rC5z/FluxControlNetModel.forward..crr r rrr r!rC6r)rr) rNrZshaperer+reshapepermuter[todtyperJrMndimloggerwarningrW ValueErrorrYrcatrI enumeraterQis_grad_enabledr\_gradient_checkpointing_funcrRziprSrVrTr) rBrprqrrrsrtrurvrwrxryrnrz batch_sizechannels height_pwwidth_pwheightwidthr~controlnet_mode_embidsr block_samples index_blockblocksingle_block_samplesr block_samplecontrolnet_blockrsingle_block_sampler rr!forwards 0                zFluxControlNetModel.forward)rarbr'r(T) NroNNNNNNNT)rrr _supports_gradient_checkpointingrintboollistrG classmethodrmr rrfloat LongTensorrdstrr FloatTensorrr __classcell__r r r_r!r"-s     L      r"cseZdZdZfddZ        ddejdeejdeejd ee d ej d ej d ej d ej dej dej de e efdBdedeeBfddZZS)FluxMultiControlNetModela `FluxMultiControlNetModel` wrapper class for Multi-FluxControlNetModel This module is a wrapper for multiple instances of the `FluxControlNetModel`. The `forward()` API is designed to be compatible with `FluxControlNetModel`. Args: controlnets (`list[FluxControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `FluxControlNetModel` as a list. cstt||_dS)N)rFrGrKrOnets)rB controlnetsr_r r!rGSs z!FluxMultiControlNetModel.__init__NTrprqrrrsrtrurvrwrxryrnrzr{c Cst|jdkre|jd} tt|||D]L\}\}}}| |||dddf||| ||| || | d \}}|dkr<|}|}q|durN|durNddt||D}|dur`|dur`ddt||D}q||fStt||||jD]M\}\}}}} | |||dddf||| ||| || | d \}}|dkr|}|}qo|dur|durddt||D}|dur|durddt||D}qo||fS) Nr#r) rprqrrrsrvryrurtrxrwrnrzcSg|]\}}||qSr r r?control_block_samplerr r r!rCz4FluxMultiControlNetModel.forward..cSrr r r?control_single_block_samplerr r r!rCrcSrr r rr r r!rCrcSrr r rr r r!rCr)rTrrr)rBrprqrrrsrtrurvrwrxryrnrzrlr@imagemodescalerrcontrol_block_samplescontrol_single_block_samplesr r r!rWs  . z FluxMultiControlNetModel.forward)NNNNNNNT)rrr__doc__rGrrrtensorrrrrdrrrrrrrr r r_r!rFsL       r)& dataclassesrtypingrrtorch.nnrKconfiguration_utilsrrloadersrutilsrr r attentionr controlnets.controlnetr r embeddingsrrrmodeling_outputsrmodeling_utilsrtransformers.transformer_fluxrr get_loggerrrrr"rr r r r!s(