o ۷i8 @sddlmZddlmZddlZddlmZddlmZm Z ddl m Z m Z ddl mZmZmZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZm Z m!Z!e"e#Z$eGdddeZ%Gdddeeee e eZ&Gdddeee e eZ'dS)) dataclass)AnyN) ConfigMixinregister_to_config)FromOriginalModelMixinPeftAdapterMixin) BaseOutputapply_lora_scale deprecatelogging)AttentionMixin) CacheMixin) zero_module)Transformer2DModelOutput) ModelMixin) QwenEmbedRopeQwenImageTransformerBlockQwenTimestepProjEmbeddingsRMSNormcompute_text_seq_len_from_maskc@seZdZUeejed<dS)QwenImageControlNetOutputcontrolnet_block_samplesN)__name__ __module__ __qualname__tupletorchTensor__annotations__r!r!g/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/models/controlnets/controlnet_qwenimage.pyr.s rcseZdZdZe        d(d ed ed edBdededededeeeefdeffdd Ze     d)dedededefddZ e d        d*de j de j de de j d e j d!e jd"eeeeefdBd#eedBdeeefdBd$ed%e jeBfd&d'ZZS)+QwenImageControlNetModelTr @<r%8r+r patch_size in_channels out_channelsN num_layersattention_head_dimnum_attention_headsjoint_attention_dimaxes_dims_ropeextra_condition_channelsc st|p|__tdt|dd_tjd_t |dd_ t |j_ t |j_t fddt|D_t g_ttjD]} jtt jjqUttj || j_d _dS) Ni'T)thetaaxes_dim scale_rope) embedding_dimgư>)epscsg|] }tjdqS))dimr1r0)r inner_dim).0_r0r1selfr!r" Ssz5QwenImageControlNetModel.__init__..F)super__init__r.r;rlist pos_embedrtime_text_embedrtxt_normnnLinearimg_intxt_in ModuleListrangetransformer_blockscontrolnet_blockslenappendrrcontrolnet_x_embeddergradient_checkpointing) r?r,r-r.r/r0r1r2r3r4r= __class__r>r"rB8s(    z!QwenImageControlNetModel.__init__c Cst|j}||d<||d<||d<||d<||}|rQ|j|j|j|j|j|j|j|j|j j|j ddt |j |_ |S)Nr/r0r1r4F)strict) dictconfig from_configrDload_state_dict state_dictrErIrJrMrrQ) cls transformerr/r0r1load_weights_from_transformerr4rX controlnetr!r!r"from_transformergs   z)QwenImageControlNetModel.from_transformerjoint_attention_kwargs? hidden_statescontrolnet_condconditioning_scaleencoder_hidden_statesencoder_hidden_states_masktimestep img_shapes txt_seq_lens return_dictreturnc  s|dur tddddd||}|||}|||} t||\} } }|j|| |jd}||j}| |}| |}| durG| ni}|duro|j dd\}}t j||ft j|jd }t j||gd d }||d <d }|jD](}t r|jr||||d| ||\}}n |||d| ||d\}}||f}qtd }t||jD] \}}||}||f}qfdd|D}t|dkrdn|}| s|St|dS)a The [`QwenImageControlNetModel`] forward method. Args: hidden_states (`torch.FloatTensor` of shape `(batch size, channel, height, width)`): Input `hidden_states`. controlnet_cond (`torch.Tensor`): The conditional input tensor of shape `(batch_size, sequence_length, hidden_size)`. conditioning_scale (`float`, defaults to `1.0`): The scale factor for ControlNet outputs. encoder_hidden_states (`torch.FloatTensor` of shape `(batch size, sequence_len, embed_dims)`): Conditional embeddings (embeddings computed from the input conditions such as prompts) to use. encoder_hidden_states_mask (`torch.Tensor` of shape `(batch_size, text_sequence_length)`, *optional*): Mask for the encoder hidden states. Expected to have 1.0 for valid tokens and 0.0 for padding tokens. Used in the attention processor to prevent attending to padding tokens. The mask can have any pattern (not just contiguous valid tokens followed by padding) since it's applied element-wise in attention. timestep ( `torch.LongTensor`): Used to indicate denoising step. img_shapes (`list[tuple[int, int, int]]`, *optional*): Image shapes for RoPE computation. txt_seq_lens (`list[int]`, *optional*): **Deprecated**. Not needed anymore, we use `encoder_hidden_states` instead to infer text sequence length. joint_attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~models.controlnet.ControlNetOutput`] instead of a plain tuple. Returns: If `return_dict` is True, a [`~models.controlnet.ControlNetOutput`] is returned, otherwise a `tuple` where the first element is the controlnet block samples. Nrj0.39.0zPassing `txt_seq_lens` to `QwenImageControlNetModel.forward()` is deprecated and will be removed in version 0.39.0. The text sequence length is now automatically inferred from `encoder_hidden_states` and `encoder_hidden_states_mask`.F standard_warn)max_txt_seq_lendevicer )dtyperq)r:attention_maskr!)rcrfrgtembimage_rotary_embracsg|]}|qSr!r!)r<samplerer!r"r@sz4QwenImageControlNetModel.forward..r)r)r rIrQrErrDrqtorrrFrJcopyshaperonesboolcatrMis_grad_enabledrR_gradient_checkpointing_funcziprNrOr)r?rcrdrerfrgrhrirjrarkru text_seq_lenr=rvblock_attention_kwargs batch_size image_seq_len image_maskjoint_attention_mask block_samplesblockr block_samplecontrolnet_blockr!rxr"forwardsn1           z QwenImageControlNetModel.forward) r r$r%r&r'r(r)r*r)rUr'r(Tr)rbNNNNNNT)rrr _supports_gradient_checkpointingrintrrB classmethodr`r rrfloat LongTensorrCrWstrrr} FloatTensorrr __classcell__r!r!rSr"r#3s  .     r#cseZdZdZfddZ       ddejdeejdee d ej d ej d ej d ee e e e fdBd ee dBdeeefdBdedee BfddZZS)QwenImageMultiControlNetModela `QwenImageMultiControlNetModel` wrapper class for Multi-QwenImageControlNetModel This module is a wrapper for multiple instances of the `QwenImageControlNetModel`. The `forward()` API is designed to be compatible with `QwenImageControlNetModel`. Args: controlnets (`list[QwenImageControlNetModel]`): Provides additional conditioning to the unet during the denoising process. You must set multiple `QwenImageControlNetModel` as a list. cstt||_dS)N)rArBrGrKnets)r? controlnetsrSr!r"rBs z&QwenImageMultiControlNetModel.__init__NTrcrdrerfrgrhrirjrarkrlc  Cs|dur tdddddt|jdkrN|jd} tt||D],\} \} }| || |||||| | d }| dkr9|}q|durK|durKd d t||D}q|Std ) NrjrmzPassing `txt_seq_lens` to `QwenImageMultiControlNetModel.forward()` is deprecated and will be removed in version 0.39.0. The text sequence length is now automatically inferred from `encoder_hidden_states` and `encoder_hidden_states_mask`.Frnrsr) rcrdrerfrgrhrirarkcSsg|]\}}||qSr!r!)r<control_block_samplerr!r!r"r@Asz9QwenImageMultiControlNetModel.forward..zJQwenImageMultiControlNetModel only supports a single controlnet-union now.)r rOr enumerater ValueError)r?rcrdrerfrgrhrirjrarkr_iimagescalercontrol_block_samplesr!r!r"rs<   z%QwenImageMultiControlNetModel.forward)NNNNNNT)rrr__doc__rBrrrCtensorrrrrrrWrrr}rrrr!r!rSr"rsB      r)( dataclassesrtypingrrtorch.nnrGconfiguration_utilsrrloadersrrutilsr r r r attentionr cache_utilsrcontrolnets.controlnetrmodeling_outputsrmodeling_utilsr"transformers.transformer_qwenimagerrrrr get_loggerrloggerrr#rr!r!r!r"s*           Q