o ۷i @sddlZddlmZddlmZmZddlmZmZm Z ddlm Z m Z m Z  dd e d ejd ed ed ejf ddZdejded eeeffddZGdddeZGdddeZGdddeZGdddeZGdddeZdS)N)QwenImageMultiControlNetModel)ModularPipelineBlocks PipelineState) ComponentSpec InputParam OutputParam)QwenImageLayeredPachifierQwenImageModularPipelineQwenImagePachifier input_name input_tensor batch_sizenum_images_per_promptreturncCszt|tjstd|d|jddkr||}n|jd|kr$|}ntd|d|d|jd|j|dd}|S)a(Repeat tensor elements to match the final batch size. This function expands a tensor's batch dimension to match the final batch size (batch_size * num_images_per_prompt) by repeating each element along dimension 0. The input tensor must have batch size 1 or batch_size. The function will: - If batch size is 1: repeat each element (batch_size * num_images_per_prompt) times - If batch size equals batch_size: repeat each element num_images_per_prompt times Args: input_name (str): Name of the input tensor (used for error messages) input_tensor (torch.Tensor): The tensor to repeat. Must have batch size 1 or batch_size. batch_size (int): The base batch size (number of prompts) num_images_per_prompt (int, optional): Number of images to generate per prompt. Defaults to 1. Returns: torch.Tensor: The repeated tensor with final batch size (batch_size * num_images_per_prompt) Raises: ValueError: If input_tensor is not a torch.Tensor or has invalid batch size Examples: tensor = torch.tensor([[1, 2, 3]]) # shape: [1, 3] repeated = repeat_tensor_to_batch_size("image", tensor, batch_size=2, num_images_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3]]) - shape: [4, 3] tensor = torch.tensor([[1, 2, 3], [4, 5, 6]]) # shape: [2, 3] repeated = repeat_tensor_to_batch_size("image", tensor, batch_size=2, num_images_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [4, 5, 6], [4, 5, 6]]) - shape: [4, 3] `z` must be a tensorrr z!` must have have batch size 1 or z , but got dim) isinstancetorchTensor ValueErrorshaperepeat_interleave)rrrr repeat_byrb/home/ubuntu/vllm_env/lib/python3.10/site-packages/diffusers/modular_pipelines/qwenimage/inputs.pyrepeat_tensor_to_batch_sizes % rlatentsvae_scale_factorcCsN|jdkr|jdkrtd|j|jdd\}}||}||}||fS)aCalculate image dimensions from latent tensor dimensions. This function converts latent space dimensions to image space dimensions by multiplying the latent height and width by the VAE scale factor. Args: latents (torch.Tensor): The latent tensor. Must have 4 or 5 dimensions. Expected shapes: [batch, channels, height, width] or [batch, channels, frames, height, width] vae_scale_factor (int): The scale factor used by the VAE to compress images. Typically 8 for most VAEs (image is 8x larger than latents in each dimension) Returns: tuple[int, int]: The calculated image dimensions as (height, width) Raises: ValueError: If latents tensor doesn't have 4 or 5 dimensions z6unpacked latents must have 4 or 5 dimensions, but got N)ndimrr)r r! latent_height latent_widthheightwidthrrr calculate_dimension_from_latentsPs r*c@steZdZdZdZedefddZedee fddZ edee fdd Z e d d Zd ed edefddZdS)QwenImageTextInputsStepa Text input processing step that standardizes text embeddings for the pipeline. This step: 1. Determines `batch_size` and `dtype` based on `prompt_embeds` 2. Ensures all text embeddings have consistent batch sizes (batch_size * num_images_per_prompt) This block should be placed after all encoder steps to process the text embeddings before they are used in subsequent pipeline steps. Inputs: num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. prompt_embeds (`Tensor`): text embeddings used to guide the image generation. Can be generated from text_encoder step. prompt_embeds_mask (`Tensor`): mask for the text embeddings. Can be generated from text_encoder step. negative_prompt_embeds (`Tensor`, *optional*): negative text embeddings used to guide the image generation. Can be generated from text_encoder step. negative_prompt_embeds_mask (`Tensor`, *optional*): mask for the negative text embeddings. Can be generated from text_encoder step. Outputs: batch_size (`int`): The batch size of the prompt embeddings dtype (`dtype`): The data type of the prompt embeddings prompt_embeds (`Tensor`): The prompt embeddings. (batch-expanded) prompt_embeds_mask (`Tensor`): The encoder attention mask. (batch-expanded) negative_prompt_embeds (`Tensor`): The negative prompt embeddings. (batch-expanded) negative_prompt_embeds_mask (`Tensor`): The negative prompt embeddings mask. (batch-expanded) qwenimagercCsd}d}||S)NzText input processing step that standardizes text embeddings for the pipeline. This step: 1. Determines `batch_size` and `dtype` based on `prompt_embeds` 2. Ensures all text embeddings have consistent batch sizes (batch_size * num_images_per_prompt)z This block should be placed after all encoder steps to process the text embeddings before they are used in subsequent pipeline steps.r)selfsummary_sectionplacement_sectionrrr descriptionsz#QwenImageTextInputsStep.descriptioncCs,tdtdtdtdtdgS)Nr prompt_embedsprompt_embeds_masknegative_prompt_embedsnegative_prompt_embeds_mask)rtemplater-rrrinputss zQwenImageTextInputsStep.inputsc CsNtdtddtdtjddtjdddtjd ddtjd ddtjd ddgS) Nrz'The batch size of the prompt embeddingsname type_hintr0dtypez&The data type of the prompt embeddingsr1zbatch-expanded)noter2r3r4)r intrr;r5r6rrrintermediate_outputss     z,QwenImageTextInputsStep.intermediate_outputscCs|dur |dur td|dur|durtd|jd|jdkr&td|dur8|jd|jdkr8td|durJ|jd|jdkrLtddSdS)NzS`negative_prompt_embeds_mask` is required when `negative_prompt_embeds` is not NonezJcannot pass `negative_prompt_embeds_mask` without `negative_prompt_embeds`rzE`prompt_embeds_mask` must have the same batch size as `prompt_embeds`zI`negative_prompt_embeds` must have the same batch size as `prompt_embeds`zN`negative_prompt_embeds_mask` must have the same batch size as `prompt_embeds`)rrr1r2r3r4rrr check_inputssz$QwenImageTextInputsStep.check_inputs componentsstatecCs*||}|j|j|j|j|jd|jjd|_|jj|_|jj\}}}|j d|j d|_|j |j|j |d|_|j d|j d|_|j |j|j ||_|jdur|jj\}}}|j d|j d|_|j |j|j |d|_|j d|j d|_|j |j|j ||_| ||||fS)Nr?rr ) get_block_stater@r1r2r3r4rrr;repeatrviewset_block_state)r-rArB block_state_seq_lenrrr__call__sD      z QwenImageTextInputsStep.__call__N)__name__ __module__ __qualname____doc__ model_namepropertystrr0listrr7r r> staticmethodr@r rrKrrrrr+ps$   r+ceZdZdZdZ  ddeedBdeedBffdd Zede fd d Z edee fd d Z edeefd dZ edeefddZdededefddZZS)QwenImageAdditionalInputsStepa Input processing step that: 1. For image latent inputs: Updates height/width if None, patchifies, and expands batch size 2. For additional batch inputs: Expands batch dimensions to match final batch size Configured inputs: - Image latent inputs: ['image_latents'] This block should be placed after the encoder steps and the text input step. Components: pachifier (`QwenImagePachifier`) Inputs: num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. height (`int`, *optional*): The height in pixels of the generated image. width (`int`, *optional*): The width in pixels of the generated image. image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. Outputs: image_height (`int`): The image height calculated from the image latents dimension image_width (`int`): The image width calculated from the image latents dimension height (`int`): if not provided, updated to image height width (`int`): if not provided, updated to image width image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. (patchified and batch-expanded) r,Nimage_latent_inputsadditional_batch_inputsc|dur tdg}|durg}t|tstdt||D]}t|ts0tdt|q t|ts?tdt||D]}t|tsQtdt|qA||_||_t dSN image_latentsz,image_latent_inputs must be a list, but got z:image_latent_inputs must be a list of InputParam, but got z0additional_batch_inputs must be a list, but got z>additional_batch_inputs must be a list of InputParam, but got rr5rrSrtype_image_latent_inputs_additional_batch_inputssuper__init__r-rWrX input_param __class__rrra,s*      z&QwenImageAdditionalInputsStep.__init__rcChd}d}|js |jr,d}|jr|ddd|jD7}|jr,|ddd|jD7}d }|||S) NzInput processing step that: 1. For image latent inputs: Updates height/width if None, patchifies, and expands batch size 2. For additional batch inputs: Expands batch dimensions to match final batch size Configured inputs: - Image latent inputs: cSg|]}|jqSrr9.0prrr Wz=QwenImageAdditionalInputsStep.description.. - Additional batch inputs: cSrjrrkrlrrrroYrpN This block should be placed after the encoder steps and the text input step.r^r_r-r. inputs_infor/rrrr0K  z)QwenImageAdditionalInputsStep.descriptioncCtdtddgSN pachifier from_config)default_creation_methodrr r6rrrexpected_components_ z1QwenImageAdditionalInputsStep.expected_componentscC8tdtdtdtdg}||j|j7}|SNrrr(r)rr5r^r_r-r7rrrr7esz$QwenImageAdditionalInputsStep.inputscCtdtddtdtddg}t|jdkr)|tdtdd|td td d|jD]}|t|j|j|jd dq,|jD]}|t|j|j|jd dqA|S) N image_heightrsF    z2QwenImageAdditionalInputsStep.intermediate_outputsrArBc Cs||}|jD]I}|j}t||}|durqt||j\}}|jp#||_|jp)||_t|ds3||_ t|ds;||_ |j |}t |||j|jd}t|||q|jD]}|j} t|| } | durdqUt | | |j|jd} t|| | qU|||||fS)Nrrrrrr)rDr^r9getattrr*r!r(r)hasattrrrry pack_latentsrrrsetattrr_rG r-rArBrHrcimage_latent_input_nameimage_latent_tensorr(r)rrrrrrKsF           z&QwenImageAdditionalInputsStep.__call__NNrLrMrNrOrPrSrrarQrRr0rr}r7r r>r rrK __classcell__rrrdrrVs&(   -rVcrU)%QwenImageEditPlusAdditionalInputsStepa Input processing step for Edit Plus that: 1. For image latent inputs (list): Collects heights/widths, patchifies each, concatenates, expands batch 2. For additional batch inputs: Expands batch dimensions to match final batch size Height/width defaults to last image in the list. Configured inputs: - Image latent inputs: ['image_latents'] This block should be placed after the encoder steps and the text input step. Components: pachifier (`QwenImagePachifier`) Inputs: num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. height (`int`, *optional*): The height in pixels of the generated image. width (`int`, *optional*): The width in pixels of the generated image. image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. Outputs: image_height (`list`): The image heights calculated from the image latents dimension image_width (`list`): The image widths calculated from the image latents dimension height (`int`): if not provided, updated to image height width (`int`): if not provided, updated to image width image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. (patchified, concatenated, and batch-expanded) zqwenimage-edit-plusNrWrXcrYrZr\rbrdrrra*      z.QwenImageEditPlusAdditionalInputsStep.__init__rcCrf) NaInput processing step for Edit Plus that: 1. For image latent inputs (list): Collects heights/widths, patchifies each, concatenates, expands batch 2. For additional batch inputs: Expands batch dimensions to match final batch size Height/width defaults to last image in the list.rgrhricSrjrrkrlrrrro-rpzEQwenImageEditPlusAdditionalInputsStep.description..rqcSrjrrkrlrrrro/rprrrsrtrrrr0 s  z1QwenImageEditPlusAdditionalInputsStep.descriptioncCrwrxr|r6rrrr}5r~z9QwenImageEditPlusAdditionalInputsStep.expected_componentscCrrrrrrrr7;sz,QwenImageEditPlusAdditionalInputsStep.inputscCstdttddtdttddg}t|jdkr-|tdtdd|td td d|jD]}|t|j|j|jd dq0|j D]}|t|j|j|jd dqE|S) Nrz=The image heights calculated from the image latents dimensionr8rzIsF    z:QwenImageEditPlusAdditionalInputsStep.intermediate_outputsrArBcCs\||}|jD]x}|j}t||}|durqt|t}|s!|g}g}g} g} t|D]1\} } t| |j\} }| | | ||j | } t |d| d| |j |jd} | | q+tj| dd} ||_| |_|jpp|d|_|jpx| d|_t||| q|jD]}|j}t||}|durqt |||j |jd}t|||q|||||fS)N[]rr rrC)rDr^r9rrrS enumerater*r!rryrrrrrcatrrr(r)rr_rG)r-rArBrHrcrris_list image_heights image_widthspacked_image_latent_tensorsiimg_latent_tensorr(r)rrrrrrKwsX           z.QwenImageEditPlusAdditionalInputsStep.__call__rrrrrdrrs&)   -rcrU)$QwenImageLayeredAdditionalInputsStepaA Input processing step for Layered that: 1. For image latent inputs: Updates height/width if None, patchifies with layered pachifier, and expands batch size 2. For additional batch inputs: Expands batch dimensions to match final batch size Configured inputs: - Image latent inputs: ['image_latents'] This block should be placed after the encoder steps and the text input step. Components: pachifier (`QwenImageLayeredPachifier`) Inputs: num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. Outputs: image_height (`int`): The image height calculated from the image latents dimension image_width (`int`): The image width calculated from the image latents dimension height (`int`): if not provided, updated to image height width (`int`): if not provided, updated to image width image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. (patchified with layered pachifier and batch-expanded) zqwenimage-layeredNrWrXcrYrZr\rbrdrrrarz-QwenImageLayeredAdditionalInputsStep.__init__rcCrf) NzInput processing step for Layered that: 1. For image latent inputs: Updates height/width if None, patchifies with layered pachifier, and expands batch size 2. For additional batch inputs: Expands batch dimensions to match final batch sizergrhricSrjrrkrlrrrrorpzDQwenImageLayeredAdditionalInputsStep.description..rqcSrjrrkrlrrrrorprrrsrtrrrr0rvz0QwenImageLayeredAdditionalInputsStep.descriptioncCrwrx)rr r6rrrr}r~z8QwenImageLayeredAdditionalInputsStep.expected_componentscCs(tdtdg}||j|j7}|S)Nrrrrrrrr7 s z+QwenImageLayeredAdditionalInputsStep.inputscCr) Nrrr8rrrr(rr)rz7 (patchified with layered pachifier and batch-expanded)rrrrrrr>,sF     z9QwenImageLayeredAdditionalInputsStep.intermediate_outputsrArBc Cs||}|jD]K}|j}t||}|durq|jd|j}|jd|j}||_||_t|ds5||_ t|ds=||_ |j |}t |||j|jd}t|||q|jD]}|j} t|| } | durfqWt | | |j|jd} t|| | qW|||||fS)Nrr"rrr)rDr^r9rrr!r(r)rrrryrrrrrr_rGrrrrrKYsH         z-QwenImageLayeredAdditionalInputsStep.__call__rrrrrdrrs&%   ,rc@speZdZdZdZedefddZedee fddZ edee fdd Z e d ed edefd d ZdS)QwenImageControlNetInputsStepay prepare the `control_image_latents` for controlnet. Insert after all the other inputs steps. Inputs: control_image_latents (`Tensor`): The control image latents to use for the denoising process. Can be generated in controlnet vae encoder step. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. height (`int`, *optional*): The height in pixels of the generated image. width (`int`, *optional*): The width in pixels of the generated image. Outputs: control_image_latents (`Tensor`): The control image latents (patchified and batch-expanded). height (`int`): if not provided, updated to control image height width (`int`): if not provided, updated to control image width r,rcCsdS)Nz\prepare the `control_image_latents` for controlnet. Insert after all the other inputs steps.rr6rrrr0sz)QwenImageControlNetInputsStep.descriptioncCs4tddtjddtdtdtdtdgS) Ncontrol_image_latentsTzlThe control image latents to use for the denoising process. Can be generated in controlnet vae encoder step.)r9requiredr:r0rrr(r))rrrr5r6rrrr7sz$QwenImageControlNetInputsStep.inputscCs*tdtjddtdtddtdtddgS)Nrz:The control image latents (patchified and batch-expanded).r8r(z0if not provided, updated to control image heightr)z/if not provided, updated to control image width)r rrr=r6rrrr>s  z2QwenImageControlNetInputsStep.intermediate_outputsrArBc Cs||}t|jtrHg}t|jD]1\}}t||j\}}|jp"||_|j p(||_ |j |}t d|d||j |jd}||q||_n-t|j|j\}}|jpU||_|j p[||_ |j |j|_t d|j|j |jd|_|j|_|||||fS)Nzcontrol_image_latents[rrr)rDr controlnetrrrr*r!r(r)ryrrrrrrG) r-rArBrHrrcontrol_image_latents_r(r)rrrrKs>          z&QwenImageControlNetInputsStep.__call__N)rLrMrNrOrPrQrRr0rSrr7r r>rno_gradr rrKrrrrrs r)r )rmodelsrmodular_pipelinerrmodular_pipeline_utilsrrr r r r rRrr=rtupler*r+rVrrrrrrrs6  8 VkR