o Gi} @sddlZddlZddlZddlmZmZddlmZddl m Z m Z ddl m Z mZddlmZmZmZd d l mZmZmZ d5dedededefddZ    d6dedBdeejBdBdeedBdeedBfddZddZGddde ZGdd d e ZGd!d"d"e Z Gd#d$d$e Z!Gd%d&d&e Z"Gd'd(d(e Z#Gd)d*d*e Z$Gd+d,d,e Z%Gd-d.d.e Z&Gd/d0d0e Z'Gd1d2d2e Z(Gd3d4d4e Z)dS)7N)QwenImageControlNetModelQwenImageMultiControlNetModel)FlowMatchEulerDiscreteScheduler) randn_tensor unwrap_module)ModularPipelineBlocks PipelineState) ComponentSpec InputParam OutputParam)QwenImageLayeredPachifierQwenImageModularPipelineQwenImagePachifier?ffffff? base_seq_len max_seq_len base_shift max_shiftcCs,||||}|||}|||}|S)N) image_seq_lenrrrrmbmurrh/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/modular_pipelines/qwenimage/before_denoise.pycalculate_shifts  r num_inference_stepsdevice timestepssigmascKs|dur |dur td|dur>dtt|jjv}|s(td|jd|jd ||d||j}t |}||fS|durpdtt|jjv}|sZtd|jd|jd ||d ||j}t |}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`list[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr#zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r#r"r$zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r$r"r"r) ValueErrorsetinspect signature set_timesteps parameterskeys __class__r#len) schedulerr!r"r#r$kwargsaccepts_timesteps accept_sigmasrrrretrieve_timesteps+s2  r2cCsZt|||}tt||d}|j||jd}t|dr'|||j|||fS)Nrset_begin_index)minintmaxr#orderhasattrr3)r.r!strength init_timestept_startr#rrr get_timestepsgs   r<c@eZdZdZdZedefddZedee fddZ edee fdd Z edee fd d Zed d ZedededefddZdS)QwenImagePrepareLatentsStepa Prepare initial random noise for the generation process Components: pachifier (`QwenImagePachifier`) Inputs: latents (`Tensor`, *optional*): Pre-generated noisy latents for image generation. height (`int`, *optional*): The height in pixels of the generated image. width (`int`, *optional*): The width in pixels of the generated image. num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`Generator`, *optional*): Torch generator for deterministic generation. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. dtype (`dtype`, *optional*, defaults to torch.float32): The dtype of the model inputs, can be generated in input step. Outputs: height (`int`): if not set, updated to default value width (`int`): if not set, updated to default value latents (`Tensor`): The initial latents to use for the denoising process qwenimagereturncCdS)Nz7Prepare initial random noise for the generation processrselfrrr descriptionz'QwenImagePrepareLatentsStep.descriptioncCtdtddgSN pachifier from_config)default_creation_methodr rrBrrrexpected_components z/QwenImagePrepareLatentsStep.expected_componentsc Cs<tdtdtdtdtdtdtdgS)Nlatentsheightwidthnum_images_per_prompt generator batch_sizedtyper templaterBrrrinputssz"QwenImagePrepareLatentsStep.inputscC*tdtddtdtddtdtjddgSNrOz$if not set, updated to default valuename type_hintrDrPrNz4The initial latents to use for the denoising processr r5torchTensorrBrrrintermediate_outputs  z0QwenImagePrepareLatentsStep.intermediate_outputscCh|dur||ddkrtd|dd||dur0||ddkr2td|dd|dSdSNrrzHeight must be divisible by z but is zWidth must be divisible by r%rOrPvae_scale_factorrrr check_inputs z(QwenImagePrepareLatentsStep.check_inputs componentsstatec Cs||}|j|j|j|jd|j}|j|j}|jp|j|_|jp%|j |_dt |j|jd}dt |j|jd}||j d||f}t |j trbt|j |krbtdt|j d|d|jdurzt||j ||jd|_|j|j|_|||||fSNrerrz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rRr"rT)get_block_statergrOrPrf_execution_devicerSrQdefault_height default_widthr5num_channels_latents isinstancerRlistr-r%rNrrTrH pack_latentsset_block_state rCrirj block_stater"rS latent_height latent_widthshaperrr__call__s4     z$QwenImagePrepareLatentsStep.__call__N__name__ __module__ __qualname____doc__ model_namepropertystrrDrrr rLr rWr r` staticmethodrgr^no_gradrr rzrrrrr>ys   r>c@r=)"QwenImageLayeredPrepareLatentsStepa Prepare initial random noise (B, layers+1, C, H, W) for the generation process Components: pachifier (`QwenImageLayeredPachifier`) Inputs: latents (`Tensor`, *optional*): Pre-generated noisy latents for image generation. height (`int`, *optional*): The height in pixels of the generated image. width (`int`, *optional*): The width in pixels of the generated image. layers (`int`, *optional*, defaults to 4): Number of layers to extract from the image num_images_per_prompt (`int`, *optional*, defaults to 1): The number of images to generate per prompt. generator (`Generator`, *optional*): Torch generator for deterministic generation. batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. dtype (`dtype`, *optional*, defaults to torch.float32): The dtype of the model inputs, can be generated in input step. Outputs: height (`int`): if not set, updated to default value width (`int`): if not set, updated to default value latents (`Tensor`): The initial latents to use for the denoising process qwenimage-layeredr@cCrA)NzNPrepare initial random noise (B, layers+1, C, H, W) for the generation processrrBrrrrDrEz.QwenImageLayeredPrepareLatentsStep.descriptioncCrFrG)r rrBrrrrLrMz6QwenImageLayeredPrepareLatentsStep.expected_componentsc CsDtdtdtdtdtdtdtdtdgS) NrNrOrPlayersrQrRrSrTrUrBrrrrWsz)QwenImageLayeredPrepareLatentsStep.inputscCrXrYr]rBrrrr`)raz7QwenImageLayeredPrepareLatentsStep.intermediate_outputscCrbrcrdrerrrrg5rhz/QwenImageLayeredPrepareLatentsStep.check_inputsrirjc Cs||}|j|j|j|jd|j}|j|j}|jp|j|_|jp%|j |_dt |j|jd}dt |j|jd}||j d|j ||f}t |jtret|j|kretdt|jd|d|jdur}t||j||jd|_|j|j|_|||||fSrk)rlrgrOrPrfrmrSrQrnror5rrprqrRrrr-r%rNrrTrHrsrtrurrrrz=s4     z+QwenImageLayeredPrepareLatentsStep.__call__Nr{rrrrrs"  rc@r=)'QwenImagePrepareLatentsWithStrengthStepa Step that adds noise to image latents for image-to-image/inpainting. Should be run after set_timesteps, prepare_latents. Both noise and image latents should alreadybe patchified. Components: scheduler (`FlowMatchEulerDiscreteScheduler`) Inputs: latents (`Tensor`): The initial random noised, can be generated in prepare latent step. image_latents (`Tensor`): image latents used to guide the image generation. Can be generated from vae_encoder step. (Can be generated from vae encoder and updated in input step.) timesteps (`Tensor`): The timesteps to use for the denoising process. Can be generated in set_timesteps step. Outputs: initial_noise (`Tensor`): The initial random noised used for inpainting denoising. latents (`Tensor`): The scaled noisy latents to use for inpainting/image-to-image denoising. r?r@cCrA)NzStep that adds noise to image latents for image-to-image/inpainting. Should be run after set_timesteps, prepare_latents. Both noise and image latents should alreadybe patchified.rrBrrrrD~rEz3QwenImagePrepareLatentsWithStrengthStep.descriptioncC tdtgSNr.r rrBrrrrLz;QwenImagePrepareLatentsWithStrengthStep.expected_componentscCs0tddtjddtjdddtddtjd dgS) NrNTzCThe initial random noised, can be generated in prepare latent step.r[requiredr\rD image_latentsz<sz.dim)rlrfrOrPziprrrSrrsumtolistrrrrtrrrrrz3s&      z(QwenImageEditPlusRoPEInputsStep.__call__Nrrrrrrsrc@sleZdZdZdZedefddZedee fddZ edee fdd Z e d edefd d Zd S)QwenImageLayeredRoPEInputsStepae Step that prepares the RoPE inputs for the denoising process. Should be place after prepare_latents step Inputs: batch_size (`int`, *optional*, defaults to 1): Number of prompts, the final batch size of model inputs should be batch_size * num_images_per_prompt. Can be generated in input step. layers (`int`, *optional*, defaults to 4): Number of layers to extract from the image height (`int`): The height in pixels of the generated image. width (`int`): The width in pixels of the generated image. prompt_embeds_mask (`Tensor`): mask for the text embeddings. Can be generated from text_encoder step. negative_prompt_embeds_mask (`Tensor`, *optional*): mask for the negative text embeddings. Can be generated from text_encoder step. Outputs: img_shapes (`list`): The shapes of the image latents, used for RoPE calculation txt_seq_lens (`list`): The sequence lengths of the prompt embeds, used for RoPE calculation negative_txt_seq_lens (`list`): The sequence lengths of the negative prompt embeds, used for RoPE calculation additional_t_cond (`Tensor`): The additional t cond, used for RoPE calculation rr@cCrrrrBrrrrDrrz*QwenImageLayeredRoPEInputsStep.descriptioncCs<tdtdtjdddtjdddtdtdgS) NrSrrOTrrPrrrUrBrrrrWxs  z%QwenImageLayeredRoPEInputsStep.inputsc CsXtdttttttfdddtdttdddtdttdddtd tjdd dgS) Nrrr)r[r\rrDrrrradditional_t_condz0The additional t cond, used for RoPE calculation)r rrrr5r^r_rBrrrr`s2z3QwenImageLayeredRoPEInputsStep.intermediate_outputsrjcCs||}|j}d|j|jd|j|jdf}|g|jdg|j|_|jdur4|jj dd nd|_ |j durE|j j dd nd|_ tdg|jj|tjd|_|||||fS)Nrrrrr)rlrmrOrfrPrrSrrrrrrrr^tensorrlongrrt)rCrirjrvr"ryrrrrzs    z'QwenImageLayeredRoPEInputsStep.__call__N)r|r}r~rrrrrDrrr rWr r`r^rr rzrrrrrRs rc@seZdZdZdZedeefddZede fddZ edee fdd Z edee fd d Zed ed edefddZdS)%QwenImageControlNetBeforeDenoiserStepa step that prepare inputs for controlnet. Insert before the Denoise Step, after set_timesteps step. Components: controlnet (`QwenImageControlNetModel`) Inputs: control_guidance_start (`float`, *optional*, defaults to 0.0): When to start applying ControlNet. control_guidance_end (`float`, *optional*, defaults to 1.0): When to stop applying ControlNet. controlnet_conditioning_scale (`float`, *optional*, defaults to 1.0): Scale for ControlNet conditioning. control_image_latents (`Tensor`): The control image latents to use for the denoising process. Can be generated in controlnet vae encoder step. timesteps (`Tensor`): The timesteps to use for the denoising process. Can be generated in set_timesteps step. Outputs: controlnet_keep (`list`): The controlnet keep values r?r@cCr)N controlnet)r rrBrrrrLrz9QwenImageControlNetBeforeDenoiserStep.expected_componentscCrA)Nzbstep that prepare inputs for controlnet. Insert before the Denoise Step, after set_timesteps step.rrBrrrrDrEz1QwenImageControlNetBeforeDenoiserStep.descriptionc Cs<tdtdtdtddtjddtddtjd dgS) Ncontrol_guidance_startcontrol_guidance_endcontrolnet_conditioning_scalecontrol_image_latentsTzlThe control image latents to use for the denoising process. Can be generated in controlnet vae encoder step.rr#rrrBrrrrWs z,QwenImageControlNetBeforeDenoiserStep.inputscCstdttddgS)Ncontrolnet_keepzThe controlnet keep values)r\rD)r rrfloatrBrrrr`sz:QwenImageControlNetBeforeDenoiserStep.intermediate_outputsrirjcsL||t|j}tjts!tjtr!tjjg_n>tjts8tjtr8tjjg_n'tjts_tjts_t|trNtj nd}|jg|jg__t|trqtj t rqj g|_ g_ t tjD] fddtjjD}j t|tr|dn|q{||||fS)Nrcs@g|]\}}dttj|kpdtj|kqS)rr)rr-r#)rservirrr*s.zBQwenImageControlNetBeforeDenoiserStep.__call__..r)rlrrrqrrrrr-rrrrrranger#rappendrrt)rCrirjrmultkeepsrrrrzsD           z.QwenImageControlNetBeforeDenoiserStep.__call__N)r|r}r~rrrrrr rLrrDr rWr r`r^rrr rzrrrrrsr)rrrr)NNNN)*r'numpyrr^modelsrr schedulersrutils.torch_utilsrrmodular_pipeliner r modular_pipeline_utilsr r r rrrr5rr rr"rrr2r<r>rrrrrrrrrrrrrrrs`      <twiZVPnF[vr