o Giq^ @sJddlZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z dd l mZmZmZd d l mZeeZ d&d ed ejdededejf ddZdejdededeeeffddZ    d'dedBdeejBdBdeedBdeedBfddZGddde ZGd d!d!e Z Gd"d#d#e Z!Gd$d%d%e Z"dS)(N)WanTransformer3DModel)UniPCMultistepScheduler)logging) randn_tensor)ModularPipelineBlocks PipelineState) ComponentSpec InputParam OutputParam)WanModularPipeline input_name input_tensor batch_sizenum_videos_per_promptreturncCszt|tjstd|d|jddkr||}n|jd|kr$|}ntd|d|d|jd|j|dd}|S)a(Repeat tensor elements to match the final batch size. This function expands a tensor's batch dimension to match the final batch size (batch_size * num_videos_per_prompt) by repeating each element along dimension 0. The input tensor must have batch size 1 or batch_size. The function will: - If batch size is 1: repeat each element (batch_size * num_videos_per_prompt) times - If batch size equals batch_size: repeat each element num_videos_per_prompt times Args: input_name (str): Name of the input tensor (used for error messages) input_tensor (torch.Tensor): The tensor to repeat. Must have batch size 1 or batch_size. batch_size (int): The base batch size (number of prompts) num_videos_per_prompt (int, optional): Number of videos to generate per prompt. Defaults to 1. Returns: torch.Tensor: The repeated tensor with final batch size (batch_size * num_videos_per_prompt) Raises: ValueError: If input_tensor is not a torch.Tensor or has invalid batch size Examples: tensor = torch.tensor([[1, 2, 3]]) # shape: [1, 3] repeated = repeat_tensor_to_batch_size("image", tensor, batch_size=2, num_videos_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3]]) - shape: [4, 3] tensor = torch.tensor([[1, 2, 3], [4, 5, 6]]) # shape: [2, 3] repeated = repeat_tensor_to_batch_size("image", tensor, batch_size=2, num_videos_per_prompt=2) repeated # tensor([[1, 2, 3], [1, 2, 3], [4, 5, 6], [4, 5, 6]]) - shape: [4, 3] `z` must be a tensorrr z!` must have have batch size 1 or , but got )dim) isinstancetorchTensor ValueErrorshaperepeat_interleave)rrrr repeat_byrb/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/modular_pipelines/wan/before_denoise.pyrepeat_tensor_to_batch_size%s % r latentsvae_scale_factor_temporalvae_scale_factor_spatialc CsT|jdkr td|j|j\}}}}}|d|d}||}||} ||| fS)aCalculate image dimensions from latent tensor dimensions. This function converts latent temporal and spatial dimensions to image temporal and spatial dimensions by multiplying the latent num_frames/height/width by the VAE scale factor. Args: latents (torch.Tensor): The latent tensor. Must have 4 or 5 dimensions. Expected shapes: [batch, channels, height, width] or [batch, channels, frames, height, width] vae_scale_factor_temporal (int): The scale factor used by the VAE to compress temporal dimension. Typically 4 for most VAEs (video is 4x larger than latents in temporal dimension) vae_scale_factor_spatial (int): The scale factor used by the VAE to compress spatial dimension. Typically 8 for most VAEs (image is 8x larger than latents in each dimension) Returns: tuple[int, int]: The calculated image dimensions as (height, width) Raises: ValueError: If latents tensor doesn't have 4 or 5 dimensions z(latents must have 5 dimensions, but got r )ndimrr) r!r"r#_num_latent_frames latent_height latent_width num_framesheightwidthrrr calculate_dimension_from_latents]s  r-num_inference_stepsdevice timestepssigmascKs|dur |dur td|dur>dtt|jjv}|s(td|jd|jd ||d||j}t |}||fS|durpdtt|jjv}|sZtd|jd|jd ||d ||j}t |}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`list[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesr0zThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)r0r/r1zv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)r1r/r/r) rsetinspect signature set_timesteps parameterskeys __class__r0len) schedulerr.r/r0r1kwargsaccepts_timesteps accept_sigmasrrrretrieve_timestepss2  r>c@seZdZdZedefddZedeefddZ edee fddZ edeefd d Z d d Z ed ededefddZdS)WanTextInputStepwanrcCs dS)NaInput processing step that: 1. Determines `batch_size` and `dtype` based on `prompt_embeds` 2. Adjusts input tensor shapes based on `batch_size` (number of prompts) and `num_videos_per_prompt` All input tensors are expected to have either batch_size=1 or match the batch_size of prompt_embeds. The tensors will be duplicated across the batch dimension to have a final batch_size of batch_size * num_videos_per_prompt.rselfrrr descriptionszWanTextInputStep.descriptioncC tdtgS)N transformer)r rrArrrexpected_componentsz$WanTextInputStep.expected_componentscCs,tdddtddtjddtdtjd d gS) Nrr default prompt_embedsTzGPre-generated text embeddings. Can be generated from text_encoder step.required type_hintrCnegative_prompt_embedszPPre-generated negative text embeddings. Can be generated from text_encoder step.rMrC)r rrrArrrinputss zWanTextInputStep.inputscCstdtddtdtjddgS)NrzdNumber of prompts, the final batch size of model inputs should be batch_size * num_videos_per_promptrOdtypezDData type of model tensor inputs (determined by `transformer.dtype`))r intrrQrArrrintermediate_outputssz%WanTextInputStep.intermediate_outputscCsN|jdur!|jdur#|jj|jjkr%td|jjd|jjddSdSdS)Nzu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` .)rJrNrr)rB components block_staterrr check_inputsszWanTextInputStep.check_inputsrUstatecCs||}||||jjd|_|jj|_|jj\}}}|jd|jd|_|j|j|j|d|_|j durX|j j\}}}|j d|jd|_ |j |j|j|d|_ | ||||fS)Nrr ) get_block_staterWrJrrrQrepeatrviewrNset_block_state)rBrUrXrVr&seq_lenrrr__call__s&     zWanTextInputStep.__call__N)__name__ __module__ __qualname__ model_namepropertystrrClistr rFr rPrSrWrno_gradrr r_rrrrr?s  r?csxeZdZdZdggfdeedeeffdd Zedefdd Zedee fd d Z d e d e de fddZ ZS)WanAdditionalInputsStepr@image_condition_latentsimage_latent_inputsadditional_batch_inputscs:t|ts|g}t|ts|g}||_||_tdS)aInitialize a configurable step that standardizes the inputs for the denoising step. It: " This step handles multiple common tasks to prepare inputs for the denoising step: 1. For encoded image latents, use it update height/width if None, and expands batch size 2. For additional_batch_inputs: Only expands batch dimensions to match final batch size This is a dynamic block that allows you to configure which inputs to process. Args: image_latent_inputs (list[str], optional): Names of image latent tensors to process. In additional to adjust batch size of these inputs, they will be used to determine height/width. Can be a single string or list of strings. Defaults to ["image_condition_latents"]. additional_batch_inputs (List[str], optional): Names of additional conditional input tensors to expand batch size. These tensors will only have their batch dimensions adjusted to match the final batch size. Can be a single string or list of strings. Defaults to []. Examples: # Configure to process image_condition_latents (default behavior) WanAdditionalInputsStep() # Configure to process image latents and additional batch inputs WanAdditionalInputsStep( image_latent_inputs=["image_condition_latents"], additional_batch_inputs=["image_embeds"] ) N)rrf_image_latent_inputs_additional_batch_inputssuper__init__)rBrjrkr8rrros  z WanAdditionalInputsStep.__init__rcCsTd}d}|js |jr"d}|jr|d|j7}|jr"|d|j7}d}|||S)NzInput processing step that: 1. For image latent inputs: Updates height/width if None, and expands batch size 2. For additional batch inputs: Expands batch dimensions to match final batch sizez Configured inputs:z - Image latent inputs: z - Additional batch inputs: zN This block should be placed after the encoder steps and the text input step.)rlrm)rBsummary_section inputs_infoplacement_sectionrrrrC=s  z#WanAdditionalInputsStep.descriptioncCsltdddtdddtddtd dtd dg}|jD] }|t|dq|jD] }|t|dq)|S) Nrr )namerIrT)rurLr+)rur,r*)r rlappendrm)rBrPimage_latent_input_namerrrrrPTs    zWanAdditionalInputsStep.inputsrUrXc Cs||}|jD]9}t||}|durqt||j|j\}}}|jp#||_|jp)||_|jp/||_t |||j |j d}t |||q|j D]} t|| } | durQqEt | | |j |j d} t || | qE|||||fS)N)rrrr)rZrlgetattrr-r"r#r*r+r,r rrsetattrrmr]) rBrUrXrVrwimage_latent_tensorr*r+r,rrrrrr_hs>          z WanAdditionalInputsStep.__call__)r`rarbrcrfrerordrCr rPrr r_ __classcell__rrrprrhs%rhc@sleZdZdZedeefddZedefddZ edee fddZ e d ed edefd d Zd S)WanSetTimestepsStepr@rcCrD)Nr:)r rrArrrrFrGz'WanSetTimestepsStep.expected_componentscCdS)Nz6Step that sets the scheduler's timesteps for inferencerrArrrrCzWanSetTimestepsStep.descriptioncCstdddtdtdgS)Nr.2rHr0r1)r rArrrrPs zWanSetTimestepsStep.inputsrUrXcCsD||}|j}t|j|j||j|j\|_|_|||||fS)N)rZ_execution_devicer>r:r.r0r1r])rBrUrXrVr/rrrr_s  zWanSetTimestepsStep.__call__N)r`rarbrcrdrfr rFrerCr rPrrgrr r_rrrrr|sr|c@seZdZdZedefddZedeefddZ edee fddZ e d d Z e     ddedededededejdBdejdBdejeejBdBdejdBdejfddZedededefddZdS) WanPrepareLatentsStepr@rcCr})NzWPrepare latents step that prepares the latents for the text-to-video generation processrrArrrrCr~z!WanPrepareLatentsStep.descriptionc Cs`tdtdtdtdtdtdtdtjdBdtdtddtd td d td d tdtjddgS)Nr+)rMr,r*r!rr )rMrI generatorrTzNumber of prompts, the final batch size of model inputs should be `batch_size * num_videos_per_prompt`. Can be generated in input step.rKrQzThe dtype of the model inputsrO)r rRrrrQrArrrrPs    zWanPrepareLatentsStep.inputscCstdtjddgS)Nr!z4The initial latents to use for the denoising processrO)r rrrArrrrSsz*WanPrepareLatentsStep.intermediate_outputscCs|jdur |j|jdks|jdur+|j|jdkr+td|jd|jd|jd|jdurL|jdks?|jd|jdkrNtd|jd|jddSdS) Nrz-`height` and `width` have to be divisible by z but are z and rTr zQ`num_frames` has to be greater than 0, and (num_frames - 1) must be divisible by r)r+r#r,rr*r")rUrVrrrrWs z"WanPrepareLatentsStep.check_inputs@QNrnum_channels_latentsr+r,r*rQr/rr!c Cs| dur | j||dS|d|jd} ||| t||jt||jf} t|tr=t||kr=tdt|d|dt| |||d} | S)N)r/rQr z/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.)rr/rQ) tor"rRr#rrfr9rr) comprrr+r,r*rQr/rr!r'rrrrprepare_latentss"   z%WanPrepareLatentsStep.prepare_latentsrUrXc Cs||}||||j}tj}|jp|j|_|jp|j|_|j p$|j |_ |j ||j |j |j|j|j|j |||j|jd |_|||||fS)N) rrr+r,r*rQr/rr!)rZrWrrfloat32r+default_heightr, default_widthr*default_num_framesrrrrrr!r])rBrUrXrVr/rQrrrr_s*    zWanPrepareLatentsStep.__call__)rrrrNNNN)r`rarbrcrdrerCrfr rPr rS staticmethodrWrRrrQr/ Generatorrrrgrr r_rrrrrsT     r)r )NNNN)#r3rmodelsr schedulersrutilsrutils.torch_utilsrmodular_pipelinerr modular_pipeline_utilsr r r r get_loggerr`loggerrerrRr tupler-r/rffloatr>r?rhr|rrrrrsb        8  &    ;Y&