o Gi" @sddlZddlZddlmZddlmZmZddlZddl Z ddl Z ddl m Z mZmZddlmZmZddlmZmZddlmZmZdd lmZdd lmZdd lmZmZm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,erddl-m.m/Z0dZ1ndZ1e2e3Z4dZ5eGdddZ6 d4de j7de j8dBde9fddZ:  ! "d5d#e;d$e;d%eBdBd+e?e;dBd,e?e<dBfd-d.Z@d7d0d1ZAGd2d3d3e&eeZBdS)8N) dataclass)AnyCallable)Gemma3ForConditionalGenerationGemmaTokenizerGemmaTokenizerFast)MultiPipelineCallbacksPipelineCallback)FromSingleFileMixinLTX2LoraLoaderMixin)AutoencoderKLLTX2AudioAutoencoderKLLTX2Video)LTX2VideoTransformer3DModel)FlowMatchEulerDiscreteScheduler)is_torch_xla_availableloggingreplace_example_docstring) randn_tensor)VideoProcessor)DiffusionPipeline)LTX2TextConnectors)LTX2PipelineOutput) LTX2VocoderTFaZ Examples: ```py >>> import torch >>> from diffusers import LTX2ConditionPipeline >>> from diffusers.pipelines.ltx2.export_utils import encode_video >>> from diffusers.pipelines.ltx2.pipeline_ltx2_condition import LTX2VideoCondition >>> from diffusers.utils import load_image >>> pipe = LTX2ConditionPipeline.from_pretrained("Lightricks/LTX-2", torch_dtype=torch.bfloat16) >>> pipe.enable_model_cpu_offload() >>> first_image = load_image( ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_first_frame.png" ... ) >>> last_image = load_image( ... "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/flf2v_input_last_frame.png" ... ) >>> first_cond = LTX2VideoCondition(frames=first_image, index=0, strength=1.0) >>> last_cond = LTX2VideoCondition(frames=last_image, index=-1, strength=1.0) >>> conditions = [first_cond, last_cond] >>> prompt = "CG animation style, a small blue bird takes off from the ground, flapping its wings." >>> negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted, static" >>> frame_rate = 24.0 >>> video = pipe( ... conditions=conditions, ... prompt=prompt, ... negative_prompt=negative_prompt, ... width=768, ... height=512, ... num_frames=121, ... frame_rate=frame_rate, ... num_inference_steps=40, ... guidance_scale=4.0, ... output_type="np", ... return_dict=False, ... ) >>> video = (video * 255).round().astype("uint8") >>> video = torch.from_numpy(video) >>> encode_video( ... video[0], ... fps=frame_rate, ... audio=audio[0].float().cpu(), ... audio_sample_rate=pipe.vocoder.config.output_sampling_rate, # should be 24000 ... output_path="video.mp4", ... ) ``` c@sNeZdZUdZejjeejjBejBe j Be d<dZ e e d<dZee d<dS)LTX2VideoConditiona Defines a single frame-conditioning item for LTX-2 Video - a single frame or a sequence of frames. Attributes: frames (`PIL.Image.Image` or `List[PIL.Image.Image]` or `np.ndarray` or `torch.Tensor`): The image (or video) to condition the video on. Accepts any type that can be handled by VideoProcessor.preprocess_video. index (`int`, defaults to `0`): The index at which the image or video will conditionally affect the video generation. strength (`float`, defaults to `1.0`): The strength of the conditioning effect. A value of `1.0` means the conditioning effect is fully applied. framesrindex?strengthN)__name__ __module__ __qualname____doc__PILImagelistnpndarraytorchTensor__annotations__rintr floatr/r/d/home/ubuntu/.local/lib/python3.10/site-packages/diffusers/pipelines/ltx2/pipeline_ltx2_condition.pyrds $ rsampleencoder_output generator sample_modecCsRt|dr|dkr|j|St|dr|dkr|jSt|dr%|jStd)N latent_distr1argmaxlatentsz3Could not access latents of provided encoder_output)hasattrr5r1moder7AttributeError)r2r3r4r/r/r0retrieve_latentsys   r;?ffffff? base_seq_len max_seq_len base_shift max_shiftcCs,||||}|||}|||}|SNr/) image_seq_lenr@rArBrCmbmur/r/r0calculate_shifts  rInum_inference_stepsdevice timestepssigmascKs|dur |dur td|dur>dtt|jjv}|s(td|jd|jd ||d||j}t |}||fS|durpdtt|jjv}|sZtd|jd|jd ||d ||j}t |}||fS|j|fd |i||j}||fS) a Calls the scheduler's `set_timesteps` method and retrieves timesteps from the scheduler after the call. Handles custom timesteps. Any kwargs will be supplied to `scheduler.set_timesteps`. Args: scheduler (`SchedulerMixin`): The scheduler to get timesteps from. num_inference_steps (`int`): The number of diffusion steps used when generating samples with a pre-trained model. If used, `timesteps` must be `None`. device (`str` or `torch.device`, *optional*): The device to which the timesteps should be moved to. If `None`, the timesteps are not moved. timesteps (`list[int]`, *optional*): Custom timesteps used to override the timestep spacing strategy of the scheduler. If `timesteps` is passed, `num_inference_steps` and `sigmas` must be `None`. sigmas (`list[float]`, *optional*): Custom sigmas used to override the timestep spacing strategy of the scheduler. If `sigmas` is passed, `num_inference_steps` and `timesteps` must be `None`. Returns: `tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the second element is the number of inference steps. NzYOnly one of `timesteps` or `sigmas` can be passed. Please choose one to set custom valuesrLzThe current scheduler class zx's `set_timesteps` does not support custom timestep schedules. Please check whether you are using the correct scheduler.)rLrKrMzv's `set_timesteps` does not support custom sigmas schedules. Please check whether you are using the correct scheduler.)rMrKrKr/) ValueErrorsetinspect signature set_timesteps parameterskeys __class__rLlen) schedulerrJrKrLrMkwargsaccepts_timesteps accept_sigmasr/r/r0retrieve_timestepss2  r[cCsX|jttd|jdd}|jttd|jdd}|||}||d||}|S)a Rescales `noise_cfg` tensor based on `guidance_rescale` to improve image quality and fix overexposure. Based on Section 3.4 from [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). Args: noise_cfg (`torch.Tensor`): The predicted noise tensor for the guided diffusion process. noise_pred_text (`torch.Tensor`): The predicted noise tensor for the text-guided diffusion process. guidance_rescale (`float`, *optional*, defaults to 0.0): A rescale factor applied to the noise predictions. Returns: noise_cfg (`torch.Tensor`): The rescaled noise prediction tensor. rTdimkeepdim)stdr'rangendim) noise_cfgnoise_pred_textguidance_rescalestd_textstd_cfgnoise_pred_rescaledr/r/r0rescale_noise_cfgs  ric=s`eZdZdZdZgZgdZdedede de de e Bd e d ed effd d Ze   ddejdejdeejBdedededejfddZ     ddeeeBdedededejdBd ejdBf d!d"Z  #         ddeeeBd$eeeBdBd%eded&ejdBd'ejdBd(ejdBd)ejdBdededejdBd ejdBfd*d+Z       dd,d-Zedd.ejd/ed0edejfd1d2Ze dd.ejd3ed4ed5ed/ed0edejfd6d7Z e 8dd.ejd9ejd:ejd;edejf dd?Z"e dd.ejd@eejBdAej#dBfdBdCZ$e dd.ejd/edBd0edBdejfdDdEZ%e  dd.ejdFedGed/edBd0edBdejf dHdIZ&ed.ejd9ejd:ejfdJdKZ'ed.ejd9ejd:ejfdLdMZ(dNedOedPedefdQdRZ)  S T U ddVe*ee*BdBd4ed5ed3edejdBde+eejeeeeff dWdXZ,d.ejdYejdZeejd[eed\eed]ed^ede+ejejejffd_d`Z-   a S T U 8    ddVe*ee*BdBdbedced4ed5ed3ed@ed ejdBdejdBdAej#dBd.ejdBde+ejejejffdddeZ.    f g    ddbedcedhedGed@ed ejdBdejdBdAej#dBd.ejdBdejfdidjZ/e0dkdlZ1e0dmdnZ2e0dodpZ3e0dqdrZ4e0dsdtZ5e0dudvZ6e0dwdxZ7e8e9e:ddddSdTdUdydzddd{dgddddddddddgdd|d#ddd.gdfdVe*ee*BdBdeeeBd$eeeBdBd4ed5ed3ed}ed~edeedBdeedBdeded@edBdedBdAej#eej#BdBd.ejdBdejdBd&ejdBd(ejdBd'ejdBd)ejdBdeeeBdeeeBdBdedede;eeZ?S)LTX2ConditionPipelinez Pipeline for video generation which allows image conditions to be inserted at arbitary parts of the video. Reference: https://github.com/Lightricks/LTX-Video TODO z>text_encoder->connectors->transformer->vae->audio_vae->vocoder)r7 prompt_embedsnegative_prompt_embedsrWvae audio_vae text_encoder tokenizer connectors transformervocoderc sRt|j||||||||dt|dddur|jjnd|_t|dddur-|jjnd|_t|dddur<|j j nd|_ t|dddurK|j jnd|_ t|dddur[|j jjnd|_t|ddurj|j jjnd|_t|dddurz|j jjnd |_t|dddur|j jjnd |_t|jd d |_t|d ddur|jj|_dSd|_dS)N)rmrnrorprqrrrsrWrm rnrrri>bilinear)vae_scale_factorresamplerp)super__init__register_modulesgetattrrmspatial_compression_ratiovae_spatial_compression_ratiotemporal_compression_ratiovae_temporal_compression_ratiornmel_compression_ratioaudio_vae_mel_compression_ratio$audio_vae_temporal_compression_ratiorrconfig patch_sizetransformer_spatial_patch_size patch_size_ttransformer_temporal_patch_size sample_rateaudio_sampling_ratemel_hop_lengthaudio_hop_lengthrvideo_processorrpmodel_max_lengthtokenizer_max_length) selfrWrmrnrorprqrrrsrUr/r0r}s@  zLTX2ConditionPipeline.__init__leftruư>text_hidden_statessequence_lengthsrK padding_side scale_factorepsreturncCs^|j\}}}} |j} tj||dd} |dkr#| |dddfk} n|dkr6||dddf} | | k} ntd|| ddddddf} || d}|||ddd}|jd d d ||}|| t d j d d d }|| t d j d d d }|||||}||}| d}| ddd|| }||d}|j| d}|S)a& Packs and normalizes text encoder hidden states, respecting padding. Normalization is performed per-batch and per-layer in a masked fashion (only over non-padded positions). Args: text_hidden_states (`torch.Tensor` of shape `(batch_size, seq_len, hidden_dim, num_layers)`): Per-layer hidden_states from a text encoder (e.g. `Gemma3ForConditionalGeneration`). sequence_lengths (`torch.Tensor of shape `(batch_size,)`): The number of valid (non-padded) tokens for each batch instance. device: (`str` or `torch.device`, *optional*): torch device to place the resulting embeddings on padding_side: (`str`, *optional*, defaults to `"left"`): Whether the text tokenizer performs padding on the `"left"` or `"right"`. scale_factor (`int`, *optional*, defaults to `8`): Scaling factor to multiply the normalized hidden states by. eps (`float`, *optional*, defaults to `1e-6`): A small positive value for numerical stability when performing normalization. Returns: `torch.Tensor` of shape `(batch_size, seq_len, hidden_dim * num_layers)`: Normed and flattened text encoder hidden states. rKrrightNrz,padding_side must be 'left' or 'right', got r\r)rrTr]infz-infrdtype)shaperr*arange unsqueezerN masked_fillviewsumr.aminamaxflattensqueezeexpandto)rrrKrrr batch_sizeseq_len hidden_dim num_layersoriginal_dtype token_indicesmask start_indicesmasked_text_hidden_statesnum_valid_positions masked_meanx_minx_maxnormalized_hidden_states mask_flatr/r/r0_pack_text_embeds/s,    z'LTX2ConditionPipeline._pack_text_embedsrr{Npromptnum_videos_per_promptmax_sequence_lengthrcCsF|p|j}|p |jj}t|tr|gn|}t|}t|dddur1d|j_|jj dur1|jj |j_ dd|D}|j|d|dddd }|j } |j } | |} | |} |j| | dd } | j} tj| d d } | jd d } |j| | ||jj|d }|j |d}|j\}}}|d|d}||||d }| |d } | |d} || fS)a Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded device: (`str` or `torch.device`): torch device to place the resulting embeddings on dtype: (`torch.dtype`): torch dtype to cast the prompt embeds to max_sequence_length (`int`, defaults to 1024): Maximum sequence length to use for the prompt. rpNrcSsg|]}|qSr/)strip).0pr/r/r0 szBLTX2ConditionPipeline._get_gemma_prompt_embeds.. max_lengthTpt)paddingr truncationadd_special_tokensreturn_tensors) input_idsattention_maskoutput_hidden_statesrr^)rKrrrr)_execution_deviceror isinstancestrrVrrpr pad_token eos_tokenrrr hidden_statesr*stackrrrrepeatr)rrrrrrKrr text_inputstext_input_idsprompt_attention_masktext_encoder_outputstext_encoder_hidden_statesrrk_rr/r/r0_get_gemma_prompt_embedstsR           z.LTX2ConditionPipeline._get_gemma_prompt_embedsTnegative_promptdo_classifier_free_guidancerkrlrnegative_prompt_attention_maskc  Cs| p|j} t|tr |gn|}|durt|} n|jd} |dur.|j||| | | | d\}}|r|dur|p7d}t|trB| |gn|}|dur_t|t|ur_tdt|dt|d| t|krxtd|d t|d |d | d |j||| | | | d\}}||||fS) a" Encodes the prompt into text encoder hidden states. Args: prompt (`str` or `list[str]`, *optional*): prompt to be encoded negative_prompt (`str` or `list[str]`, *optional*): The prompt or prompts not to guide the image generation. If not defined, one has to pass `negative_prompt_embeds` instead. Ignored when not using guidance (i.e., ignored if `guidance_scale` is less than `1`). do_classifier_free_guidance (`bool`, *optional*, defaults to `True`): Whether to use classifier free guidance or not. num_videos_per_prompt (`int`, *optional*, defaults to 1): Number of videos that should be generated per prompt. torch device to place the resulting embeddings on prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. negative_prompt_embeds (`torch.Tensor`, *optional*): Pre-generated negative text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. device: (`torch.device`, *optional*): torch device dtype: (`torch.dtype`, *optional*): torch dtype Nr)rrrrrKrz?`negative_prompt` should be the same type to `prompt`, but got z != .z`negative_prompt`: z has batch size z, but `prompt`: zT. Please make sure that passed `negative_prompt` matches the batch size of `prompt`.) rrrrVrrtype TypeErrorrN)rrrrrrkrlrrrrrKrrr/r/r0 encode_promptsP )      z#LTX2ConditionPipeline.encode_promptc s|ddks |ddkrtd|d|d|dur8tfdd|Ds8tdjd fd d |D|durK|durKtd |d |d|durW|durWtd|durnt|tsnt|tsntdt||durz|durztd|dur|durtd|dur|dur|j|jkrtd|jd|jd|j|jkrtd|jd|jd| dur| jdkrtd| jd| dur| jdkrtd| jddSdS)Nrtrz8`height` and `width` have to be divisible by 32 but are z and rc3s|]}|jvVqdSrD_callback_tensor_inputsrkrr/r0 $s  z5LTX2ConditionPipeline.check_inputs..z2`callback_on_step_end_tensor_inputs` has to be in z , but found csg|] }|jvr|qSr/rrrr/r0r(sz6LTX2ConditionPipeline.check_inputs..zCannot forward both `prompt`: z and `prompt_embeds`: z2. Please make sure to only forward one of the two.zeProvide either `prompt` or `prompt_embeds`. Cannot leave both `prompt` and `prompt_embeds` undefined.z2`prompt` has to be of type `str` or `list` but is zEMust provide `prompt_attention_mask` when specifying `prompt_embeds`.zWMust provide `negative_prompt_attention_mask` when specifying `negative_prompt_embeds`.zu`prompt_embeds` and `negative_prompt_embeds` must have the same shape when passed directly, but got: `prompt_embeds` z != `negative_prompt_embeds` z`prompt_attention_mask` and `negative_prompt_attention_mask` must have the same shape when passed directly, but got: `prompt_attention_mask` z% != `negative_prompt_attention_mask` zOnly unpacked (5D) video latents of shape `[batch_size, latent_channels, latent_frames, latent_height, latent_width] are supported, but got ze dims. If you have packed (3D) latents, please unpack them (e.g. using the `_unpack_latents` method).rvzuOnly unpacked (4D) audio latents of shape `[batch_size, num_channels, audio_length, mel_bins] are supported, but got zk dims. If you have packed (3D) latents, please unpack them (e.g. using the `_unpack_audio_latents` method).) rNallrrrr'rrrb) rrheightwidth"callback_on_step_end_tensor_inputsrkrlrrr7 audio_latentsr/rr0 check_inputssh   z"LTX2ConditionPipeline.check_inputsr7rrc Csl|j\}}}}}||}||} ||} ||d||| || |}|dddddddd dd dd}|S) Nrrrrvrrr)rreshapepermuter) r7rrr num_channels num_framesrrpost_patch_num_framespost_patch_heightpost_patch_widthr/r/r0 _pack_latentsXs ( z#LTX2ConditionPipeline._pack_latentsrrrc CsV|d}|||||d|||}|dddddddd dd dddd}|S) Nrrrvrrrrrr)sizerrr)r7rrrrrrr/r/r0_unpack_latentsps 0z%LTX2ConditionPipeline._unpack_latentsr latents_mean latents_stdscaling_factorcCsP|ddddd|j|j}|ddddd|j|j}||||}|SNrrrrrKrr7rrrr/r/r0_normalize_latents}z(LTX2ConditionPipeline._normalize_latentscCsP|ddddd|j|j}|ddddd|j|j}||||}|Srrrr/r/r0_denormalize_latentsrz*LTX2ConditionPipeline._denormalize_latents noise_scaler3cCs.t|j||j|jd}||d||}|S)Nr3rKrr)rrrKr)r7r r3noisenoised_latentsr/r/r0_create_noised_statesz*LTX2ConditionPipeline._create_noised_statec Cs|dur5|dur5|j\}}}}||}||}||d||||}|dddddddddd}|S|dddd}|S)Nrrrrvrrr)rrrr transpose) r7rrrr latent_lengthlatent_mel_binspost_patch_latent_lengthpost_patch_mel_binsr/r/r0_pack_audio_latentss $z)LTX2ConditionPipeline._pack_audio_latentsr num_mel_binscCsr|dur+|dur+|d}||||d||}|dddddddddd}|S|dd|fdd}|S)Nrrrrrvrr)rrrr unflattenr)r7rrrrrr/r/r0_unpack_audio_latentss $z+LTX2ConditionPipeline._unpack_audio_latentscCs,||j|j}||j|j}|||SrDrrKrr7rrr/r/r0_normalize_audio_latents z.LTX2ConditionPipeline._normalize_audio_latentscCs,||j|j}||j|j}|||SrDrrr/r/r0_denormalize_audio_latentsrz0LTX2ConditionPipeline._denormalize_audio_latents start_framesequence_num_framestarget_num_framescCs,|j}t|||}|d||d}|S)a Trim a conditioning sequence to the allowed number of frames. Args: start_frame (int): The target frame number of the first frame in the sequence. sequence_num_frames (int): The number of frames in the sequence. target_num_frames (int): The target number of frames in the generated video. Returns: int: updated sequence length r)rmin)rrrrrrr/r/r0trim_conditioning_sequences z0LTX2ConditionPipeline.trim_conditioning_sequencey conditionsc Csggg}}}|durg}t|tr|g}|j} |d| d} t|D]\} } t| jtjjr6| jg} n-t| jtjrL| jj dkrLtj | jdd} nt| jt j r`| jj dkr`| j d} n| j} |jj| ||dd}| j}|dkrx|| }|| krtd|d | d | d q%|d }t|d| dd}||||}|ddddd|f}||j|jj|d || j||q%|||fS)a Preprocesses the condition images/videos to torch tensors. Args: conditions (`LTX2VideoCondition` or `List[LTX2VideoCondition]`, *optional*, defaults to `None`): A list of image/video condition instances. height (`int`, *optional*, defaults to `512`): The desired height in pixels. width (`int`, *optional*, defaults to `768`): The desired width in pixels. num_frames (`int`, *optional*, defaults to `121`): The desired number of frames in the generated video. device (`torch.device`, *optional*, defaults to `None`): The device on which to put the preprocessed image/video tensors. Returns: `Tuple[List[torch.Tensor], List[float], List[int]]`: Returns a 3-tuple of lists of length `len(conditions)` as follows: 1. The first list is a list of preprocessed video tensors of shape [batch_size=1, num_channels, num_frames, height, width]. 2. The second list is a list of conditioning strengths. 3. The third list is a list of indices in latent space to insert the corresponding condition. Nrrr)axiscrop) resize_modezThe starting latent index z of condition z6 is too big for the specified number of latent frames z!. This condition will be skipped.r)rrK)rrr enumeraterr%r&r(r)rb expand_dimsr*r+rrpreprocess_videorloggerwarningrmaxr appendrrmrr )rr$rrrrKconditioning_framesconditioning_strengthsconditioning_indicesframe_scale_factorlatent_num_framesi conditionvideo_like_condcondition_pixelslatent_start_idxcond_num_frames start_idxtruncated_cond_framesr/r/r0preprocess_conditionssH      z+LTX2ConditionPipeline.preprocess_conditionsconditioning_maskcondition_latentscondition_strengthscondition_indices latent_height latent_widthcCst|}t|||D]2\} } } | d} | ||} | | }| |dd| |f<| |dd| |f<| |dd| |f<q |||fS)az Applies visual conditioning frames to an initial latent. Args: latents (`torch.Tensor`): Initial packed (patchified) latents of shape [batch_size, patch_seq_len, hidden_dim]. conditioning_mask (`torch.Tensor`, *optional*): Initial packed (patchified) conditioning mask of shape [batch_size, patch_seq_len, 1] with values in [0, 1] where 0 means that the denoising model output will be fully used and 1 means that the condition will be fully used (with intermediate values specifying a blend of the denoised and latent values). Returns: `Tuple[torch.Tensor, torch.Tensor, torch.Tensor]`: Returns a 3-tuple of tensors where: 1. The first element is the packed video latents (with unchanged shape [batch_size, patch_seq_len, hidden_dim]) with the conditions applied 2. The second element is the packed conditioning mask with conditioning strengths applied 3. The third element holds the clean conditioning latents. rN)r* zeros_likezipr)rr7r=r>r?r@rArB clean_latentscondr latent_idxnum_cond_tokensstart_token_idx end_token_idxr/r/r0apply_visual_conditioning8s    z/LTX2ConditionPipeline.apply_visual_conditioningrnum_channels_latentsc  Cs||j} ||j} |d|jd}|||| | f}|d|| | f}| dur5|| |jj|jj|jjj} ntj || |d} | |}| | |j |j } | ||j |j }| jdksg| jdd|jddkr{td| jd|jdd|fdt| trt|jjd| d } |j||||| d \}}}g}|D]+}t|j|| d d }|||jj|jjj| |d}| ||j |j }||q|j| ||||| | d \} }}t| j| | j| jd}d||}||| d|} | ||fS)NrrKrrrz$Provided `latents` tensor has shape z, but the expected shape is rz~ does not support using a list of generators. The first generator in the list will be used for all (pseudo-)random operations.rrr6)r3r4)rArBr r) rrrrmrrrrr*zeros new_zerosrrrrbrrNrr'r+r,rUr!r<r;encoderr.rKrrKr)rr$rrMrrrr rrKr3r7rArBr3r mask_shaper=condition_framesr?r@r>condition_tensorcondition_latentrEr  scaled_maskr/r/r0prepare_latentscsp     &"         z%LTX2ConditionPipeline.prepare_latents@r\audio_latent_lengthc Cs| dur"|| } || |jj|jj} || ||} | j||dS||j} |||| f} t|t rEt ||krEt dt |d|dt | |||d} || } | S)NrNz/You have passed a list of generators of length z+, but requested an effective batch size of z@. Make sure the batch size matches the length of the generators.r ) rrrnrrr rrrr'rVrNr) rrrMrYrr rrKr3r7rrr/r/r0prepare_audio_latentss      z+LTX2ConditionPipeline.prepare_audio_latentscC|jSrD_guidance_scalerr/r/r0guidance_scalez$LTX2ConditionPipeline.guidance_scalecCr[rD)_guidance_rescalerr/r/r0rer_z<X2ConditionPipeline.guidance_rescalecCs |jdkS)Nrr\rr/r/r0rs z1LTX2ConditionPipeline.do_classifier_free_guidancecCr[rD)_num_timestepsrr/r/r0 num_timestepsr_z#LTX2ConditionPipeline.num_timestepscCr[rD)_current_timesteprr/r/r0current_timestepr_z<X2ConditionPipeline.current_timestepcCr[rD)_attention_kwargsrr/r/r0attention_kwargsr_z<X2ConditionPipeline.attention_kwargscCr[rD) _interruptrr/r/r0 interruptr_zLTX2ConditionPipeline.interruptg8@(g@pil frame_raterJrMrLr^rerdecode_timestepdecode_noise_scale output_type return_dictrfcallback_on_step_endrcQ Cst|ttfr |j}|j||||||||||d | |_| |_||_d|_d|_ |dur4t|t r4d}n|durBt|t rBt |}n|j d}|durSt|t sS|g}| dura| dur_| dnd} |j}|j|||j|||||||d \}}}}|jrtj||gdd}tj||gdd}d||jd } |j|| d d \}!}"}#|d|jd}$||j}%||j}&|durtd |j \}'}'}$}%}&|$|%|&}(|jjj})|||||)|||| tj||| \}}*}+|jrt|*|*g}*||},|j|j t!|j"}-t#|,|-}.|durtd |j \}'}'}.}'t$|dddur$|j%jj&nd}/|/|j'}0t$|dddur9|j%jj(nd}1|j)|||1|.|/| tj|||d }| durZt*+dd||n| } t,|(|j-j.dd|j-j.dd|j-j.dd|j-j.dd}2t/0|j-}3t1|3||| | |2d\}'}'t1|j-||| | |2d\} }t2t | ||j-j3d}4t | |_4|j||j|jf}5|jj5j6|j d|$|%|&|j7|d}6|jj89|j d|.|j7}7|jr|6:dd|6j;d}6|7:dd|7j;d}7|j<|d}8t=| D]u\}9}:|j>rq|:|_ |jrt|gdn|};|;|j};|jr+t|gdn|}<|<|j}<|:?|;j d}=|=@d d|*Ad }>|jBd!C|jd=id"|;d#|<d$|!d%|"d&|>d'|=d(|#d)|#d*|$d+|%d,|&d-|d.|.d/|6d0|7d1|d2d\}?}@Wdn 1swY|?!}?|@!}@|jr|?Cd\}A}B|A|jD|B|A}?|@Cd\}C}D|C|jD|D|C}@|jEdkrtF|?|B|jEd3}?tF|@|D|jEd3}@|?Gd}E|j-jH|9}F||?|F}G|Gd|*d|E|+!|*d|E|?j}H||H|F|?j}I|j-jI|I|:|dd4d}|3jI|@|:|dd4d}|durLi}J|D] }KtJ|K|J|K<q.|||9|:|J}L|LKd5|}|LKd6|}|9t | dksg|9d|4krk|9d|j-j3dkrk|8LtMrrtNOqWdn 1swY|P||$|%|&|jQ|jR}|S||jTjU|jTjV|jTjjW}|X||j%jU|j%jV}|jY||.|0d7}|d8kr|}M|}Nn||j}|jTjjZsd}=nMt[|j |||jd9}Ot|t s|g|}|dur|}n t|t s|g|}tj\|||jd:}=tj\|||jd:ddddddf}d||||O}||jTj}|jTj]||=dd4d}M|j^j_|M|d;}M||j%j}|j%j]|dd4d}P|`|P}N|a|sS|M|NfStb|M|Nd<S)>u Function invoked when calling the pipeline for generation. Args: conditions (`List[LTXVideoCondition], *optional*`): The list of frame-conditioning items for the video generation. prompt (`str` or `List[str]`, *optional*): The prompt or prompts to guide the image generation. If not defined, one has to pass `prompt_embeds`. instead. height (`int`, *optional*, defaults to `512`): The height in pixels of the generated image. This is set to 480 by default for the best results. width (`int`, *optional*, defaults to `768`): The width in pixels of the generated image. This is set to 848 by default for the best results. num_frames (`int`, *optional*, defaults to `121`): The number of video frames to generate frame_rate (`float`, *optional*, defaults to `24.0`): The frames per second (FPS) of the generated video. num_inference_steps (`int`, *optional*, defaults to 40): The number of denoising steps. More denoising steps usually lead to a higher quality image at the expense of slower inference. sigmas (`List[float]`, *optional*): Custom sigmas to use for the denoising process with schedulers which support a `sigmas` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. timesteps (`List[int]`, *optional*): Custom timesteps to use for the denoising process with schedulers which support a `timesteps` argument in their `set_timesteps` method. If not defined, the default behavior when `num_inference_steps` is passed will be used. Must be in descending order. guidance_scale (`float`, *optional*, defaults to `4.0`): Guidance scale as defined in [Classifier-Free Diffusion Guidance](https://huggingface.co/papers/2207.12598). `guidance_scale` is defined as `w` of equation 2. of [Imagen Paper](https://huggingface.co/papers/2205.11487). Guidance scale is enabled by setting `guidance_scale > 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, usually at the expense of lower image quality. guidance_rescale (`float`, *optional*, defaults to 0.0): Guidance rescale factor proposed by [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) `guidance_scale` is defined as `φ` in equation 16. of [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891). Guidance rescale factor should fix overexposure when using zero terminal SNR. noise_scale (`float`, *optional*, defaults to `None`): The interpolation factor between random noise and denoised latents at each timestep. Applying noise to the `latents` and `audio_latents` before continue denoising. If not set, will be inferred from the sigma schedule. num_videos_per_prompt (`int`, *optional*, defaults to 1): The number of videos to generate per prompt. generator (`torch.Generator` or `List[torch.Generator]`, *optional*): One or a list of [torch generator(s)](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for video generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. audio_latents (`torch.Tensor`, *optional*): Pre-generated noisy latents, sampled from a Gaussian distribution, to be used as inputs for audio generation. Can be used to tweak the same generation with different prompts. If not provided, a latents tensor will be generated by sampling using the supplied random `generator`. prompt_embeds (`torch.Tensor`, *optional*): Pre-generated text embeddings. Can be used to easily tweak text inputs, *e.g.* prompt weighting. If not provided, text embeddings will be generated from `prompt` input argument. prompt_attention_mask (`torch.Tensor`, *optional*): Pre-generated attention mask for text embeddings. negative_prompt_embeds (`torch.FloatTensor`, *optional*): Pre-generated negative text embeddings. For PixArt-Sigma this negative prompt should be "". If not provided, negative_prompt_embeds will be generated from `negative_prompt` input argument. negative_prompt_attention_mask (`torch.FloatTensor`, *optional*): Pre-generated attention mask for negative text embeddings. decode_timestep (`float`, defaults to `0.0`): The timestep at which generated video is decoded. decode_noise_scale (`float`, defaults to `None`): The interpolation factor between random noise and denoised latents at the decode timestep. output_type (`str`, *optional*, defaults to `"pil"`): The output format of the generate image. Choose between [PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`. return_dict (`bool`, *optional*, defaults to `True`): Whether or not to return a [`~pipelines.ltx.LTX2PipelineOutput`] instead of a plain tuple. attention_kwargs (`dict`, *optional*): A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under `self.processor` in [diffusers.models.attention_processor](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py). callback_on_step_end (`Callable`, *optional*): A function that calls at the end of each denoising steps during the inference. The function is called with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int, callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by `callback_on_step_end_tensor_inputs`. callback_on_step_end_tensor_inputs (`List`, *optional*): The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the `._callback_tensor_inputs` attribute of your pipeline class. max_sequence_length (`int`, *optional*, defaults to `1024`): Maximum sequence length to use with the `prompt`. Examples: Returns: [`~pipelines.ltx.LTX2PipelineOutput`] or `tuple`: If `return_dict` is `True`, [`~pipelines.ltx.LTX2PipelineOutput`] is returned, otherwise a `tuple` is returned where the first element is a list with the generated images. ) rrrrrkrlrrr7rFNrrr) rrrrrkrlrrrrKrg.T) additive_maskzGot latents of shape [batch_size, latent_dim, latent_frames, latent_height, latent_width], `latent_num_frames`, `latent_height`, `latent_width` will be inferred.zwGot audio_latents of shape [batch_size, num_channels, audio_num_frames, mel_bins], `audio_num_frames` will be inferred.rnrXru)rMrYrr rrKr3r7base_image_seq_lenr{max_image_seq_lenr=rBgffffff?rCgffffff@)rMrH)fps)r)r)totalrr cond_uncondraudio_hidden_statesencoder_hidden_statesaudio_encoder_hidden_statestimestepaudio_timestepencoder_attention_maskaudio_encoder_attention_maskrrrrtaudio_num_frames video_coords audio_coordsrfro)re)ror7rk)rlatentr rN)rn)raudior/)crr r tensor_inputsrr]r`rergrcrr'rVrrrrr*catrrrqrrr+inforrr in_channelsrWfloat32rrr.rroundrrnmel_binsrlatent_channelsrZr(linspacerIrWgetcopydeepcopyr[r-orderraropeprepare_video_coordsrK audio_ropeprepare_audio_coordsrrb progress_barr(rhrrr cache_contextchunkr^rerirrMsteplocalspopupdate XLA_AVAILABLExm mark_steprrrrrmrrrrrtimestep_conditioningrtensordecoderpostprocess_videorsmaybe_free_model_hooksr)Qrr$rrrrrrkrJrMrLr^rer rr3r7rrkrrlrrlrmrnrorfrprrrrKadditive_attention_maskconnector_prompt_embedsconnector_audio_prompt_embedsconnector_attention_maskr3rArBrvideo_sequence_lengthrMr=rE duration_saudio_latents_per_secondr~rrnum_channels_latents_audiorHaudio_schedulernum_warmup_stepsrope_interpolation_scalerrrr4tlatent_model_inputaudio_latent_model_inputrzvideo_timestepnoise_pred_videonoise_pred_audionoise_pred_video_uncondnoise_pred_video_textnoise_pred_audio_uncondnoise_pred_audio_textbszsigmadenoised_sampledenoised_sample_conddenoised_latents_condcallback_kwargsrcallback_outputsvideorr generated_mel_spectrogramsr/r/r0__call__s(                                 &   6b          zLTX2ConditionPipeline.__call__)rrur)rr{ruNN) NTrNNNNr{ruNN)NNNNNNN)rr)rrD)NN)Nr!r"r#N) NrrLr!r"r#rNNNN) rrurrXr\NNNN)@r!r"r#r$model_cpu_offload_seq_optional_componentsrrrr rrrrrrr} staticmethodr*r+rrKr-r.rr'rrboolrrrrrr Generatorr rrrrr rtupler<rKrWrZpropertyr^rerrbrdrfrhno_gradrEXAMPLE_DOC_STRINGdictrrr __classcell__r/r/rr0rjs 7 F  M      [ D"          R -     Y   !              rj)Nr1)r<r=r>r?)NNNN)r\)CrrP dataclassesrtypingrrnumpyr( PIL.Imager%r* transformersrrr callbacksr r loadersr r models.autoencodersr rmodels.transformersr schedulersrutilsrrrutils.torch_utilsrrrpipeline_utilsrrqrpipeline_outputrrsrtorch_xla.core.xla_modelcore xla_modelrr get_loggerr!r+rrr+rrr;r-r.rIrKr'r[rirjr/r/r/r0s          4      <