o i0@sBddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZmZmZmZdd lmZdd lmZmZmZmZm Z m!Z!m"Z"d ej#fd dZ$dXddZ%dej&de'd ej&fddZ(   dYdede)dedej#dej*de+de+dBdedBd ej&dBfddZ,  dZd ej-j.de)dedej#dej*de+de+dBd ej&dBfd!d"Z/d#e0ed$e'd%e'dedej*dej#d e0efd&d'Z1d#e0ed$e'd%e'dedej*dej#d e0efd(d)Z2d#e0ed$e'd%e'dedej*dej#d e0efd*d+Z3 , d[d-ed.e0ed/edej*dej#d0e+d1ej&dBd efd2d3Z4d4ed5e0ed6ed efd7d8Z5d9ej&d:ej&d;ej&d ej&fdd\d?ed@ej&dAej&dBe7d e f dCdDZ8d:ej&dAe+ej&Bd ej&fdEdFZ9e):dGdHZ;dIe)d e)fdJdKZ<  L Md]dNedOe)dPe)dBdQe'dRe'd e)f dSdTZ=d$e'd%e'dUe7d dfdVdWZ>dS)^N)Noiser)ConditioningItemVideoConditionByKeyframeIndexVideoConditionByLatentIndex) encode_audio)Modality) TilingConfig VideoEncoder)GemmaTextEncoder) LatentTools)AudioLatentShape LatentStateVideoLatentShapeVideoPixelShape)ImageConditioningInput)decode_audio_from_file decode_imagedecode_video_from_fileget_videostream_fpsload_image_and_preprocessresize_aspect_ratio_preservingvideo_preprocessreturncCs&tjrtdtjStdS)Ncudacpu)torchr is_availabledevicecurrent_devicerrL/home/ubuntu/LTX-2/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py get_devices  r!cCs ttjtjdS)N)gccollectrr empty_cache synchronizerrrr cleanup_memory$s r&latentexpected_frames_countcCsx|jd}||kr|ddddd|f}|S||kr:t|j}|||d<tj||j|jd}tj||gdd}|S)N)rdtype)dim)shapelistrzerosrr*cat)r'r( actual_frames shape_as_listpadrrr _conform_latent_length*s    r3 video_encoder file_path output_shaperr* start_time max_duration tiling_configc Cst|}||jkrtd|d|jd|p|j|}t||||d} t| |j|j||} || |p6t } t |j} t | | S)aLoad video from a file, and construct the video latent conforming to video output shape. Args: video_encoder: Model used to encode pixel frames to latent space. file_path: Path to the video file. output_shape: Target pixel shape (height, width, frames, fps) for the conditioning. device: Device to run the encoder and hold tensors on. dtype: Dtype for the output latents. start_time: Start time in seconds to begin reading the video (default 0.0). max_duration: Maximum duration in seconds. If None, uses output_shape.frames at output_shape.fps (default None). tiling_config: Tiling configuration for the encoder. Defaults to TilingConfig.default(). Returns: Encoded video latents of shape (1, C, T, H, W) with T = required_latent_frames, or None (currently this function always returns a tensor). zInput video FPS z does not match output FPS z, not supported)pathrr8r9)rfps ValueErrorframesrrheightwidth tiled_encoderdefaultrfrom_pixel_shaper3) r5r6r7rr*r8r9r:r< frame_genr>latentsrequired_latent_framesrrr video_latent_from_file:s   rG audio_encoderc CsT|p|j|j}t||||}|durdSt||d||}t|j} t|| S)aLoad audio from a file, and construct the audio latent conforming to video output shape. Args: audio_encoder: Model used to encode audio to latent space. file_path: Path to the audio or video file containing an audio stream. output_shape: Target video pixel shape; used to derive required latent frames and, when max_duration is None, the audio duration (output_shape.frames / fps). device: Device to run the encoder and hold tensors on. dtype: Dtype for the output latents. start_time: Start time in seconds to begin reading the audio (default 0.0). max_duration: Maximum duration in seconds. If None, uses the full span implied by output_shape (default None). Returns: Encoded audio latents of shape (1, C, T, ...) with T = required_latent_frames, or None if the file has no audio stream. N)r>r<rrtor from_video_pixel_shaper3) rHr6r7rr*r8r9audio_inrErFrrr audio_latent_from_file^s  rLimagesr?r@c Csjg}|D].}t|j|||||jd}||} |jdkr$t| |jdd} n t| |j|jd} || q|S)zCreate a list of conditionings by replacing the latent at the first frame with the encoded image if present and using other encoded images as the keyframe conditionings. image_pathr?r@r*rcrfrr'strength latent_idx) keyframesrR frame_idx)rr;rPrUrrRrappend) rMr?r@r5r*r conditioningsimgimage encoded_image conditioningrrr combined_image_conditioningss0   r\c CJg}|D]}t|j|||||jd}||} |t| |j|jdq|S)NrNrQ)rr;rPrVrrRrU rMr?r@r5r*rrWrXrYrZrrr 'image_conditionings_by_replacing_latents&r_c Cr])NrN)rTrUrR)rr;rPrVrrUrRr^rrr ,image_conditionings_by_adding_guiding_latentsr`?toolsrWnoiser noise_scaleinitial_latentcCs(||||}t|||}|||}|S)zCreate a noised latent state from empty state, conditionings, and noiser. Creates an empty latent state, applies conditionings, and then adds noise using the provided noiser. Returns the final noised state ready for diffusion. )create_initial_statestate_with_conditionings)rbrWrcr*rrdrestaterrr create_noised_states  ri latent_stateconditioning_items latent_toolscCs|D] }|j||d}q|S)zApply a list of conditionings to a latent state. Iterates through the conditioning items and applies each one to the latent state in sequence. Returns the modified state with all conditionings applied. )rjrl)apply_to)rjrkrlr[rrr rgsrgdenoised denoise_maskcleancCs |||d||jS)z5Blend denoised output with clean state based on mask.)floatrIr*)rnrorprrr post_process_latents rsTrhcontextsigmaenabledc Cs&t||j|t|j||j|d|jdS)zCreate a Modality from a latent state. Constructs a Modality object with the latent state's data, timesteps derived from the denoise mask and sigma, positions, and the provided context. N)rvr'ru timesteps positionsrt context_maskattention_mask)rr'timesteps_from_maskrorxrz)rhrtrurvrrr modality_from_latent_states  r|cCs@t|tjr|dkr|jdgdg|dR}||S)aXCompute timesteps from a denoise mask and sigma value. Multiplies the denoise mask by sigma to produce timesteps for each position in the latent state. Areas where the mask is 0 will have zero timesteps. When sigma is ``(B,)`` it is reshaped to ``(B, 1, ...)`` so the batch dimension aligns correctly with ``denoise_mask``. rq) isinstancerTensorr+view)rorurrr r{s r{u‘’“”—– ′−z ''""-- '-textcCs8|t}t|D]\}}|r||dSq |S)zaClean a response from curly quotes and leading non-letter characters which Gemma tends to insert.N) translate_UNICODE_REPLACEMENTS enumerateisalpha)richarrrr clean_response&s r* text_encoderpromptrOimage_long_sideseedcCsfd}|r t|d}t|}t||tj}|j|||d}n|j||d}t d|t |S)z=Generate an enhanced prompt from a text encoder and a prompt.N)rO)rzEnhanced prompt: ) rrtensorrrIuint8 enhance_i2v enhance_t2vlogginginfor)rrrOrrrYrrr generate_enhanced_prompt1s  r is_two_stagec CsX|rdnd}||dks||dkr*td|d|d|d|r!dnd d |d d S) zAssert that the resolution is divisible by the required divisor. For two-stage pipelines, the resolution must be divisible by 64. For one-stage pipelines, the resolution must be divisible by 32. @ rz Resolution (xz) is not divisible by z. For z two-stagez one-stagez2 pipelines, height and width must be multiples of .N)r=)r?r@rdivisorrrr assert_resolutionEs  r)rN)r4NN)r4N)raN)T)Nrr)?r"rrltx_core.components.noisersrltx_core.conditioningrrrltx_core.model.audio_vaerltx_core.model.transformerrltx_core.model.video_vaerr ltx_core.text_encoders.gemmar ltx_core.toolsr ltx_core.typesr r rrltx_pipelines.utils.argsrltx_pipelines.utils.media_iorrrrrrrrr!r&rintr3strr*rrrGnnModulerLr-r\r_r`rirgrsboolr|r{ maketransrrrrrrrr sF      $  * ! %    "