o iD@s@UddlZddlZddlmZmZddlmZddlmZddl Z ddl Z ddl Z ddl mZddlmZddlmZddlmZdd lmZmZdd lmZeeZd e jd ed e jfddZde jdeded e jfddZde jde j de j!d e jfddZ"efde#dedede j!de j ded e jfddZ$dee jdedede j!de j d e jf dd Z%de#d e j&fd!d"Z'd#e j(j)d$e j*j+d%ed dfd&d'Z,d#e j(j)d(ed e j*j+fd)d*Z-d#e j(j)d$e j*j+d+e j.d dfd,d-Z/d.e jee jBd/ed%edBd0e#d1ed df d2d3Z0d4d4d5d5d6d6d7Z1e2e#e3fe4d8<d9e j.d e j&fd:d;Z5dZ6ddS)WN) GeneratorIterator)Fraction)BytesIO) rearrange)Image)DeviceLikeType)tqdm)AudioVideoPixelShape)DEFAULT_IMAGE_CRFimage long_sidereturnc Cst|jdd\}}t||}|t|}t||}t||}t|||}t|dd} | jddkr8| dS| S)a[ Resize image preserving aspect ratio (filling target long side). Preserves the input dimensions order. Args: image: Input image tensor with shape (F (optional), H, W, C) long_side: Target long side size. Returns: Tensor with shape (F (optional), H, W, C) F = 1 if input is 3D, otherwise input shape[0] zb c f h w -> b f h w cr)shapemaxfloatintresize_and_center_cropr) r rheightwidthmax_sidescale target_height target_widthresizedresultr M/home/ubuntu/LTX-2/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io.pyresize_aspect_ratio_preservings     r"tensorrrc Cs|jdkr t|d}n|jdkrt|d}n td|jd|j\}}}}t||||}t||}t||}tjj j |||fddd }||d } ||d } |d d d d | | || | |f}t|d }|S) ac Resize tensor preserving aspect ratio (filling target), then center crop to exact dimensions. Args: latent: Input tensor with shape (H, W, C) or (F, H, W, C) height: Target height width: Target width Returns: Tensor with shape (1, C, 1, height, width) for 3D input or (1, C, F, height, width) for 4D input zh w c -> 1 c h wzf h w c -> f c h wz1Expected input with 3 or 4 dimensions; got shape .bilinearF)sizemode align_cornersrNzf c h w -> 1 c f h w) ndimr ValueErrorrrmathceiltorchnn functional interpolate) r#rr_src_hsrc_wrnew_hnew_wcrop_top crop_leftr r r!r+s     ( rlatentdevicedtypecCs|ddj||dS)Ng_@?)r;r<)to)r:r;r<r r r!normalize_latentNsr? image_pathcrfcCsDt|d}t||d}tj|tj|d}t|||}t|||}|S)z Loads an image from a path and preprocesses it for conditioning. Note: The image is resized to the nearest multiple of 2 for compatibility with video codecs. )r@)r rAr<r;) decode_image preprocessr/r#float32rr?)r@rrr<r;rAr r r r!load_image_and_preprocessRs   rFframescCsPd}|D]!}t|tj||}t|||}|dur|ntj||gdd}q|S)aPreprocesses a video frame generator for conditioning. Args: frames: Generator of video frames as tensors of shape (1, H, W, C), dtype uint8. height: Target height in pixels. width: Target width in pixels. dtype: Target dtype for the output tensor. device: Target device for the output tensor. Returns: Tensor of shape (1, C, F, height, width) with values in [-1, 1]. Nr)dim)rr>r/rEr?cat)rGrrr<r;rfframer r r!video_preprocessfs   rLcCs$t|}t|dddf}|S)N.r$)ropennparray)r@r np_arrayr r r!rCs rC container audio_streamaudiocCs|j}|jdkr|dddf}|jddkr!|jddkr!|j}|jddkr1td|jd|jtjkrFt|dd}|d tj}t j j | dd d d d }|j|_t|||dS) Nrrrz,Expected samples with 2 channels; got shape r&gr=g@s16stereo)formatlayout)waveformr+rTr,r<r/int16clipr>av AudioFrame from_ndarray contiguousreshapecpunumpy sampling_rate sample_rate_resample_audio)rQrRrSsamplesframe_inr r r! _write_audios"  riaudio_sample_ratecCs0|jd|d}||j_d|j_td||j_|S)z/ Prepare the audio stream for writing. aacraterVr) add_stream codec_contextrerXr time_base)rQrjrRr r r!_prepare_audio_streams rqrhc Cs|j}|jpd}|jp d}|jp|j}tjjj|||d}d}||D]} | j dur.|| _ || j 7}|j| _| | | q$| D]} | | qDdS)NfltprV)rWrXrmr) rorWrXrer]rS resamplerAudioResamplerresampleptsrgmuxencode) rQrRrhcc target_format target_layout target_rateaudio_resampleraudio_next_ptsrframepacketr r r!rfs&       rfvideofps output_pathvideo_chunks_numberc CsTt|tjr t|g}t|}|j\}}}}tj|dd} | jdt |d} || _ || _ d| _ |dur9t | |j} dtjdtttjt fddfdtttjt fddffd d } t| |||d D]#} | d }|D]}tjj|d d}| |D]}| |q|qmqb| D]}| |q|durt| | || td|dS)Nw)r)libx264rlyuv420p first_chunktiles_generatorrcss|V|EdHdS)Nr )rrr r r! all_tilesszencode_video..all_tiles)totalrbrgb24rWzVideo saved to ) isinstancer/Tensoriternextrr]rMrnrrrpix_fmtrqrdrtupler r>rc VideoFramer_rxrwricloseloggerinfo)rrrSrrrr3rrrQstreamrRr video_chunkvideo_chunk_cpu frame_arrayrKrr r r! encode_videos@        rg`@g@gA)u8u8prUs16ps32s32p_INT_FORMAT_MAXrKcCsR|jj}|tj}|tvr|t|}|jjs't|j j }| d|j }|S)zaConvert an audio frame to a float32 ndarray with values in [-1, 1] and shape (channels, samples).rT) rWname to_ndarrayastyperNrEr is_planarlenrXchannelsrarZ)rKfmtarrrr r r!_audio_frame_to_floats  rpathcCs>t|}ztdd|jD}t|jW|S|w)zRead video stream FPS.cs|] }|jdkr|VqdSrNtype.0sr r r! z&get_videostream_fps..)r]rMrstreamsr average_rater)rrQ video_streamr r r!get_videostream_fps s  rcCst|}z:tdd|jD}t|j}|jpd}|dkr*tdd||D}|j j }|j j }t d||||dW| S| w)a3Read video stream metadata as a VideoPixelShape with batch=1. If frame count is missing in the container, decodes the stream to count frames. Args: path: Path to the video file. Returns: VideoPixelShape with batch=1, frames, height, width, and fps populated from the stream. csrrrrr r r!r!rz+get_videostream_metadata..rcss|]}dVqdS)rNr )rr3r r r!r%sr)batchrGrrr)r]rMrrrrrGsumdecoderorrr r)rrQrr num_framesrrr r r!get_videostream_metadatas   r start_time max_durationcCs|t|}z tdd|jD}Wnty|YdSw|j}t||j}|r0||n|j |j}|j ||dg} d} |j ddD]0} | j durOqGt | j |j} | | j| j} | |krdqG| |krjn| durp| } | t| qG|| sdStj| dd}t|| |}|dkr|d |df}|durt||}|d d|f}t||d}t||d S) aDecodes audio from a file, optionally seeking to a start time and limiting duration. Args: path: Path to the audio/video file containing an audio stream. device: Device to place the resulting tensor on. start_time: Start time in seconds to begin reading audio from. max_duration: Maximum audio duration in seconds. If None, reads to end of stream. Returns: An Audio object with waveform of shape (1, channels, samples), or None if no audio stream. csr)rSNrrr r r!r;rz)decode_audio_from_file..Nrr)rSrT)axis.)rYrd)r]rMrr StopIterationrrmrrpdurationseekrrvrrgreappendrrN concatenateroundr/ from_numpyr> unsqueezer )rr;rrrQrRre start_ptsend_timergfirst_frame_timerK frame_time frame_endrS skip_samples max_samplesrYr r r!decode_audio_from_file-sJ     rstarting_frame frame_capc cst|}zJtdd|jD}t||D]*\}}||kr!qtj| tj |d d}|V|durB|d8}|dkrBnqW| dSW| dS| w)aDecodes video from a file by sequential frame index, without relying on pts. Args: path: Path to the video file. device: Device to place the resulting tensors on. starting_frame: Number of leading frames to skip (default 0). frame_cap: Maximum number of frames to yield. If None, no frame limit (default None). Yields: Frames as tensors of shape (1, H, W, C), dtype uint8. csrrrrr r r!r}rz(decode_video_by_frame..rBrNr) r]rMrr enumeraterr/r#to_rgbruint8rr) rr;rrrQrindexrKr#r r r!decode_video_by_framels$   rc cs<t|}ztdd|jD}t|j}|dkr%|jt|||d|dur-||nd}||D]V}|j duret|j } t || } |durPt || nd} t ||| | dEdHW| dS|j |} | |kroq4|dury| |kryntj|tj|ddVq4W| dSW| dS| w)a7Decodes video from a file using presentation timestamps for time-based trimming. If a frame with no pts is encountered, falls back to :func:`decode_video_by_frame` using FPS-derived frame indices. Args: path: Path to the video file. device: Device to place the resulting tensors on. start_time: Start time in seconds (default 0.0). max_duration: Maximum duration in seconds to decode. If None, reads to end of stream (default None). Yields: Frames as tensors of shape (1, H, W, C), dtype uint8. csrrrrr r r!rrz)decode_video_from_file..rrN)rr;rrrB)r]rMrrrrprrrrvrrrrr/r#rrrr) rr;rrrQrrprrKrrrrr r r!decode_video_from_files8        $ r output_file image_arraycCstj|ddd}zQ|jddt|ddd}|jd d d }|jdd d }|d|d|f}||_||_tjj|d dj d d}| | || | W| dS| w) Nrmp4rrrveryfast)rApreset)rmoptionsrrrr) r]rMrnstrrrrrr_reformatrwrxr)rrrArQrrrav_framer r r!encode_single_framesr video_filecCsPt|}ztdd|jD}t||}W|n|w|jddS)Ncsrrrrr r r!rrz&decode_single_frame..rr)r]rMrrrrr)rrQrrKr r r!decode_single_frames  rcCs||dkr|St}t||||}Wdn1swYt| }t|}Wd|S1s7wY|S)Nr)rrgetvaluer)r rAr video_bytesrrr r r!rDs     rD)rN)rN)?loggingr-collections.abcrr fractionsriorr]rcrNr/einopsrPILrtorch._prims_commonrr ltx_core.typesr r ltx_pipelines.utils.constantsr getLogger__name__rrrr"rr;r<r?rrFrLndarrayrCrQ ContainerrS AudioStreamrirqr^rfrrdictr__annotations__rrrrrrrrOrrDr r r r!s         "#   "   0    B " 4"