o iL"@s~ddlZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZmZdd lmZmZdd lmZdd lmZmZeeZd edBdejdBde dejdededBf ddZdejdedBdedBde dedede edBedBffddZ! d9dejdedBdedBde dedede"de edBedBffddZ#dejdejfd d!Z$dejd"ej%dejfd#d$Z& % &d:d ed'ejd(ejd)ej%d*eejej%gejfde dejded+e'd,e"dejfd-d.Z(d/dd&d0d1e&ej)d0fdejdedBdedBde deded2ed3edBd,e"d4e'd5ed*eejej%gejfd6ej*d+e'de edBedBffd7d8Z+dS);N)replace)partial)Callable)tqdm)Res2sDiffusionStep)DiffusionStepProtocol)X0Model) to_denoised to_velocity)post_process_latenttimesteps_from_mask)get_res2s_coefficients)Denoiser LatentStatestatedenoisedsteppersigmasstep_idxreturncCs>|dus|dur |St||j|j}t|||j|||dS)zOAdvance one diffusion step for a single modality, or return ``None`` if absent.Nlatent)r denoise_mask clean_latentrstepr)rrrrrrM/home/ubuntu/LTX-2/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py _step_statesr video_state audio_state transformerdenoiserc CsZtt|ddD]\}}||||||\}} t|||||}t|| |||}q ||fS)a Perform the joint audio-video denoising loop over a diffusion schedule. Either ``video_state`` or ``audio_state`` may be ``None`` for absent modalities; the absent modality is passed through unchanged. ### Parameters sigmas: A 1D tensor of noise levels (diffusion sigmas) defining the sampling schedule. All steps except the last element are iterated over. video_state: The current video :class:`LatentState`, or ``None`` if video is absent. audio_state: The current audio :class:`LatentState`, or ``None`` if audio is absent. stepper: An implementation of :class:`DiffusionStepProtocol` that updates a latent given the current latent, its denoised estimate, the full ``sigmas`` schedule, and the current step index. transformer: The diffusion model passed to the denoiser at each step. denoiser: A callable implementing :class:`Denoiser`. It is invoked as ``denoiser(transformer, video_state, audio_state, sigmas, step_index)`` and must return ``(denoised_video, denoised_audio)``. ### Returns tuple[LatentState | None, LatentState | None] Final ``(video_state, audio_state)`` after the denoising loop. N) enumeraterr) rrrrr r!r_denoised_videodenoised_audiorrreuler_denoising_loop"s "r'@ge_gammac sd}d}dtjdtjdtdtjdBdttjtjff fdd } tt|dd D]\} } |||||| \} } |durH| durHt| |j|j} |durX| durXt| |j|j} || d d kr|durn| durnt || d }|dur|| dur|t || d }||fS|dur| dur| |j | || |\}} t || |j | || d }|dur| dur| |j | || |\}} t || |j | || d }q*||fS) a Perform the joint audio-video denoising loop using gradient-estimation sampling. Same interface as :func:`euler_denoising_loop` with an additional ``ge_gamma`` parameter for velocity correction. ### Parameters ge_gamma: Gradient estimation coefficient controlling the velocity correction term. Default is 2.0. Paper: https://openreview.net/pdf?id=o2ND9v0CeK ### Returns tuple[LatentState | None, LatentState | None] See :func:`euler_denoising_loop` for return value description. N noisy_sampledenoised_samplesigmaprevious_velocityrcs<t|||}|dur||}||}t|||}||fS)N)r r )r*r+r,r-current_velocitydelta_vtotal_velocityr)rrupdate_velocity_and_samplefs   zLgradient_estimating_euler_denoising_loop..update_velocity_and_sampler"rr) torchTensorfloattupler#rr rrrrr)rrrrr r!r)previous_audio_velocityprevious_video_velocityr2rr$r%r&rr1r(gradient_estimating_euler_denoising_loopMsR    r:xcCs$||jddd|jdddS)N)r"T)dimkeepdim)sub_meandiv_std)r;rrr_channelwise_normalizes$rC generatorcCs4tj|j|tj|jd}|||}t|S)N)rDdtypedevice)r4randnshapefloat64rFr@rBrC)r;rDnoiserrr_get_new_noisesrKF?sampler+step_noise_generator new_noise_fn legacy_modeetac Cs|} ||j|} |s/t|j| |} t|j| |d} t| | g}d}|j||||| | d}|rDt||j|j }|S)Nr3r)rMr+r step_indexrJrQ) clonerr rdoubler4stackrr r)rrMr+rNrOrrrrPrQ sigmas_copy new_noise timestepsnext_timestepsx_nextrrr_inject_sde_noises$  r[r"Td noise_seednoise_seed_substepbongmathbongmath_max_iter model_dtypec2 Cs|p|}|dur td|jj}|dur|d}tj|d|}tj|d|}tt|| | d}t|||d}t||dd}t|t sItdt |d }|d d krjtj |dd tj d d g|jdgd d}t |d d|dd  }i}d}tt|D]}||}||d }|dur|jnd}|dur|jnd}||||||\}}|dur|durt||j|j}|dur|durt||j|j}||} t| ||\}!}"}#t||}$|dur|dur||}%|| |!|%}&nd}%d}&|dur1|dur1||}'|| |!|'}(nd}'d}(|&durM|durM||||&t||$gd d}&|(dure|dure||||(t||$gd d}(| r| dkr|dkrt| D]3})|&dur|%dur|&| |!|%}||}%|(dur|'dur|(| |!|'}||}'qv|dur|&durt||&| dnd}*|dur|(durt||(| dnd}+|||*|+t|$g|jd d\},}-|dur|,durt|,|j|j},|dur |-dur t|-|j|j}-|dur-|%dur-|,dur-|,|}.|| |"|%|#|.}/nd}/|durQ|'durQ|-durQ|-|}0|| |"|'|#|0}1nd}1|/durf|durf||||/||d}/|1dury|dury||||1||d}1|dur|/durt||/| d}|dur|1durt||1| d}q|d d kr||||||\}}|dur|durt||j|j}t||| d}|dur|durt||j|j}t||| d}||fS)a Joint audio-video denoising loop using the res_2s second-order sampler. Iterates over the diffusion schedule with a two-stage Runge-Kutta step: evaluates the denoiser at the current point and at a midpoint (with SDE noise), then combines both with RK coefficients. Supports anchor-point refinement (bong iteration) and optional SDE noise injection. Requires :class:`Res2sDiffusionStep` as ``stepper``. Either modality may be ``None`` (absent). ### Parameters transformer: The diffusion model passed to the denoiser at each step. denoiser: Callable implementing :class:`Denoiser`. noise_seed: Seed for step-level SDE noise; substep seed defaults to ``noise_seed + 10000``. noise_seed_substep: Optional seed for substep SDE noise; if None, derived from ``noise_seed``. eta: Controls stochastic noise injection strength (0=deterministic, 1=maximum). Applies to main diffusion steps; substeps always use 0.5. Default 0.5. bongmath: Whether to run iterative anchor refinement (bong iteration) when step size is small. bongmath_max_iter: Max iterations for bong refinement when enabled. new_noise_fn: Callable ``(latent, generator) -> noise`` for SDE injection. model_dtype: Dtype for latent state updates (e.g. bfloat16). ### Returns tuple[LatentState | None, LatentState | None] Final ``(video_state, audio_state)`` after the denoising loop. Nz;At least one of video_state or audio_state must be providedi')rF)rrOrP)rNrQrLz1stepper must be an instance of Res2sDiffusionStepr3r"rg/nR?g)r=)rrMr+rrgQ?r)rrrrR) ValueErrorrrFr4 Generator manual_seedrr[ isinstancerlencattensorlogrTcpurrangerSr rritemr sqrtrUrto)2rrrrr r!r]r^rQr_r`rOrarP present_state state_devicerNsubstep_noise_generatorsde_noise_injecting_fnstep_noise_injecting_fnsubstep_noise_injecting_fn n_full_stepshs phi_cachec2rr, sigma_nextx_anchor_videox_anchor_audiodenoised_video_1denoised_audio_1ha21b1b2 sub_sigma eps_1_video x_mid_video eps_1_audio x_mid_audior$mid_video_statemid_audio_statedenoised_video_2denoised_audio_2 eps_2_video x_next_video eps_2_audio x_next_audiorrr res2s_audio_video_denoising_loops1   *0             r)r()FrL),logging dataclassesr functoolsrtypingrr4r#ltx_core.components.diffusion_stepsrltx_core.components.protocolsrltx_core.model.transformerrltx_core.utilsr r ltx_pipelines.utils.helpersr r ltx_pipelines.utils.res2sr ltx_pipelines.utils.typesrr getLogger__name__loggerr5intrr7r'r6r:rCrcrKboolr[bfloat16rErrrrrs           2 E   )