Yeah, we need to test LTX2 fast video. I want to check what is the concurrency and everything. Check this repo also for reference https://github.com/Lightricks/LTX-2


ToolSearch

Question: select:WebFetch


Question: select:WebFetch


WebFetch

Prompt: Extract all details about LTX-2 video model: what it is, installation instructions, inference/usage code, supported features (text-to-video, image-to-video, etc.), model variants, hardware requirements, concurrency details, API usage, and any performance benchmarks or speed information. Include all code examples.


Bash

List home directory contents


=1.14
=6.31.1
844424930361864-72-m_telugu.wav
844424933537177-514-m_telugu.wav
ab_test.txt
Attarintiki_Daredi_Telugu_Full_Movie_Pawan_Kalyan_Samantha_Pranitha_DSP_Trivikram_Part_4_256KBPS.webm
audio_jepa_paper.txt
bare_train_log.txt
bench-codecs
bench_firered2.py
bench_firered.py
benchmark_luxtts.py
benchmark_luxtts_v2.py
benchmark_max.py
benchmark_vllm_high.py
benchmark_vllm.py
bob5_BOB5HRg8tSE.wav
bob5_clean
bob5_clean_concat.py
bob5_clean_extract.py
bob5_diarized
bob5_extract_exclusive.py
bob5_extract_timeline.py
bob5_final
bob5_final_sp03_rediarize
bob5_lalal_16k.wav
bob5_lalal_diarized
bob5_lalal_pyannote.py
bob5_local
bob5_nemo_diarize.py


# LTX-2 Video Generation Model: Comprehensive Overview

## What is LTX-2?

LTX-2 is described as "the first DiT-based audio-video foundation model" that integrates synchronized audio and video generation, high-fidelity output, multiple performance tiers, and production-ready capabilities in a single system.

## Installation Instructions

Clone and set up the environment:
```bash
git clone https://github.com/Lightricks/LTX-2.git
cd LTX-2
uv sync --frozen
source .venv/bin/activate
```

## Required Models & Checkpoints

**Main Model Checkpoint** (choose one):
- `ltx-2.3-22b-dev.safetensors`
- `ltx-2.3-22b-distilled.safetensors`

**Upscalers**:
- Spatial upscalers (x2 or x1.5)
- Temporal upscaler x2

**Supporting Models**:
- Distilled LoRA (`ltx-2.3-22b-distilled-lora-384.safetensors`)
- Gemma 3 text encoder

**LoRA Modules** (optional):
- IC-LoRA variants (Union Control, Motion Track, Detailer, Pose)
- Camera control LoRAs (Dolly, Jib, Static movements)

## Available Pipeline Implementations

| Pipeline | Purpose |
|----------|---------|
| TI2VidTwoStagesPipeline | Production text/image-to-video with 2x upsampling (recommended) |
| TI2VidTwoStagesHQPipeline | Two-stage with res_2s sampler for efficiency |
| TI2VidOneStagePipeline | Single-stage for quick prototyping |
| DistilledPipeline | Fastest inference (8 predefined sigmas) |
| ICLoraPipeline | Video-to-video and image-to-video transformations |
| KeyframeInterpolationPipeline | Keyframe image interpolation |
| A2VidPipelineTwoStage | Audio-conditioned video generation |
| RetakePipeline | Regional video regeneration |

## Supported Features

**Generation Modes**:
- Text-to-video
- Image-to-video
- Audio-to-video
- Video-to-video
- Keyframe interpolation
- Region retaking

**Control Mechanisms**:
- Motion tracking
- Pose control
- Camera control (multiple directions)
- Union control
- Detail enhancement

## Performance Optimization Strategies

1. **Use DistilledPipeline**: "Fastest inference with only 8 predefined sigmas" (8 steps stage 1, 4 steps stage 2)

2. **FP8 Quantization**: `--quantization fp8-cast` flag or `quantization=QuantizationPolicy.fp8_cast()` in Python code. For Hopper GPUs, use `fp8-scaled-mm` variant.

3. **Attention Optimizations**: Install xFormers via `uv sync --extra xformers` or Flash Attention 3 for Hopper GPUs

4. **Gradient Estimation**: Reduce inference steps from 40 to 20-30 while maintaining quality

5. **Memory Management**: Disable automatic cleanup between stages if VRAM permits

6. **Architecture Choice**: Single-stage pipeline for non-high-resolution requirements

## Prompting Guidelines

Effective prompts should:
- Use detailed, chronological action descriptions
- Include specific movements, appearances, camera angles
- Stay within 200 words
- Start directly with main action
- Maintain literal, precise language
- Follow cinematographer-like shot descriptions

The system supports automatic prompt enhancement via an `enhance_prompt` parameter.

## Integration & Tools

**ComfyUI Support**: Available via separate repository at github.com/Lightricks/ComfyUI-LTXVideo/

**Training Capabilities**: The ltx-trainer package enables LoRA fine-tuning, full model fine-tuning, and IC-LoRA training

## Package Architecture

The monorepo contains three main components:
- **ltx-core**: Model implementation and inference stack
- **ltx-pipelines**: High-level generation pipelines
- **ltx-trainer**: Training and fine-tuning tools

## Hardware & Performance Notes

No specific VRAM requirements or speed benchmarks were provided in the documentation excerpt. The documentation emphasizes that quantization and attention optimizations significantly impact memory and performance characteristics.


ls /home/ubuntu/ | grep -i ltx; nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader 2>/dev/null || echo "No GPU info"

Check for existing LTX setup and GPU info


NVIDIA A100-SXM4-80GB, 81920 MiB, 81156 MiB


Good — A100 80GB with nearly all VRAM free. No existing LTX-2 setup. Here's what I found:

## LTX-2 Overview

- **First DiT-based audio-video foundation model** from Lightricks
- 22B parameter model with multiple pipeline options
- Supports: text-to-video, image-to-video, audio-to-video, video-to-video, keyframe interpolation

## Key Pipelines (by speed)

| Pipeline | Use Case | Steps |
|---|---|---|
| **DistilledPipeline** | Fastest — 8 sigmas | 8 stage1, 4 stage2 |
| TI2VidTwoStagesHQPipeline | Efficient with res_2s sampler | ~20-30 |
| TI2VidTwoStagesPipeline | Production quality, 2x upsampling | ~40 |
| TI2VidOneStagePipeline | Quick prototyping | ~40 |

## Performance Options

- **FP8 quantization** (`--quantization fp8-cast`) — significant VRAM savings
- **xFormers / Flash Attention** for attention optimization
- Gradient estimation can cut steps from 40 → 20-30

## Your Setup

- **GPU**: A100 80GB — should handle the 22B model comfortably, especially with FP8

Want me to:
1. **Clone the repo and set it up** so we can benchmark inference speed and test concurrency (running multiple generations in parallel)?
2. **Just clone and do a single test run first** to see baseline performance?

What kind of concurrency are you thinking — multiple simultaneous generations on the same GPU, or multi-GPU / API-level concurrency?


multiple simulataneous on same gpu


Let's get it set up and then benchmark concurrent generations.


Clone LTX-2 repo


Cloning into 'LTX-2'...


cd /home/ubuntu/LTX-2 && cat pyproject.toml

Check project setup requirements


[tool.uv.sources]
ltx-core = { workspace = true }
ltx-pipelines = { workspace = true }

[tool.uv.workspace]
members = ["packages/*"]

[[tool.uv.index]]
name = "pypi"
url = "https://pypi.org/simple"

[dependency-groups]
dev = [
    "google-cloud-storage>=2.0",
    "matplotlib>=3.7",
    "pre-commit>=4.3.0",
    "ruff>=0.14.3",
    "pytest~=9.0",
]

[tool.ruff]
target-version = "py311"
line-length = 120
exclude = []

[tool.ruff.lint]
select = [
    "E", # pycodestyle
    "F", # pyflakes
    "W", # pycodestyle (warnings)
    "I", # isort
    "N", # pep8-naming
    "ANN", # flake8-annotations
    "B", # flake8-bugbear
    "A", # flake8-builtins
    "COM", # flake8-commas
    "C4", # flake8-comprehensions
    "DTZ", # flake8-datetimez
    "EXE", # flake8-executable
    "PIE", # flake8-pie
    "T20", # flake8-print
    "PT", # flake8-pytest
    "SIM", # flake8-simplify
    "ARG", # flake8-unused-arguments
    "PTH", # flake8--use-pathlib
    "ERA", # flake8-eradicate
    "RUF", # ruff specific rules
    "PL", # pylint
]
ignore = [
    "ANN002", # Missing type annotation for *args
    "ANN003", # Missing type annotation for **kwargs
    "ANN204", # Missing type annotation for special method
    "COM812", # Missing trailing comma
    "PTH123", # `open()` should be replaced by `Path.open()`
    "PLR2004", # Magic value used in comparison, consider replacing with a constant variable
    "N812", # Lowercase  imported as non-lowercase
]

[tool.ruff.lint.pylint]
max-args = 10

[tool.ruff.lint.isort]
known-first-party = ["ltx_core", "ltx_pipelines", "ltx_trainer"]


which uv || pip install uv 2>/dev/null; uv --version

Check uv availability


/home/ubuntu/.local/bin/uv
uv 0.10.12 (x86_64-unknown-linux-gnu)


Check workspace packages


ltx-core
ltx-pipelines
ltx-trainer


Install dependencies with uv


 + six==1.17.0
 + smmap==5.0.2
 + sympy==1.14.0
 + tifffile==2025.5.10
 + tokenizers==0.22.2
 + tomli==2.4.0
 + torch==2.9.1
 + torchaudio==2.9.1
 + torchcodec==0.9.1
 + torchvision==0.24.1
 + tqdm==4.67.1
 + transformers==4.57.6
 + triton==3.5.1
 + typer==0.21.1
 + typing-extensions==4.15.0
 + typing-inspection==0.4.2
 + tzdata==2025.3
 + urllib3==2.6.3
 + virtualenv==20.36.1
 + wandb==0.24.0


Good, dependencies installed. Now let me check the pipeline code and figure out how to run inference and test concurrency.


Check pipeline source files


a2vid_two_stage.py
distilled.py
ic_lora.py
__init__.py
keyframe_interpolation.py
retake.py
ti2vid_one_stage.py
ti2vid_two_stages_hq.py
ti2vid_two_stages.py
utils


.venv/lib/python3.10/site-packages/tqdm/completion.sh
.venv/lib/python3.10/site-packages/virtualenv/activation/bash/activate.sh
.venv/lib/python3.10/site-packages/wandb/docker/wandb-entrypoint.sh


No files found


Check pipelines package structure


pyproject.toml
README.md
src


1	# LTX-2 Pipelines
2
3	High-level pipeline implementations for generating audio-video content with Lightricks' **LTX-2** model. This package provides ready-to-use pipelines for text-to-video, image-to-video, video-to-video, and keyframe interpolation tasks.
4
5	Pipelines are built using building blocks from [`ltx-core`](../ltx-core/) (schedulers, guiders, noisers, patchifiers) and handle the complete inference flow including model loading, encoding, decoding, and file I/O.
6
7	---
8
9	## 📋 Overview
10
11	LTX-2 Pipelines provides production-ready implementations that abstract away the complexity of the diffusion process, model loading, and memory management. Each pipeline is optimized for specific use cases and offers different trade-offs between speed, quality, and memory usage.
12
13	**Key Features:**
14
15	- 🎬 **Multiple Pipeline Types**: Text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake
16	- ⚡ **Optimized Performance**: Support for FP8 transformers, gradient estimation, and memory optimization
17	- 🎯 **Production Ready**: Two-stage pipelines for best quality output
18	- 🔧 **LoRA Support**: Easy integration with trained LoRA adapters
19	- 📦 **Self-Contained**: Handles model loading, encoding, decoding, and file I/O
20	- 🚀 **CLI Support**: All pipelines can be run as command-line scripts
21
22	---
23
24	## 🚀 Quick Start
25
26	`ltx-pipelines` provides ready-made inference pipelines for text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake. Built using building blocks from [`ltx-core`](../ltx-core/), these pipelines handle the complete inference flow including model loading, encoding, decoding, and file I/O.
27
28	## 🔧 Installation
29
30	```bash
31	# From the repository root
32	uv sync --frozen
33
34	# Or install as a package
35	pip install -e packages/ltx-pipelines
36	```
37
38	### Running Pipelines
39
40	All pipelines can be run directly from the command line. Each pipeline module is executable:
41
42	```bash
43	# Run a pipeline (example: two-stage text-to-video)
44	python -m ltx_pipelines.ti2vid_two_stages \
45	    --checkpoint-path path/to/checkpoint.safetensors \
46	    --distilled-lora path/to/distilled_lora.safetensors 0.8 \
47	    --spatial-upsampler-path path/to/upsampler.safetensors \
48	    --gemma-root path/to/gemma \
49	    --prompt "A beautiful sunset over the ocean" \
50	    --output-path output.mp4
51
52	# View all available options for any pipeline
53	python -m ltx_pipelines.ti2vid_two_stages --help
54	```
55
56	Available pipeline modules:
57
58	- `ltx_pipelines.ti2vid_two_stages` - Two-stage text/image-to-video (recommended).
59	- `ltx_pipelines.ti2vid_two_stages_hq` - Two-stage text/image-to-video (different sampler, better quality).
60	- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video.
61	- `ltx_pipelines.distilled` - Fast text/image-to-video pipeline using only the distilled model.
62	- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA.
63	- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation.
64	- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio.
65	- `ltx_pipelines.retake` - Regenerate a time region of an existing video.
66
67	Use `--help` with any pipeline module to see all available options and parameters.
68
69	---
70
71	## 🎯 Pipeline Selection Guide
72
73	### Quick Decision Tree
74
75	```text
76	Do you have an existing video to modify?
77	├─ YES → Use RetakePipeline (regenerate a specific time region)
78	│
79	Do you have an audio file to drive generation?
80	├─ YES → Use A2VidPipelineTwoStage (audio-to-video)
81	│
82	Do you need to condition on existing images/videos?
83	├─ YES → Do you have reference videos for video-to-video?
84	│  ├─ YES → Use ICLoraPipeline
85	│  └─ NO → Do you have multiple keyframe images to interpolate?
86	│     ├─ YES → Use KeyframeInterpolationPipeline
87	│     └─ NO → Use TI2VidTwoStagesPipeline (image conditioning only)
88	│
89	└─ NO → Text-to-video only
90	   ├─ Do you need best quality?
91	   │  └─ YES → Use TI2VidTwoStagesPipeline (recommended for production)
92	   │
93	   └─ Do you need fastest inference?
94	      └─ YES → Use DistilledPipeline (with 8 predefined sigmas)
95	```
96
97	> **Note:** [`TI2VidOneStagePipeline`](src/ltx_pipelines/ti2vid_one_stage.py) is primarily for educational purposes. For best quality, use two-stage pipelines ([`TI2VidTwoStagesPipeline`](src/ltx_pipelines/ti2vid_two_stages.py), [`TI2VidTwoStagesHQPipeline`](src/ltx_pipelines/ti2vid_two_stages_hq.py), [`ICLoraPipeline`](src/ltx_pipelines/ic_lora.py), [`KeyframeInterpolationPipeline`](src/ltx_pipelines/keyframe_interpolation.py), [`A2VidPipelineTwoStage`](src/ltx_pipelines/a2vid_two_stage.py), or [`DistilledPipeline`](src/ltx_pipelines/distilled.py)). For editing existing videos, use [`RetakePipeline`](src/ltx_pipelines/retake.py).
98
99	### Features Comparison
100
101	| Pipeline | Stages | [Multimodal Guidance](#%EF%B8%8F-multimodal-guidance) | Upsampling | Conditioning | Best For |
102	| -------- | ------ | --- | ---------- | ------------- | -------- |
103	| **TI2VidTwoStagesPipeline** | 2 | ✅ | ✅ | Image | **Production quality** (recommended) |
104	| **TI2VidTwoStagesHQPipeline** | 2 | ✅ | ✅ | Image | Same as above, res_2s sampler (higher quality) |
105	| **TI2VidOneStagePipeline** | 1 | ✅ | ❌ | Image | Educational, prototyping |
106	| **DistilledPipeline** | 2 | ❌ | ✅ | Image | Fastest inference (8 sigmas) |
107	| **ICLoraPipeline** | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |
108	| **KeyframeInterpolationPipeline** | 2 | ✅ | ✅ | Keyframes | Animation, interpolation |
109	| **A2VidPipelineTwoStage** | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
110	| **RetakePipeline** | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |
111
112	---
113
114	## 📦 Available Pipelines
115
116	### 1. TI2VidTwoStagesPipeline
117
118	**Best for:** High-quality text/image-to-video generation with upsampling. **Recommended for production use.**
119
120	**Source**: [`src/ltx_pipelines/ti2vid_two_stages.py`](src/ltx_pipelines/ti2vid_two_stages.py)
121
122	Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](#%EF%B8%8F-multimodal-guidance), Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning. Highest quality output, slower than one-stage but significantly better quality.
123
124	**Use when:** Production-quality video generation, higher resolution needed, quality over speed, text-to-video with image conditioning.
125
126	---
127
128	### 2. TI2VidTwoStagesHQPipeline
129
130	**Best for:** Same two-stage text/image-to-video as TI2VidTwoStagesPipeline but with a different sampler and step count.
131
132	**Source**: [`src/ltx_pipelines/ti2vid_two_stages_hq.py`](src/ltx_pipelines/ti2vid_two_stages_hq.py)
133
134	Uses the **res_2s** second-order sampler instead of Euler. Same stage structure (stage 1 at target resolution with CFG, stage 2 upsampling with distilled LoRA) and image conditioning support. Typically allows fewer steps for comparable quality; trade-offs differ from the default Euler-based pipeline.
135
136	**Use when:** You want the same two-stage workflow with fewer steps or prefer the res_2s sampling behavior.
137
138	---
139
140	### 3. TI2VidOneStagePipeline
141
142	**Best for:** Educational purposes and quick prototyping.
143
144	**Source**: [`src/ltx_pipelines/ti2vid_one_stage.py`](src/ltx_pipelines/ti2vid_one_stage.py)
145
146	> **⚠ Important:** This pipeline is primarily for educational purposes. For production-quality results, use `TI2VidTwoStagesPipeline` or other two-stage pipelines.
147
148	Single-stage generation (no upsampling) with [multimodal guidance](#%EF%B8%8F-multimodal-guidance) and image conditioning support. Faster inference but lower resolution output (typically 512x768).
149
150	**Use when:** Learning how the pipeline works, quick prototyping, testing, or when high resolution is not needed.
151
152	---
153
154	### 4. DistilledPipeline
155
156	**Best for:** Fastest inference with good quality using a distilled model with predefined sigma schedule.
157
158	**Source**: [`src/ltx_pipelines/distilled.py`](src/ltx_pipelines/distilled.py)
159
160	Two-stage generation with 8 predefined sigmas (8 steps in stage 1, 4 steps in stage 2). No guidance required. Fastest inference among all pipelines. Supports image conditioning. Requires spatial upsampler.
161
162	**Use when:** Fastest inference is critical, batch processing many videos, or when you have a distilled model checkpoint.
163
164	---
165
166	### 5. ICLoraPipeline
167
168	**Best for:** Video-to-video and image-to-video transformations using IC-LoRA.
169
170	**Source**: [`src/ltx_pipelines/ic_lora.py`](src/ltx_pipelines/ic_lora.py)
171
172	Two-stage generation with IC-LoRA support. Can condition on reference videos (video-to-video) or images at specific frames. CFG guidance in stage 1, upsampling in stage 2. Requires IC-LoRA trained model.
173
174	**Note:** ICLoraPipeline can only be used with a distilled model.
175
176	**Use when:** Video-to-video transformations, image-to-video with strong control, or when you have reference videos to guide generation.
177
178	---
179
180	### 6. KeyframeInterpolationPipeline
181
182	**Best for:** Generating videos by interpolating between keyframe images.
183
184	**Source**: [`src/ltx_pipelines/keyframe_interpolation.py`](src/ltx_pipelines/keyframe_interpolation.py)
185
186	Two-stage generation with keyframe interpolation. Uses guiding latents (additive conditioning) instead of replacing latents for smoother transitions. [Multimodal guidance](#%EF%B8%8F-multimodal-guidance) in stage 1, upsampling in stage 2.
187
188	**Use when:** You have keyframe images and want to interpolate between them, creating smooth transitions, or animation/motion interpolation tasks.
189
190	---
191
192	### 7. A2VidPipelineTwoStage
193
194	**Best for:** Generating video driven by an input audio.
195
196	**Source**: [`src/ltx_pipelines/a2vid_two_stage.py`](src/ltx_pipelines/a2vid_two_stage.py)
197
198	Two-stage audio-to-video generation. Stage 1 generates video at half resolution with audio conditioning (video-only denoising with the audio frozen), then Stage 2 upsamples by 2x and refines the video while keeping the audio fixed, using a distilled LoRA. The input audio is encoded via the audio VAE and used as the initial audio latent, but the original audio waveform is passed through and returned in the output to preserve fidelity. Supports image conditioning and prompt enhancement.
199
200	**Extra CLI arguments:** `--audio-path` (required), `--audio-start-time`, `--audio-max-duration`.
201
202	**Use when:** You have an audio clip and want to generate a matching video, audio-reactive video generation, or music visualization.
203
204	---
205
206	### 8. RetakePipeline
207
208	**Best for:** Regenerating a specific time region of an existing video while keeping the rest unchanged.
209
210	**Source**: [`src/ltx_pipelines/retake.py`](src/ltx_pipelines/retake.py)
211
212	Single-stage generation that encodes the source video and audio into latents, applies a temporal region mask to mark `[start_time, end_time]` for regeneration, and denoises only the masked region from a text prompt. Content outside the time window is preserved. Supports independent control over video and audio regeneration (`regenerate_video`, `regenerate_audio` flags), and can use either the full model with CFG guidance or the distilled model with a fixed sigma schedule.
213
214	**Extra CLI arguments:** `--video-path` (required), `--start-time` (required), `--end-time` (required).
215
216	**Constraints:** Source video frame count must satisfy the 8k+1 format (e.g. 97, 193) and resolution must be multiples of 32.
217
218	**Use when:** You want to re-do a specific section of a generated video (e.g. fix a bad segment), selectively regenerate audio or video in a time window, or iterate on part of a result without re-generating the entire clip.
219
220	---
221
222	## 🎨 Conditioning Types
223
224	Pipelines use different conditioning methods from [`ltx-core`](../ltx-core/) for controlling generation. See the [ltx-core conditioning documentation](../ltx-core/README.md#conditioning--control) for details.
225
226	### Image Conditioning
227
228	All pipelines support image conditioning, but with different methods:
229
230	- **Replacing Latents** ([`image_conditionings_by_replacing_latent`](src/ltx_pipelines/utils/helpers.py)):
231	  - Used by: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`, `DistilledPipeline`, `ICLoraPipeline`
232	  - Replaces the latent at a specific frame with the encoded image
233	  - Strong control over specific frames
234
235	- **Guiding Latents** ([`image_conditionings_by_adding_guiding_latent`](src/ltx_pipelines/utils/helpers.py)):
236	  - Used by: `KeyframeInterpolationPipeline`
237	  - Adds the image as a guiding signal rather than replacing
238	  - Better for smooth interpolation between keyframes
239
240	### Video Conditioning
241
242	- **Video Conditioning** (ICLoraPipeline only):
243	  - Conditions on entire reference videos
244	  - Useful for video-to-video transformations
245	  - Uses `VideoConditionByKeyframeIndex` from [`ltx-core`](../ltx-core/)
246
247	---
248
249	## 🎛 Multimodal Guidance
250
251	LTX-2 pipelines use **multimodal guidance** to steer the diffusion process for both video and audio modalities. Each modality (video, audio) has its own guider with independent parameters, allowing fine-grained control over generation quality and adherence to prompts.
252
253	### Guidance Parameters
254
255	The `MultiModalGuiderParams` dataclass controls guidance behavior:
256
257	| Parameter | Description |
258	| --------- | ----------- |
259	| `cfg_scale` | **Classifier-Free Guidance** scale. Higher values make the output adhere more strongly to the text prompt. Typical values: 2.0–5.0. Set to **1.0** to disable. |
260	| `stg_scale` | **Spatio-Temporal Guidance** scale. Controls perturbation-based guidance for improved temporal coherence. Typical values: 0.5–1.5. Set to **0.0** to disable. |
261	| `stg_blocks` | Which transformer blocks to perturb for STG (e.g., `[29]` for the last block). Set to **`[]`** to disable STG. |
262	| `rescale_scale` | Rescales the guided prediction to match the variance of the conditional prediction. Helps prevent over-saturation. Typical values: 0.5–0.7. Set to **0.0** to disable. |
263	| `modality_scale` | **Modality CFG** scale. Steers the model away from unsynced video and audio results, improving audio-visual coherence. Set to **1.0** to disable. |
264	| `skip_step` | Skip guidance every N steps. Can speed up inference with minimal quality loss. Set to **0** to disable (never skip). |
265
266	### How It Works
267
268	The multimodal guider combines three guidance signals during each denoising step:
269
270	1. **CFG (Text Guidance)**: Steers generation toward the text prompt by computing `(cond - uncond_text)`.
271	2. **STG (Perturbation Guidance)**: Improves structural coherence by perturbing specific transformer blocks and steering away from the perturbed prediction.
272	3. **Modality CFG**: For joint audio-video generation, steers the model away from unsynced video and audio results.
273
274	### Example Configuration
275
276	```python
277	from ltx_core.components.guiders import MultiModalGuiderParams
278
279	# Video guider: moderate CFG, STG enabled, modality isolation
280	video_guider_params = MultiModalGuiderParams(
281	    cfg_scale=3.0,
282	    stg_scale=1.0,
283	    rescale_scale=0.7,
284	    modality_scale=3.0,
285	    stg_blocks=[29],
286	)
287
288	# Audio guider: higher CFG for stronger prompt adherence
289	audio_guider_params = MultiModalGuiderParams(
290	    cfg_scale=7.0,
291	    stg_scale=1.0,
292	    rescale_scale=0.7,
293	    modality_scale=3.0,
294	    stg_blocks=[29],
295	)
296	```
297
298	> **Tip:** Start with the default values from [`constants.py`](src/ltx_pipelines/utils/constants.py) and adjust based on your use case. Higher `cfg_scale` = stronger prompt adherence but potentially less natural motion; higher `stg_scale` = better temporal coherence but slower inference (requires extra forward passes).
299	>
300	> **Tip:** When generating video with audio, set `modality_scale` > 1.0 (e.g., 3.0) to improve audio-visual sync. If generating video-only, set it to 1.0 to disable.
301
302	---
303
304	## ⚡ Optimization Tips
305
306
307	### Memory Optimization
308
309	**FP8 Quantization (Lower Memory Footprint):**
310
311	For smaller GPU memory footprint, use the `--quantization` flag and set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.
312
313	Two quantization policies are available:
314
315	| Policy | CLI Flag | Description |
316	| ------ | -------- | ----------- |
317	| **FP8 Cast** | `--quantization fp8-cast` | Downcasts transformer linear weights to FP8 during loading; upcasts on the fly during inference. No extra dependencies. |
318	| **FP8 Scaled MM** | `--quantization fp8-scaled-mm` | Uses FP8 scaled matrix multiplication via TensorRT-LLM (`tensorrt_llm` must be installed). Best performance on Hopper GPUs. |
319
320	**CLI:**
321
322	```bash
323	# FP8 Cast (works on any GPU with FP8 support)
324	PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
325	    --quantization fp8-cast --checkpoint-path=...
326
327	# FP8 Scaled MM (requires tensorrt_llm, best on Hopper GPUs)
328	PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
329	    --quantization fp8-scaled-mm --checkpoint-path=...
330	```
331
332	**Programmatically:**
333
334	When authoring custom scripts, pass a `QuantizationPolicy` to pipeline classes:
335
336	```python
337	from ltx_core.quantization import QuantizationPolicy
338
339	pipeline = TI2VidTwoStagesPipeline(
340	    checkpoint_path=ltx_model_path,
341	    distilled_lora=distilled_lora,
342	    spatial_upsampler_path=upsampler_path,
343	    gemma_root=gemma_root_path,
344	    loras=[],
345	    quantization=QuantizationPolicy.fp8_cast(),  # or QuantizationPolicy.fp8_scaled_mm()
346	)
347	pipeline(...)
348	```
349
350	You still need to use `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` when launching:
351
352	```bash
353	PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python my_denoising_pipeline.py
354	```
355
356	**Memory Cleanup Between Stages:**
357
358	By default, pipelines clean GPU memory (especially transformer weights) between stages. If you have enough memory, you can skip this cleanup to reduce running time:
359
360	```python
361	# In pipeline implementations, memory cleanup happens automatically
362	# between stages. For custom pipelines, you can skip:
363	# utils.cleanup_memory()  # Comment out if you have enough VRAM
364	```
365
366	### Denoising Loop Optimization
367
368	**Gradient Estimation Denoising Loop:**
369
370	Instead of the standard Euler denoising loop, you can use gradient estimation for fewer steps (~20-30 instead of 40):
371
372	```python
373	from ltx_pipelines.utils import gradient_estimating_euler_denoising_loop
374
375	# Use gradient estimation denoising loop
376	def denoising_loop(sigmas, video_state, audio_state, stepper):
377	    return gradient_estimating_euler_denoising_loop(
378	        sigmas=sigmas,
379	        video_state=video_state,
380	        audio_state=audio_state,
381	        stepper=stepper,
382	        denoise_fn=your_denoise_function,
383	        ge_gamma=2.0,  # Gradient estimation coefficient
384	    )
385	```
386
387	This allows you to use **20-30 steps instead of 40** while maintaining quality. The gradient estimation function is available in [`pipeline_utils.py`](src/ltx_pipelines/utils/helpers.py).
388
389	---
390
391	## 🔧 Requirements
392
393	- **LTX-2 Model Checkpoint** - Local `.safetensors` file
394	- **Gemma Text Encoder** - Local Gemma model directory
395	- **Spatial Upscaler** - Required for two-stage pipelines (except one-stage)
396	- **Distilled LoRA** - Required for two-stage pipelines (except one-stage and distilled)
397
398	---
399
400	## 📖 Example: Image-to-Video
401
402	```python
403	from ltx_core.loader import LTXV_LORA_COMFY_RENAMING_MAP, LoraPathStrengthAndSDOps
404	from ltx_pipelines.ti2vid_two_stages import TI2VidTwoStagesPipeline
405	from ltx_core.components.guiders import MultiModalGuiderParams
406
407	distilled_lora = [
408	    LoraPathStrengthAndSDOps(
409	        "/path/to/distilled_lora.safetensors",
410	        0.6,
411	        LTXV_LORA_COMFY_RENAMING_MAP
412	    ),
413	]
414
415	pipeline = TI2VidTwoStagesPipeline(
416	    checkpoint_path="/path/to/checkpoint.safetensors",
417	    distilled_lora=distilled_lora,
418	    spatial_upsampler_path="/path/to/upsampler.safetensors",
419	    gemma_root="/path/to/gemma",
420	    loras=[],
421	)
422
423	video_guider_params = MultiModalGuiderParams(
424	    cfg_scale=3.0,
425	    stg_scale=1.0,
426	    rescale_scale=0.7,
427	    modality_scale=3.0,
428	    skip_step=0,
429	    stg_blocks=[29],
430	)
431
432	audio_guider_params = MultiModalGuiderParams(
433	    cfg_scale=7.0,
434	    stg_scale=1.0,
435	    rescale_scale=0.7,
436	    modality_scale=3.0,
437	    skip_step=0,
438	    stg_blocks=[29],
439	)
440
441	# Generate video from image
442	pipeline(
443	    prompt="A serene landscape with mountains in the background",
444	    output_path="output.mp4",
445	    seed=42,
446	    height=512,
447	    width=768,
448	    num_frames=121,
449	    frame_rate=25.0,
450	    num_inference_steps=40,
451	    video_guider_params=video_guider_params,
452	    audio_guider_params=audio_guider_params,
453	    images=[ImageConditioningInput("input_image.jpg", 0, 1.0, 33)],  # Image at frame 0, strength 1.0, CRF 33
454	)
455	```
456
457	---
458
459	## 🔗 Related Projects
460
461	- **[LTX-Core](../ltx-core/)** - Core model implementation and inference components (schedulers, guiders, noisers, patchifiers)
462	- **[LTX-Trainer](../ltx-trainer/)** - Training and fine-tuning tools
463

# LTX-2 Pipelines

High-level pipeline implementations for generating audio-video content with Lightricks' **LTX-2** model. This package provides ready-to-use pipelines for text-to-video, image-to-video, video-to-video, and keyframe interpolation tasks.

Pipelines are built using building blocks from [`ltx-core`](../ltx-core/) (schedulers, guiders, noisers, patchifiers) and handle the complete inference flow including model loading, encoding, decoding, and file I/O.

---

## 📋 Overview

LTX-2 Pipelines provides production-ready implementations that abstract away the complexity of the diffusion process, model loading, and memory management. Each pipeline is optimized for specific use cases and offers different trade-offs between speed, quality, and memory usage.

**Key Features:**

- 🎬 **Multiple Pipeline Types**: Text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake
- ⚡ **Optimized Performance**: Support for FP8 transformers, gradient estimation, and memory optimization
- 🎯 **Production Ready**: Two-stage pipelines for best quality output
- 🔧 **LoRA Support**: Easy integration with trained LoRA adapters
- 📦 **Self-Contained**: Handles model loading, encoding, decoding, and file I/O
- 🚀 **CLI Support**: All pipelines can be run as command-line scripts

---

## 🚀 Quick Start

`ltx-pipelines` provides ready-made inference pipelines for text-to-video, image-to-video, video-to-video, audio-to-video, keyframe interpolation, and retake. Built using building blocks from [`ltx-core`](../ltx-core/), these pipelines handle the complete inference flow including model loading, encoding, decoding, and file I/O.

## 🔧 Installation

```bash
# From the repository root
uv sync --frozen

# Or install as a package
pip install -e packages/ltx-pipelines
```

### Running Pipelines

All pipelines can be run directly from the command line. Each pipeline module is executable:

```bash
# Run a pipeline (example: two-stage text-to-video)
python -m ltx_pipelines.ti2vid_two_stages \
    --checkpoint-path path/to/checkpoint.safetensors \
    --distilled-lora path/to/distilled_lora.safetensors 0.8 \
    --spatial-upsampler-path path/to/upsampler.safetensors \
    --gemma-root path/to/gemma \
    --prompt "A beautiful sunset over the ocean" \
    --output-path output.mp4

# View all available options for any pipeline
python -m ltx_pipelines.ti2vid_two_stages --help
```

Available pipeline modules:

- `ltx_pipelines.ti2vid_two_stages` - Two-stage text/image-to-video (recommended).
- `ltx_pipelines.ti2vid_two_stages_hq` - Two-stage text/image-to-video (different sampler, better quality).
- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video.
- `ltx_pipelines.distilled` - Fast text/image-to-video pipeline using only the distilled model.
- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA.
- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation.
- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio.
- `ltx_pipelines.retake` - Regenerate a time region of an existing video.

Use `--help` with any pipeline module to see all available options and parameters.

---

## 🎯 Pipeline Selection Guide

### Quick Decision Tree

```text
Do you have an existing video to modify?
├─ YES → Use RetakePipeline (regenerate a specific time region)
│
Do you have an audio file to drive generation?
├─ YES → Use A2VidPipelineTwoStage (audio-to-video)
│
Do you need to condition on existing images/videos?
├─ YES → Do you have reference videos for video-to-video?
│  ├─ YES → Use ICLoraPipeline
│  └─ NO → Do you have multiple keyframe images to interpolate?
│     ├─ YES → Use KeyframeInterpolationPipeline
│     └─ NO → Use TI2VidTwoStagesPipeline (image conditioning only)
│
└─ NO → Text-to-video only
   ├─ Do you need best quality?
   │  └─ YES → Use TI2VidTwoStagesPipeline (recommended for production)
   │
   └─ Do you need fastest inference?
      └─ YES → Use DistilledPipeline (with 8 predefined sigmas)
```

> **Note:** [`TI2VidOneStagePipeline`](src/ltx_pipelines/ti2vid_one_stage.py) is primarily for educational purposes. For best quality, use two-stage pipelines ([`TI2VidTwoStagesPipeline`](src/ltx_pipelines/ti2vid_two_stages.py), [`TI2VidTwoStagesHQPipeline`](src/ltx_pipelines/ti2vid_two_stages_hq.py), [`ICLoraPipeline`](src/ltx_pipelines/ic_lora.py), [`KeyframeInterpolationPipeline`](src/ltx_pipelines/keyframe_interpolation.py), [`A2VidPipelineTwoStage`](src/ltx_pipelines/a2vid_two_stage.py), or [`DistilledPipeline`](src/ltx_pipelines/distilled.py)). For editing existing videos, use [`RetakePipeline`](src/ltx_pipelines/retake.py).

### Features Comparison

| Pipeline | Stages | [Multimodal Guidance](#%EF%B8%8F-multimodal-guidance) | Upsampling | Conditioning | Best For |
| -------- | ------ | --- | ---------- | ------------- | -------- |
| **TI2VidTwoStagesPipeline** | 2 | ✅ | ✅ | Image | **Production quality** (recommended) |
| **TI2VidTwoStagesHQPipeline** | 2 | ✅ | ✅ | Image | Same as above, res_2s sampler (higher quality) |
| **TI2VidOneStagePipeline** | 1 | ✅ | ❌ | Image | Educational, prototyping |
| **DistilledPipeline** | 2 | ❌ | ✅ | Image | Fastest inference (8 sigmas) |
| **ICLoraPipeline** | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |
| **KeyframeInterpolationPipeline** | 2 | ✅ | ✅ | Keyframes | Animation, interpolation |
| **A2VidPipelineTwoStage** | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
| **RetakePipeline** | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |

---

## 📦 Available Pipelines

### 1. TI2VidTwoStagesPipeline

**Best for:** High-quality text/image-to-video generation with upsampling. **Recommended for production use.**

**Source**: [`src/ltx_pipelines/ti2vid_two_stages.py`](src/ltx_pipelines/ti2vid_two_stages.py)

Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](#%EF%B8%8F-multimodal-guidance), Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning. Highest quality output, slower than one-stage but significantly better quality.

**Use when:** Production-quality video generation, higher resolution needed, quality over speed, text-to-video with image conditioning.

---

### 2. TI2VidTwoStagesHQPipeline

**Best for:** Same two-stage text/image-to-video as TI2VidTwoStagesPipeline but with a different sampler and step count.

**Source**: [`src/ltx_pipelines/ti2vid_two_stages_hq.py`](src/ltx_pipelines/ti2vid_two_stages_hq.py)

Uses the **res_2s** second-order sampler instead of Euler. Same stage structure (stage 1 at target resolution with CFG, stage 2 upsampling with distilled LoRA) and image conditioning support. Typically allows fewer steps for comparable quality; trade-offs differ from the default Euler-based pipeline.

**Use when:** You want the same two-stage workflow with fewer steps or prefer the res_2s sampling behavior.

---

### 3. TI2VidOneStagePipeline

**Best for:** Educational purposes and quick prototyping.

**Source**: [`src/ltx_pipelines/ti2vid_one_stage.py`](src/ltx_pipelines/ti2vid_one_stage.py)

> **⚠ Important:** This pipeline is primarily for educational purposes. For production-quality results, use `TI2VidTwoStagesPipeline` or other two-stage pipelines.

Single-stage generation (no upsampling) with [multimodal guidance](#%EF%B8%8F-multimodal-guidance) and image conditioning support. Faster inference but lower resolution output (typically 512x768).

**Use when:** Learning how the pipeline works, quick prototyping, testing, or when high resolution is not needed.

---

### 4. DistilledPipeline

**Best for:** Fastest inference with good quality using a distilled model with predefined sigma schedule.

**Source**: [`src/ltx_pipelines/distilled.py`](src/ltx_pipelines/distilled.py)

Two-stage generation with 8 predefined sigmas (8 steps in stage 1, 4 steps in stage 2). No guidance required. Fastest inference among all pipelines. Supports image conditioning. Requires spatial upsampler.

**Use when:** Fastest inference is critical, batch processing many videos, or when you have a distilled model checkpoint.

---

### 5. ICLoraPipeline

**Best for:** Video-to-video and image-to-video transformations using IC-LoRA.

**Source**: [`src/ltx_pipelines/ic_lora.py`](src/ltx_pipelines/ic_lora.py)

Two-stage generation with IC-LoRA support. Can condition on reference videos (video-to-video) or images at specific frames. CFG guidance in stage 1, upsampling in stage 2. Requires IC-LoRA trained model.

**Note:** ICLoraPipeline can only be used with a distilled model.

**Use when:** Video-to-video transformations, image-to-video with strong control, or when you have reference videos to guide generation.

---

### 6. KeyframeInterpolationPipeline

**Best for:** Generating videos by interpolating between keyframe images.

**Source**: [`src/ltx_pipelines/keyframe_interpolation.py`](src/ltx_pipelines/keyframe_interpolation.py)

Two-stage generation with keyframe interpolation. Uses guiding latents (additive conditioning) instead of replacing latents for smoother transitions. [Multimodal guidance](#%EF%B8%8F-multimodal-guidance) in stage 1, upsampling in stage 2.

**Use when:** You have keyframe images and want to interpolate between them, creating smooth transitions, or animation/motion interpolation tasks.

---

### 7. A2VidPipelineTwoStage

**Best for:** Generating video driven by an input audio.

**Source**: [`src/ltx_pipelines/a2vid_two_stage.py`](src/ltx_pipelines/a2vid_two_stage.py)

Two-stage audio-to-video generation. Stage 1 generates video at half resolution with audio conditioning (video-only denoising with the audio frozen), then Stage 2 upsamples by 2x and refines the video while keeping the audio fixed, using a distilled LoRA. The input audio is encoded via the audio VAE and used as the initial audio latent, but the original audio waveform is passed through and returned in the output to preserve fidelity. Supports image conditioning and prompt enhancement.

**Extra CLI arguments:** `--audio-path` (required), `--audio-start-time`, `--audio-max-duration`.

**Use when:** You have an audio clip and want to generate a matching video, audio-reactive video generation, or music visualization.

---

### 8. RetakePipeline

**Best for:** Regenerating a specific time region of an existing video while keeping the rest unchanged.

**Source**: [`src/ltx_pipelines/retake.py`](src/ltx_pipelines/retake.py)

Single-stage generation that encodes the source video and audio into latents, applies a temporal region mask to mark `[start_time, end_time]` for regeneration, and denoises only the masked region from a text prompt. Content outside the time window is preserved. Supports independent control over video and audio regeneration (`regenerate_video`, `regenerate_audio` flags), and can use either the full model with CFG guidance or the distilled model with a fixed sigma schedule.

**Extra CLI arguments:** `--video-path` (required), `--start-time` (required), `--end-time` (required).

**Constraints:** Source video frame count must satisfy the 8k+1 format (e.g. 97, 193) and resolution must be multiples of 32.

**Use when:** You want to re-do a specific section of a generated video (e.g. fix a bad segment), selectively regenerate audio or video in a time window, or iterate on part of a result without re-generating the entire clip.

---

## 🎨 Conditioning Types

Pipelines use different conditioning methods from [`ltx-core`](../ltx-core/) for controlling generation. See the [ltx-core conditioning documentation](../ltx-core/README.md#conditioning--control) for details.

### Image Conditioning

All pipelines support image conditioning, but with different methods:

- **Replacing Latents** ([`image_conditionings_by_replacing_latent`](src/ltx_pipelines/utils/helpers.py)):
  - Used by: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`, `DistilledPipeline`, `ICLoraPipeline`
  - Replaces the latent at a specific frame with the encoded image
  - Strong control over specific frames

- **Guiding Latents** ([`image_conditionings_by_adding_guiding_latent`](src/ltx_pipelines/utils/helpers.py)):
  - Used by: `KeyframeInterpolationPipeline`
  - Adds the image as a guiding signal rather than replacing
  - Better for smooth interpolation between keyframes

### Video Conditioning

- **Video Conditioning** (ICLoraPipeline only):
  - Conditions on entire reference videos
  - Useful for video-to-video transformations
  - Uses `VideoConditionByKeyframeIndex` from [`ltx-core`](../ltx-core/)

---

## 🎛 Multimodal Guidance

LTX-2 pipelines use **multimodal guidance** to steer the diffusion process for both video and audio modalities. Each modality (video, audio) has its own guider with independent parameters, allowing fine-grained control over generation quality and adherence to prompts.

### Guidance Parameters

The `MultiModalGuiderParams` dataclass controls guidance behavior:

| Parameter | Description |
| --------- | ----------- |
| `cfg_scale` | **Classifier-Free Guidance** scale. Higher values make the output adhere more strongly to the text prompt. Typical values: 2.0–5.0. Set to **1.0** to disable. |
| `stg_scale` | **Spatio-Temporal Guidance** scale. Controls perturbation-based guidance for improved temporal coherence. Typical values: 0.5–1.5. Set to **0.0** to disable. |
| `stg_blocks` | Which transformer blocks to perturb for STG (e.g., `[29]` for the last block). Set to **`[]`** to disable STG. |
| `rescale_scale` | Rescales the guided prediction to match the variance of the conditional prediction. Helps prevent over-saturation. Typical values: 0.5–0.7. Set to **0.0** to disable. |
| `modality_scale` | **Modality CFG** scale. Steers the model away from unsynced video and audio results, improving audio-visual coherence. Set to **1.0** to disable. |
| `skip_step` | Skip guidance every N steps. Can speed up inference with minimal quality loss. Set to **0** to disable (never skip). |

### How It Works

The multimodal guider combines three guidance signals during each denoising step:

1. **CFG (Text Guidance)**: Steers generation toward the text prompt by computing `(cond - uncond_text)`.
2. **STG (Perturbation Guidance)**: Improves structural coherence by perturbing specific transformer blocks and steering away from the perturbed prediction.
3. **Modality CFG**: For joint audio-video generation, steers the model away from unsynced video and audio results.

### Example Configuration

```python
from ltx_core.components.guiders import MultiModalGuiderParams

# Video guider: moderate CFG, STG enabled, modality isolation
video_guider_params = MultiModalGuiderParams(
    cfg_scale=3.0,
    stg_scale=1.0,
    rescale_scale=0.7,
    modality_scale=3.0,
    stg_blocks=[29],
)

# Audio guider: higher CFG for stronger prompt adherence
audio_guider_params = MultiModalGuiderParams(
    cfg_scale=7.0,
    stg_scale=1.0,
    rescale_scale=0.7,
    modality_scale=3.0,
    stg_blocks=[29],
)
```

> **Tip:** Start with the default values from [`constants.py`](src/ltx_pipelines/utils/constants.py) and adjust based on your use case. Higher `cfg_scale` = stronger prompt adherence but potentially less natural motion; higher `stg_scale` = better temporal coherence but slower inference (requires extra forward passes).
>
> **Tip:** When generating video with audio, set `modality_scale` > 1.0 (e.g., 3.0) to improve audio-visual sync. If generating video-only, set it to 1.0 to disable.

---

## ⚡ Optimization Tips

### Memory Optimization

**FP8 Quantization (Lower Memory Footprint):**

For smaller GPU memory footprint, use the `--quantization` flag and set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`.

Two quantization policies are available:

| Policy | CLI Flag | Description |
| ------ | -------- | ----------- |
| **FP8 Cast** | `--quantization fp8-cast` | Downcasts transformer linear weights to FP8 during loading; upcasts on the fly during inference. No extra dependencies. |
| **FP8 Scaled MM** | `--quantization fp8-scaled-mm` | Uses FP8 scaled matrix multiplication via TensorRT-LLM (`tensorrt_llm` must be installed). Best performance on Hopper GPUs. |

**CLI:**

```bash
# FP8 Cast (works on any GPU with FP8 support)
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
    --quantization fp8-cast --checkpoint-path=...

# FP8 Scaled MM (requires tensorrt_llm, best on Hopper GPUs)
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python -m ltx_pipelines.ti2vid_two_stages \
    --quantization fp8-scaled-mm --checkpoint-path=...
```

**Programmatically:**

When authoring custom scripts, pass a `QuantizationPolicy` to pipeline classes:

```python
from ltx_core.quantization import QuantizationPolicy

pipeline = TI2VidTwoStagesPipeline(
    checkpoint_path=ltx_model_path,
    distilled_lora=distilled_lora,
    spatial_upsampler_path=upsampler_path,
    gemma_root=gemma_root_path,
    loras=[],
    quantization=QuantizationPolicy.fp8_cast(),  # or QuantizationPolicy.fp8_scaled_mm()
)
pipeline(...)
```

You still need to use `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` when launching:

```bash
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python my_denoising_pipeline.py
```

**Memory Cleanup Between Stages:**

By default, pipelines clean GPU memory (especially transformer weights) between stages. If you have enough memory, you can skip this cleanup to reduce running time:

```python
# In pipeline implementations, memory cleanup happens automatically
# between stages. For custom pipelines, you can skip:
# utils.cleanup_memory()  # Comment out if you have enough VRAM
```

### Denoising Loop Optimization

**Gradient Estimation Denoising Loop:**

Instead of the standard Euler denoising loop, you can use gradient estimation for fewer steps (~20-30 instead of 40):

```python
from ltx_pipelines.utils import gradient_estimating_euler_denoising_loop

# Use gradient estimation denoising loop
def denoising_loop(sigmas, video_state, audio_state, stepper):
    return gradient_estimating_euler_denoising_loop(
        sigmas=sigmas,
        video_state=video_state,
        audio_state=audio_state,
        stepper=stepper,
        denoise_fn=your_denoise_function,
        ge_gamma=2.0,  # Gradient estimation coefficient
    )
```

This allows you to use **20-30 steps instead of 40** while maintaining quality. The gradient estimation function is available in [`pipeline_utils.py`](src/ltx_pipelines/utils/helpers.py).

---

## 🔧 Requirements

- **LTX-2 Model Checkpoint** - Local `.safetensors` file
- **Gemma Text Encoder** - Local Gemma model directory
- **Spatial Upscaler** - Required for two-stage pipelines (except one-stage)
- **Distilled LoRA** - Required for two-stage pipelines (except one-stage and distilled)

---

## 📖 Example: Image-to-Video

```python
from ltx_core.loader import LTXV_LORA_COMFY_RENAMING_MAP, LoraPathStrengthAndSDOps
from ltx_pipelines.ti2vid_two_stages import TI2VidTwoStagesPipeline
from ltx_core.components.guiders import MultiModalGuiderParams

distilled_lora = [
    LoraPathStrengthAndSDOps(
        "/path/to/distilled_lora.safetensors",
        0.6,
        LTXV_LORA_COMFY_RENAMING_MAP
    ),
]

pipeline = TI2VidTwoStagesPipeline(
    checkpoint_path="/path/to/checkpoint.safetensors",
    distilled_lora=distilled_lora,
    spatial_upsampler_path="/path/to/upsampler.safetensors",
    gemma_root="/path/to/gemma",
    loras=[],
)

video_guider_params = MultiModalGuiderParams(
    cfg_scale=3.0,
    stg_scale=1.0,
    rescale_scale=0.7,
    modality_scale=3.0,
    skip_step=0,
    stg_blocks=[29],
)

audio_guider_params = MultiModalGuiderParams(
    cfg_scale=7.0,
    stg_scale=1.0,
    rescale_scale=0.7,
    modality_scale=3.0,
    skip_step=0,
    stg_blocks=[29],
)

# Generate video from image
pipeline(
    prompt="A serene landscape with mountains in the background",
    output_path="output.mp4",
    seed=42,
    height=512,
    width=768,
    num_frames=121,
    frame_rate=25.0,
    num_inference_steps=40,
    video_guider_params=video_guider_params,
    audio_guider_params=audio_guider_params,
    images=[ImageConditioningInput("input_image.jpg", 0, 1.0, 33)],  # Image at frame 0, strength 1.0, CRF 33
)
```

---

## 🔗 Related Projects

- **[LTX-Core](../ltx-core/)** - Core model implementation and inference components (schedulers, guiders, noisers, patchifiers)
- **[LTX-Trainer](../ltx-trainer/)** - Training and fine-tuning tools


Now let me look at the distilled pipeline (fastest) to understand how to script a concurrency test.


1	import logging
2	from collections.abc import Iterator
3
4	import torch
5
6	from ltx_core.components.noisers import GaussianNoiser
7	from ltx_core.loader import LoraPathStrengthAndSDOps
8	from ltx_core.loader.registry import Registry
9	from ltx_core.model.video_vae import TilingConfig, get_video_chunks_number
10	from ltx_core.quantization import QuantizationPolicy
11	from ltx_core.types import Audio
12	from ltx_pipelines.utils.args import (
13	    ImageConditioningInput,
14	    default_2_stage_distilled_arg_parser,
15	    detect_checkpoint_path,
16	)
17	from ltx_pipelines.utils.blocks import (
18	    AudioDecoder,
19	    DiffusionStage,
20	    ImageConditioner,
21	    PromptEncoder,
22	    VideoDecoder,
23	    VideoUpsampler,
24	)
25	from ltx_pipelines.utils.constants import (
26	    DISTILLED_SIGMA_VALUES,
27	    STAGE_2_DISTILLED_SIGMA_VALUES,
28	    detect_params,
29	)
30	from ltx_pipelines.utils.denoisers import SimpleDenoiser
31	from ltx_pipelines.utils.helpers import (
32	    assert_resolution,
33	    combined_image_conditionings,
34	    get_device,
35	)
36	from ltx_pipelines.utils.media_io import encode_video
37	from ltx_pipelines.utils.types import ModalitySpec
38
39
40	class DistilledPipeline:
41	    """
42	    Two-stage distilled video generation pipeline.
43	    Stage 1 generates video at half of the target resolution, then Stage 2 upsamples
44	    by 2x and refines with additional denoising steps for higher quality output.
45	    """
46
47	    def __init__(
48	        self,
49	        distilled_checkpoint_path: str,
50	        gemma_root: str,
51	        spatial_upsampler_path: str,
52	        loras: list[LoraPathStrengthAndSDOps],
53	        device: torch.device | None = None,
54	        quantization: QuantizationPolicy | None = None,
55	        registry: Registry | None = None,
56	        torch_compile: bool = False,
57	    ):
58	        self.device = device or get_device()
59	        self.dtype = torch.bfloat16
60
61	        self.prompt_encoder = PromptEncoder(
62	            distilled_checkpoint_path, gemma_root, self.dtype, self.device, registry=registry
63	        )
64	        self.image_conditioner = ImageConditioner(distilled_checkpoint_path, self.dtype, self.device, registry=registry)
65	        self.stage = DiffusionStage(
66	            distilled_checkpoint_path,
67	            self.dtype,
68	            self.device,
69	            loras=tuple(loras),
70	            quantization=quantization,
71	            registry=registry,
72	            torch_compile=torch_compile,
73	        )
74	        self.upsampler = VideoUpsampler(
75	            distilled_checkpoint_path, spatial_upsampler_path, self.dtype, self.device, registry=registry
76	        )
77	        self.video_decoder = VideoDecoder(distilled_checkpoint_path, self.dtype, self.device, registry=registry)
78	        self.audio_decoder = AudioDecoder(distilled_checkpoint_path, self.dtype, self.device, registry=registry)
79
80	    def __call__(
81	        self,
82	        prompt: str,
83	        seed: int,
84	        height: int,
85	        width: int,
86	        num_frames: int,
87	        frame_rate: float,
88	        images: list[ImageConditioningInput],
89	        tiling_config: TilingConfig | None = None,
90	        enhance_prompt: bool = False,
91	        streaming_prefetch_count: int | None = None,
92	    ) -> tuple[Iterator[torch.Tensor], Audio]:
93	        assert_resolution(height=height, width=width, is_two_stage=True)
94
95	        generator = torch.Generator(device=self.device).manual_seed(seed)
96	        noiser = GaussianNoiser(generator=generator)
97	        dtype = torch.bfloat16
98
99	        (ctx_p,) = self.prompt_encoder(
100	            [prompt],
101	            enhance_first_prompt=enhance_prompt,
102	            enhance_prompt_image=images[0][0] if len(images) > 0 else None,
103	            streaming_prefetch_count=streaming_prefetch_count,
104	        )
105	        video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding
106
107	        # Stage 1: Initial low resolution video generation.
108	        stage_1_sigmas = torch.Tensor(DISTILLED_SIGMA_VALUES).to(self.device)
109	        stage_1_w, stage_1_h = width // 2, height // 2
110	        stage_1_conditionings = self.image_conditioner(
111	            lambda enc: combined_image_conditionings(
112	                images=images,
113	                height=stage_1_h,
114	                width=stage_1_w,
115	                video_encoder=enc,
116	                dtype=dtype,
117	                device=self.device,
118	            )
119	        )
120
121	        video_state, audio_state = self.stage(
122	            denoiser=SimpleDenoiser(video_context, audio_context),
123	            sigmas=stage_1_sigmas,
124	            noiser=noiser,
125	            width=stage_1_w,
126	            height=stage_1_h,
127	            frames=num_frames,
128	            fps=frame_rate,
129	            video=ModalitySpec(context=video_context, conditionings=stage_1_conditionings),
130	            audio=ModalitySpec(context=audio_context),
131	            streaming_prefetch_count=streaming_prefetch_count,
132	        )
133
134	        # Stage 2: Upsample and refine the video at higher resolution with distilled LORA.
135	        upscaled_video_latent = self.upsampler(video_state.latent[:1])
136
137	        stage_2_sigmas = torch.Tensor(STAGE_2_DISTILLED_SIGMA_VALUES).to(self.device)
138	        stage_2_conditionings = self.image_conditioner(
139	            lambda enc: combined_image_conditionings(
140	                images=images,
141	                height=height,
142	                width=width,
143	                video_encoder=enc,
144	                dtype=dtype,
145	                device=self.device,
146	            )
147	        )
148
149	        video_state, audio_state = self.stage(
150	            denoiser=SimpleDenoiser(video_context, audio_context),
151	            sigmas=stage_2_sigmas,
152	            noiser=noiser,
153	            width=width,
154	            height=height,
155	            frames=num_frames,
156	            fps=frame_rate,
157	            video=ModalitySpec(
158	                context=video_context,
159	                conditionings=stage_2_conditionings,
160	                noise_scale=stage_2_sigmas[0].item(),
161	                initial_latent=upscaled_video_latent,
162	            ),
163	            audio=ModalitySpec(
164	                context=audio_context,
165	                noise_scale=stage_2_sigmas[0].item(),
166	                initial_latent=audio_state.latent,
167	            ),
168	            streaming_prefetch_count=streaming_prefetch_count,
169	        )
170
171	        decoded_video = self.video_decoder(video_state.latent, tiling_config, generator)
172	        decoded_audio = self.audio_decoder(audio_state.latent)
173	        return decoded_video, decoded_audio
174
175
176	@torch.inference_mode()
177	def main() -> None:
178	    logging.getLogger().setLevel(logging.INFO)
179	    checkpoint_path = detect_checkpoint_path(distilled=True)
180	    params = detect_params(checkpoint_path)
181	    parser = default_2_stage_distilled_arg_parser(params=params)
182	    args = parser.parse_args()
183	    pipeline = DistilledPipeline(
184	        distilled_checkpoint_path=args.distilled_checkpoint_path,
185	        spatial_upsampler_path=args.spatial_upsampler_path,
186	        gemma_root=args.gemma_root,
187	        loras=tuple(args.lora) if args.lora else (),
188	        quantization=args.quantization,
189	        torch_compile=args.compile,
190	    )
191	    tiling_config = TilingConfig.default()
192	    video_chunks_number = get_video_chunks_number(args.num_frames, tiling_config)
193	    video, audio = pipeline(
194	        prompt=args.prompt,
195	        seed=args.seed,
196	        height=args.height,
197	        width=args.width,
198	        num_frames=args.num_frames,
199	        frame_rate=args.frame_rate,
200	        images=args.images,
201	        tiling_config=tiling_config,
202	        enhance_prompt=args.enhance_prompt,
203	        streaming_prefetch_count=args.streaming_prefetch_count,
204	    )
205
206	    encode_video(
207	        video=video,
208	        fps=args.frame_rate,
209	        audio=audio,
210	        output_path=args.output_path,
211	        video_chunks_number=video_chunks_number,
212	    )
213
214
215	if __name__ == "__main__":
216	    main()
217

import logging
from collections.abc import Iterator

import torch

from ltx_core.components.noisers import GaussianNoiser
from ltx_core.loader import LoraPathStrengthAndSDOps
from ltx_core.loader.registry import Registry
from ltx_core.model.video_vae import TilingConfig, get_video_chunks_number
from ltx_core.quantization import QuantizationPolicy
from ltx_core.types import Audio
from ltx_pipelines.utils.args import (
    ImageConditioningInput,
    default_2_stage_distilled_arg_parser,
    detect_checkpoint_path,
)
from ltx_pipelines.utils.blocks import (
    AudioDecoder,
    DiffusionStage,
    ImageConditioner,
    PromptEncoder,
    VideoDecoder,
    VideoUpsampler,
)
from ltx_pipelines.utils.constants import (
    DISTILLED_SIGMA_VALUES,
    STAGE_2_DISTILLED_SIGMA_VALUES,
    detect_params,
)
from ltx_pipelines.utils.denoisers import SimpleDenoiser
from ltx_pipelines.utils.helpers import (
    assert_resolution,
    combined_image_conditionings,
    get_device,
)
from ltx_pipelines.utils.media_io import encode_video
from ltx_pipelines.utils.types import ModalitySpec

class DistilledPipeline:
    """
    Two-stage distilled video generation pipeline.
    Stage 1 generates video at half of the target resolution, then Stage 2 upsamples
    by 2x and refines with additional denoising steps for higher quality output.
    """

    def __init__(
        self,
        distilled_checkpoint_path: str,
        gemma_root: str,
        spatial_upsampler_path: str,
        loras: list[LoraPathStrengthAndSDOps],
        device: torch.device | None = None,
        quantization: QuantizationPolicy | None = None,
        registry: Registry | None = None,
        torch_compile: bool = False,
    ):
        self.device = device or get_device()
        self.dtype = torch.bfloat16

        self.prompt_encoder = PromptEncoder(
            distilled_checkpoint_path, gemma_root, self.dtype, self.device, registry=registry
        )
        self.image_conditioner = ImageConditioner(distilled_checkpoint_path, self.dtype, self.device, registry=registry)
        self.stage = DiffusionStage(
            distilled_checkpoint_path,
            self.dtype,
            self.device,
            loras=tuple(loras),
            quantization=quantization,
            registry=registry,
            torch_compile=torch_compile,
        )
        self.upsampler = VideoUpsampler(
            distilled_checkpoint_path, spatial_upsampler_path, self.dtype, self.device, registry=registry
        )
        self.video_decoder = VideoDecoder(distilled_checkpoint_path, self.dtype, self.device, registry=registry)
        self.audio_decoder = AudioDecoder(distilled_checkpoint_path, self.dtype, self.device, registry=registry)

    def __call__(
        self,
        prompt: str,
        seed: int,
        height: int,
        width: int,
        num_frames: int,
        frame_rate: float,
        images: list[ImageConditioningInput],
        tiling_config: TilingConfig | None = None,
        enhance_prompt: bool = False,
        streaming_prefetch_count: int | None = None,
    ) -> tuple[Iterator[torch.Tensor], Audio]:
        assert_resolution(height=height, width=width, is_two_stage=True)

        generator = torch.Generator(device=self.device).manual_seed(seed)
        noiser = GaussianNoiser(generator=generator)
        dtype = torch.bfloat16

        (ctx_p,) = self.prompt_encoder(
            [prompt],
            enhance_first_prompt=enhance_prompt,
            enhance_prompt_image=images[0][0] if len(images) > 0 else None,
            streaming_prefetch_count=streaming_prefetch_count,
        )
        video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding

        # Stage 1: Initial low resolution video generation.
        stage_1_sigmas = torch.Tensor(DISTILLED_SIGMA_VALUES).to(self.device)
        stage_1_w, stage_1_h = width // 2, height // 2
        stage_1_conditionings = self.image_conditioner(
            lambda enc: combined_image_conditionings(
                images=images,
                height=stage_1_h,
                width=stage_1_w,
                video_encoder=enc,
                dtype=dtype,
                device=self.device,
            )
        )

        video_state, audio_state = self.stage(
            denoiser=SimpleDenoiser(video_context, audio_context),
            sigmas=stage_1_sigmas,
            noiser=noiser,
            width=stage_1_w,
            height=stage_1_h,
            frames=num_frames,
            fps=frame_rate,
            video=ModalitySpec(context=video_context, conditionings=stage_1_conditionings),
            audio=ModalitySpec(context=audio_context),
            streaming_prefetch_count=streaming_prefetch_count,
        )

        # Stage 2: Upsample and refine the video at higher resolution with distilled LORA.
        upscaled_video_latent = self.upsampler(video_state.latent[:1])

        stage_2_sigmas = torch.Tensor(STAGE_2_DISTILLED_SIGMA_VALUES).to(self.device)
        stage_2_conditionings = self.image_conditioner(
            lambda enc: combined_image_conditionings(
                images=images,
                height=height,
                width=width,
                video_encoder=enc,
                dtype=dtype,
                device=self.device,
            )
        )

        video_state, audio_state = self.stage(
            denoiser=SimpleDenoiser(video_context, audio_context),
            sigmas=stage_2_sigmas,
            noiser=noiser,
            width=width,
            height=height,
            frames=num_frames,
            fps=frame_rate,
            video=ModalitySpec(
                context=video_context,
                conditionings=stage_2_conditionings,
                noise_scale=stage_2_sigmas[0].item(),
                initial_latent=upscaled_video_latent,
            ),
            audio=ModalitySpec(
                context=audio_context,
                noise_scale=stage_2_sigmas[0].item(),
                initial_latent=audio_state.latent,
            ),
            streaming_prefetch_count=streaming_prefetch_count,
        )

        decoded_video = self.video_decoder(video_state.latent, tiling_config, generator)
        decoded_audio = self.audio_decoder(audio_state.latent)
        return decoded_video, decoded_audio

@torch.inference_mode()
def main() -> None:
    logging.getLogger().setLevel(logging.INFO)
    checkpoint_path = detect_checkpoint_path(distilled=True)
    params = detect_params(checkpoint_path)
    parser = default_2_stage_distilled_arg_parser(params=params)
    args = parser.parse_args()
    pipeline = DistilledPipeline(
        distilled_checkpoint_path=args.distilled_checkpoint_path,
        spatial_upsampler_path=args.spatial_upsampler_path,
        gemma_root=args.gemma_root,
        loras=tuple(args.lora) if args.lora else (),
        quantization=args.quantization,
        torch_compile=args.compile,
    )
    tiling_config = TilingConfig.default()
    video_chunks_number = get_video_chunks_number(args.num_frames, tiling_config)
    video, audio = pipeline(
        prompt=args.prompt,
        seed=args.seed,
        height=args.height,
        width=args.width,
        num_frames=args.num_frames,
        frame_rate=args.frame_rate,
        images=args.images,
        tiling_config=tiling_config,
        enhance_prompt=args.enhance_prompt,
        streaming_prefetch_count=args.streaming_prefetch_count,
    )

    encode_video(
        video=video,
        fps=args.frame_rate,
        audio=audio,
        output_path=args.output_path,
        video_chunks_number=video_chunks_number,
    )

if __name__ == "__main__":
    main()


Now let me check what models we need to download and look at the args parser.