can u check where the modei audio dataset


Let me look for the actual dataset directories and any config files that reference dataset paths.


Bash

ls -d /home/ubuntu/data* /home/ubuntu/*/data* 2>/dev/null | head -20

List data directories in home


/home/ubuntu/hf_staging/dataset_combined.jsonl
/home/ubuntu/hpml_nyu/data
/home/ubuntu/lewm-tts/dataset.py
/home/ubuntu/lewm-tts/dataset_v5.py
/home/ubuntu/modi_dataset/dataset_info.json
/home/ubuntu/modi_vibevoice_dataset/data-00000-of-00001.arrow
/home/ubuntu/modi_vibevoice_dataset/dataset_info.json
/home/ubuntu/training/datasets
/home/ubuntu/transcripts/data


ls -d /home/ubuntu/maya3*/data /home/ubuntu/spark*/data /home/ubuntu/lewm*/data /home/ubuntu/LeWM*/data 2>/dev/null; find /home/ubuntu -maxdepth 3 -type d -name "data" 2>/dev/null | grep -v venv | grep -v node_modules | grep -v site-packages | grep -v .cursor | grep -v .npm | grep -v .local | head -20

Find data directories excluding venvs


/home/ubuntu/tts/tts/data
/home/ubuntu/tts/tools/data
/home/ubuntu/hpml_nyu/data
/home/ubuntu/transcripts/data
/home/ubuntu/training/indic-asr-benchmark-6k/data
/home/ubuntu/training/maya-tts-llama32-1b-pretraining/data


Found it! I think you meant **"modi"** dataset. There are two directories:

- **`/home/ubuntu/modi_dataset/`** — contains `dataset_info.json`
- **`/home/ubuntu/modi_vibevoice_dataset/`** — contains `data-00000-of-00001.arrow` and `dataset_info.json`

Let me check what's in them:


ls -lah /home/ubuntu/modi_dataset/ && echo "---" && ls -lah /home/ubuntu/modi_vibevoice_dataset/

List contents of both modi dataset dirs


total 24K
drwxrwxr-x  2 ubuntu ubuntu 4.0K Mar 25 02:06 .
drwxr-x--x 80 ubuntu ubuntu  12K Mar 27 21:39 ..
-rw-rw-r--  1 ubuntu ubuntu  240 Mar 14 18:47 dataset_info.json
-rw-rw-r--  1 ubuntu ubuntu  601 Mar 14 18:47 state.json
---
total 44M
drwxrwxr-x  2 ubuntu ubuntu 4.0K Mar 25 02:05 .
drwxr-x--x 80 ubuntu ubuntu  12K Mar 27 21:39 ..
-rw-rw-r--  1 ubuntu ubuntu  44M Mar 16 06:10 data-00000-of-00001.arrow
-rw-rw-r--  1 ubuntu ubuntu  240 Mar 16 06:14 dataset_info.json
-rw-rw-r--  1 ubuntu ubuntu  601 Mar 16 06:14 state.json


     1→{
     2→  "citation": "",
     3→  "description": "",
     4→  "features": {
     5→    "audio": {
     6→      "sampling_rate": 24000,
     7→      "_type": "Audio"
     8→    },
     9→    "text": {
    10→      "dtype": "string",
    11→      "_type": "Value"
    12→    }
    13→  },
    14→  "homepage": "",
    15→  "license": ""
    16→}


Here's what I found for your **Modi** audio dataset:

| Directory | Size | Contents |
|---|---|---|
| `/home/ubuntu/modi_dataset/` | ~empty (just metadata) | `dataset_info.json`, `state.json` — no arrow data file |
| `/home/ubuntu/modi_vibevoice_dataset/` | **44MB** | `data-00000-of-00001.arrow` + metadata |

Both are HuggingFace-format datasets with **audio at 24kHz** and **text** columns.

The main one with actual data is **`/home/ubuntu/modi_vibevoice_dataset/`** (44MB arrow file). The `modi_dataset/` directory appears to have the same schema but is missing the data file — it may have been an earlier/incomplete version.


somehere we have 20hrs of modi proceesed data check


Agent

Find 20hr Modi processed audio data

Prompt:
Search thoroughly for a large processed audio dataset (~20 hours) related to "modi" anywhere on this machine. Look for:
1. Large directories with many .wav/.flac/.mp3 files
2. Large .arrow/.parquet/.tar files that could contain hours of audio
3. Any references to "modi" in config files, scripts, or manifests
4. Check /home/ubuntu and subdirectories, also /data, /mnt, /tmp, /opt if they exist
5. Look for manifest files (.json, .jsonl, .csv, .tsv) that reference modi audio paths
6. Check training configs that reference dataset paths with "modi"

Focus on finding where the bulk 20-hour processed dataset lives, not just the small 44MB one we already found.


The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). STOP what you are doing and wait for the user to tell you how to proceed.


[Request interrupted by user for tool use]


'/home/ubuntu/lewm-tts' for this what dataset we used?


Bash

List lewm-tts directory


dataset.py
dataset_v5.py
inference_engine.py
inference.py
inference_v5.py
model.py
model_v5.py
output
output_v3
output_v3b
output_v3c
output_v4
output_v5
output_v5_pred
output_v5_pred_dur
output_v5_pred_mask
preprocess_codec.py
preprocess.py
processed_data
processed_data_100mel
processed_data_codec
__pycache__
README.md
run_v1.sh
sample_ar_prompted_v3b.wav
sample_ar_prompted.wav
sample_ar_v3c.wav
sample_gt_0.wav
sample_gt_100.wav
sample_gt_500.wav
sample_gt_v3b.wav
sample_gt_v4_0.wav
sample_gt_v4_100.wav
sample_gt_v4_500.wav
sample_recon_ep43_0.wav
sample_recon_ep43_100.wav
sample_recon_ep43_500.wav
sample_reconstruct_v3.wav
sample_recon_v3b_0.wav
sample_recon_v3b_100.wav
sample_recon_v3b_500.wav
sample_recon_v3b.wav
sample_recon_v3c_0.wav
sample_recon_v3c_100.wav
sample_recon_v3c_500.wav
sample_recon_v4_0.wav
sample_recon_v4_100.wav
sample_recon_v4_500.wav
samples_v3c
samples_v4c
samples_v4_ep120
samples_v5
samples_v5b
samples_v5_cfg
samples_v5_ep120
samples_v5_ep50
samples_v5_mask
samples_v5_newtext
samples_v5_pred
samples_v5_pred_ep44
samples_v5_prompted
sample_synth_ep43.wav
sample_synth_v3b.wav
sample_synth_v3.wav
sample_synth_v4_0.wav
sample_synth_v4_1.wav
sample_tf_0.wav
sample_tf_100.wav
sample_tf_500.wav
sample_tf_v3b_0.wav
sample_tf_v3b_100.wav
sample_tf_v3c_0.wav
sample_tf_v4_0.wav
test_gt.wav
test_noisy_same_l1.wav
test_our_mel_vocos.wav
test_recon_v3c.wav
test_vocos_roundtrip.wav
train.py
train_v3b.log
train_v3c.log
train_v3.log
train_v4.log
train_v5.log
train_v5_pred.log
train_v5_pred.py
train_v5.py
v2


lewm-tts/train_v5_pred.py:20:from dataset_v5 import build_dataloader
lewm-tts/train_v5_pred.py:85:    manifest_path = os.path.join(args.data_dir, "manifest.json")
lewm-tts/train_v5_pred.py:86:    loader, dataset = build_dataloader(
lewm-tts/train_v5_pred.py:87:        manifest_path, batch_size=args.batch_size,
lewm-tts/train_v5_pred.py:90:    print(f"Training: {len(dataset)} samples, {len(loader)} batches/epoch")
lewm-tts/train_v5_pred.py:283:    parser.add_argument("--data_dir", default="processed_data_codec")
lewm-tts/train.py:14:from dataset import build_dataloader
lewm-tts/train.py:44:    manifest_path = os.path.join(args.data_dir, "manifest.json")
lewm-tts/train.py:45:    loader, dataset = build_dataloader(
lewm-tts/train.py:46:        manifest_path, batch_size=args.batch_size,
lewm-tts/train.py:49:    print(f"Training: {len(dataset)} samples, {len(loader)} batches/epoch")
lewm-tts/train.py:216:    parser.add_argument("--data_dir", default="/home/ubuntu/lewm-tts/processed_data")
lewm-tts/train_v5.py:14:from dataset_v5 import build_dataloader
lewm-tts/train_v5.py:37:    manifest_path = os.path.join(args.data_dir, "manifest.json")
lewm-tts/train_v5.py:38:    loader, dataset = build_dataloader(
lewm-tts/train_v5.py:39:        manifest_path, batch_size=args.batch_size,
lewm-tts/train_v5.py:42:    print(f"Training: {len(dataset)} samples, {len(loader)} batches/epoch")
lewm-tts/train_v5.py:173:    parser.add_argument("--data_dir", default="processed_data_codec")
lewm-tts/v2/train.py:12:from dataset import build_dataloader
lewm-tts/v2/train.py:33:    manifest_path = os.path.join(args.data_dir, "manifest.json")
lewm-tts/v2/train.py:34:    loader, dataset = build_dataloader(
lewm-tts/v2/train.py:35:        manifest_path, batch_size=args.batch_size, num_workers=args.num_workers,
lewm-tts/v2/train.py:37:    print(f"Training: {len(dataset)} samples, {len(loader)} batches/epoch")
lewm-tts/v2/train.py:165:    parser.add_argument("--data_dir", default="/home/ubuntu/lewm-tts/v2/processed_data")


lewm-tts/v2/dataset.py:60:    dataset = TTSDataset(manifest_path)
lewm-tts/v2/dataset.py:62:        dataset, batch_size=batch_size, shuffle=shuffle,
lewm-tts/v2/dataset.py:66:    return loader, dataset
lewm-tts/dataset.py:87:    dataset = TTSDataset(manifest_path)
lewm-tts/dataset.py:89:        dataset,
lewm-tts/dataset.py:97:    return loader, dataset
lewm-tts/dataset_v5.py:61:    dataset = CodecDataset(manifest_path, max_codec_frames=max_codec_frames)
lewm-tts/dataset_v5.py:63:        dataset, batch_size=batch_size, shuffle=True,
lewm-tts/dataset_v5.py:67:    return loader, dataset


ls /home/ubuntu/lewm-tts/processed_data/ | head -20 && echo "---" && ls /home/ubuntu/lewm-tts/processed_data/ | wc -l && echo "---" && du -sh /home/ubuntu/lewm-tts/processed_data/ /home/ubuntu/lewm-tts/processed_data_100mel/ /home/ubuntu/lewm-tts/processed_data_codec/ 2>/dev/null

Check processed data directories


config.json
manifest.json
---
2
---
3.1M	/home/ubuntu/lewm-tts/processed_data/
2.9G	/home/ubuntu/lewm-tts/processed_data_100mel/
2.9M	/home/ubuntu/lewm-tts/processed_data_codec/


     1→[
     2→  {
     3→    "id": 0,
     4→    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000000.wav",
     5→    "mel_path": "/home/ubuntu/lewm-tts/processed_data/mels/000000.pt",
     6→    "text": "मेरे प्यारे देशवासियों, नमस्कार। मन की बात में आपका स्वागत है, अभिन",
     7→    "duration": 9.52,
     8→    "mel_frames": 893
     9→  },
    10→  {
    11→    "id": 1,
    12→    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000001.wav",
    13→    "mel_path": "/home/ubuntu/lewm-tts/processed_data/mels/000001.pt",
    14→    "text": "ंदन है। मन की बात देश और देशवासियों की उपलब्धियों को सामने लान",
    15→    "duration": 8.8,
    16→    "mel_frames": 826
    17→  },
    18→  {
    19→    "id": 2,
    20→    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000002.wav",

[
  {
    "id": 0,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000000.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data/mels/000000.pt",
    "text": "मेरे प्यारे देशवासियों, नमस्कार। मन की बात में आपका स्वागत है, अभिन",
    "duration": 9.52,
    "mel_frames": 893
  },
  {
    "id": 1,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000001.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data/mels/000001.pt",
    "text": "ंदन है। मन की बात देश और देशवासियों की उपलब्धियों को सामने लान",
    "duration": 8.8,
    "mel_frames": 826
  },
  {
    "id": 2,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data/audio/000002.wav",


     1→"""
     2→Data preprocessing for LeWM TTS.
     3→- Loads Modi dataset from HuggingFace Arrow format
     4→- Segments long clips into 3-15 second chunks using silence detection
     5→- Extracts mel spectrograms
     6→- Saves processed dataset
     7→"""
     8→
     9→import os
    10→import json
    11→import numpy as np
    12→import torch
    13→import torchaudio
    14→from datasets import load_from_disk
    15→from pathlib import Path
    16→import argparse
    17→
    18→
    19→# ─── Mel spectrogram config ───
    20→MEL_CONFIG = {
    21→    "sample_rate": 24000,
    22→    "n_fft": 1024,
    23→    "hop_length": 256,
    24→    "n_mels": 100,
    25→    "f_min": 0,
    26→    "f_max": None,      # full bandwidth (matches Vocos mel-24khz)
    27→    "power": 1.0,
    28→    "clip_val": 1e-7,   # log clamp value (matches Vocos safe_log)
    29→}
    30→
    31→# ─── Segmentation config ───
    32→SEG_CONFIG = {
    33→    "min_duration": 3.0,   # seconds
    34→    "max_duration": 12.0,  # seconds
    35→    "target_duration": 8.0,  # preferred chunk size
    36→    "silence_threshold_db": -40,
    37→    "min_silence_len": 0.3,  # seconds
    38→}
    39→
    40→
    41→def detect_silence_points(waveform: np.ndarray, sr: int,
    42→                          threshold_db: float = -40,
    43→                          min_silence_len: float = 0.3) -> list:
    44→    """Find silence points in audio for splitting."""
    45→    # Convert to dB
    46→    frame_length = int(0.025 * sr)  # 25ms frames
    47→    hop = int(0.010 * sr)           # 10ms hop
    48→
    49→    # RMS energy per frame
    50→    n_frames = 1 + (len(waveform) - frame_length) // hop

"""
Data preprocessing for LeWM TTS.
- Loads Modi dataset from HuggingFace Arrow format
- Segments long clips into 3-15 second chunks using silence detection
- Extracts mel spectrograms
- Saves processed dataset
"""

import os
import json
import numpy as np
import torch
import torchaudio
from datasets import load_from_disk
from pathlib import Path
import argparse

# ─── Mel spectrogram config ───
MEL_CONFIG = {
    "sample_rate": 24000,
    "n_fft": 1024,
    "hop_length": 256,
    "n_mels": 100,
    "f_min": 0,
    "f_max": None,      # full bandwidth (matches Vocos mel-24khz)
    "power": 1.0,
    "clip_val": 1e-7,   # log clamp value (matches Vocos safe_log)
}

# ─── Segmentation config ───
SEG_CONFIG = {
    "min_duration": 3.0,   # seconds
    "max_duration": 12.0,  # seconds
    "target_duration": 8.0,  # preferred chunk size
    "silence_threshold_db": -40,
    "min_silence_len": 0.3,  # seconds
}

def detect_silence_points(waveform: np.ndarray, sr: int,
                          threshold_db: float = -40,
                          min_silence_len: float = 0.3) -> list:
    """Find silence points in audio for splitting."""
    # Convert to dB
    frame_length = int(0.025 * sr)  # 25ms frames
    hop = int(0.010 * sr)           # 10ms hop

    # RMS energy per frame
    n_frames = 1 + (len(waveform) - frame_length) // hop


14:from datasets import load_from_disk
201:def preprocess_dataset(input_path: str, output_path: str):
203:    print(f"Loading dataset from {input_path}...")
204:    ds = load_from_disk(input_path)
284:    parser.add_argument("--input", default="/home/ubuntu/modi_dataset")
287:    preprocess_dataset(args.input, args.output)


Here's the full picture:

**Source dataset:** `/home/ubuntu/modi_dataset` (loaded via HuggingFace `load_from_disk`)
- This is the **Modi "Mann Ki Baat"** Hindi speech dataset (24kHz, audio+text)

**Preprocessing** (`preprocess.py`):
- Segments long clips into 3-15s chunks using silence detection
- Extracts 100-band mel spectrograms
- Saves to processed directories

**Processed versions used for training:**

| Directory | Size | Used by |
|---|---|---|
| `processed_data/` | 3.1 MB | `train.py` (v1, mel-based) |
| `processed_data_100mel/` | **2.9 GB** | Likely the full 20hr processed set |
| `processed_data_codec/` | 2.9 MB | `train_v5.py`, `train_v5_pred.py` (codec-based) |

The **`processed_data_100mel/`** at 2.9GB is almost certainly where your full 20hrs of processed Modi data lives. The smaller `processed_data/` and `processed_data_codec/` contain manifests + compressed representations (mel `.pt` files or codec tokens), not raw audio.


wher is actually audio


ls /home/ubuntu/lewm-tts/processed_data_100mel/ | head -20

List processed_data_100mel contents


audio
config.json
manifest.json


ls /home/ubuntu/lewm-tts/processed_data_100mel/audio/ | head -10 && echo "---" && ls /home/ubuntu/lewm-tts/processed_data_100mel/audio/ | wc -l && echo "---" && du -sh /home/ubuntu/lewm-tts/processed_data_100mel/audio/

Check audio files count and size


000000.wav
000001.wav
000002.wav
000003.wav
000004.wav
000005.wav
000006.wav
000007.wav
000008.wav
000009.wav
---
7009
---
2.9G	/home/ubuntu/lewm-tts/processed_data_100mel/audio/


There it is:

**`/home/ubuntu/lewm-tts/processed_data_100mel/audio/`**

- **7,009 wav files**
- **2.9 GB** total
- Segmented chunks (3-15s each) from the Modi dataset

That's your ~20 hours of processed audio.


what is this melframe '/home/ubuntu/lewm-tts/processed_data_100mel/manifest.json'


     1→[
     2→  {
     3→    "id": 0,
     4→    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav",
     5→    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000000.pt",
     6→    "text": "मेरे प्यारे देशवासियों, नमस्कार। मन की बात में आपका स्वागत है, अभिन",
     7→    "duration": 9.52,
     8→    "mel_frames": 893
     9→  },
    10→  {
    11→    "id": 1,
    12→    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000001.wav",
    13→    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000001.pt",
    14→    "text": "ंदन है। मन की बात देश और देशवासियों की उपलब्धियों को सामने लान",
    15→    "duration": 8.8,
    16→    "mel_frames": 826
    17→  },
    18→  {
    19→    "id": 2,
    20→    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000002.wav",
    21→    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000002.pt",
    22→    "text": "े का एक मजबूत platform है। देश ने ऐसी ही उपलब्धि अभी दिल्ली में ह",
    23→    "duration": 9.14,
    24→    "mel_frames": 857
    25→  },
    26→  {
    27→    "id": 3,
    28→    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000003.wav",
    29→    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000003.pt",
    30→    "text": "Global AI Impact Summit के दौरान देखी। कई देशों के नेता, उद्योग जगत के leade",

[
  {
    "id": 0,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000000.pt",
    "text": "मेरे प्यारे देशवासियों, नमस्कार। मन की बात में आपका स्वागत है, अभिन",
    "duration": 9.52,
    "mel_frames": 893
  },
  {
    "id": 1,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000001.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000001.pt",
    "text": "ंदन है। मन की बात देश और देशवासियों की उपलब्धियों को सामने लान",
    "duration": 8.8,
    "mel_frames": 826
  },
  {
    "id": 2,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000002.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000002.pt",
    "text": "े का एक मजबूत platform है। देश ने ऐसी ही उपलब्धि अभी दिल्ली में ह",
    "duration": 9.14,
    "mel_frames": 857
  },
  {
    "id": 3,
    "audio_path": "/home/ubuntu/lewm-tts/processed_data_100mel/audio/000003.wav",
    "mel_path": "/home/ubuntu/lewm-tts/processed_data_100mel/mels/000003.pt",
    "text": "Global AI Impact Summit के दौरान देखी। कई देशों के नेता, उद्योग जगत के leade",


**`mel_frames`** = the number of time frames in the mel spectrogram for that audio clip.

It's calculated from the audio duration and the hop length:

```
mel_frames = (duration * sample_rate) / hop_length
```

For example, clip 0:
- duration = 9.52s, sample_rate = 24000, hop_length = 256
- `(9.52 × 24000) / 256 ≈ 893` frames

Each frame is a column in the mel spectrogram — a 100-dim vector (100 mel bins) representing the frequency content of a ~10.7ms window of audio. So `mel_frames: 893` means that clip's mel spectrogram is a `[100 × 893]` tensor, stored at the corresponding `mel_path`.


totak hrs we have?


python3 -c "
import json
with open('/home/ubuntu/lewm-tts/processed_data_100mel/manifest.json') as f:
    data = json.load(f)
total_sec = sum(item['duration'] for item in data)
print(f'Clips: {len(data)}')
print(f'Total: {total_sec:.1f} seconds = {total_sec/3600:.2f} hours')
"

Calculate total hours from manifest


Clips: 7009
Total: 64359.4 seconds = 17.88 hours


**~17.9 hours** across 7,009 clips.


https://huggingface.co/YatharthS/LuxTTS what is the possibity finetunig this ... making this work for so much concurency like 500 councurncy on a100


ToolSearch

Question: select:WebFetch


Question: select:WebFetch


WebFetch

Prompt: Extract all details about this TTS model: architecture, model size, parameters, what it's based on, supported languages, inference speed/RTF, how it works (e.g. is it autoregressive, diffusion, VITS-based, etc.), any finetuning instructions, dependencies, and hardware requirements. Include any benchmarks or performance numbers mentioned.


# LuxTTS Model Details

Based on the webpage content, here's what's documented about LuxTTS:

## Architecture & Approach
- **Based on**: ZipVoice, distilled to 4 steps
- **Vocoder**: 48kHz vocoder (higher quality than standard 24kHz)
- **Sampling**: Higher quality sampling technique than standard Euler

## Model Size & Performance
- **VRAM Requirement**: <1GB (fits on any local GPU)
- **Inference Speed**:
  - 150x realtime on single GPU
  - Faster than realtime on CPU
- **Audio Quality**: 48kHz speech generation (vs. 24kHz typical)

## Key Features
- Voice cloning capability (SOTA quality comparable to models 10x larger)
- Clear, high-fidelity speech synthesis
- Extremely efficient and lightweight

## Language Support
- **English** (as indicated by model tags)

## Usage & Resources
- **GitHub Repository**: https://github.com/ysharma3501/LuxTTS.git
- **Colab Notebook**: [Available for experimentation](https://colab.research.google.com/drive/1cDaxtbSDLRmu6tRV_781Of_GSjHSo1Cu?usp=sharing)

## License
- Apache-2.0

## Popularity
- 6,811 downloads last month
- 182 likes
- 26 Spaces using the model

**Note**: The webpage doesn't provide detailed information about parameter count, specific finetuning instructions, or complete dependency lists. For these details, refer to the [GitHub repository](https://github.com/ysharma3501/LuxTTS.git).


Prompt: Extract everything: architecture details, model parameters count, how inference works, finetuning support, dependencies, code structure, how voice cloning works, any batching/concurrency support, and any performance benchmarks. Is it based on DiT, flow matching, VITS, or what? What's ZipVoice?


# LuxTTS: Architecture & Implementation Analysis

## Model Foundation
LuxTTS is built on **ZipVoice architecture** but with significant optimizations. The README states it "uses the same architecture but distilled to 4 steps with an improved sampling technique." ZipVoice itself isn't detailed in this content, but references suggest it's from k2-fsa/ZipVoice repository.

## Key Technical Details

**Architecture Type**: Not explicitly stated as DiT, Flow Matching, or VITS in the provided content. The "4 steps" reference suggests a diffusion-based or iterative generation approach.

**Vocoder**: Custom 48kHz vocoder (vs. ZipVoice's 24kHz default), built on Vocos architecture.

**Parameter Count**: Not disclosed in the documentation.

**Inference Speed**:
- "150x realtime on a single GPU"
- "Faster than realtime on CPUs"
- Float32 current implementation; Float16 promised for "almost 2x" speedup

**Memory**: "Fits within 1gb vram"

## Voice Cloning Mechanism

1. **Prompt Encoding**: `encode_prompt(audio_file, duration, rms)` extracts voice characteristics from reference audio
2. **Generation**: `generate_speech(text, encoded_prompt, num_steps...)` produces speech
3. **Minimum Input**: 3-second reference audio recommended

## Inference Parameters

- `num_steps`: 3-4 optimal (quality vs. speed tradeoff)
- `t_shift`: 0.9 default (higher = better quality, worse WER)
- `speed`: Controls playback speed (lower = slower)
- `return_smooth`: Reduces artifacts but decreases clarity
- `ref_duration`: Lower values speed inference; higher (1000) reduces artifacts

## Dependencies
Listed in `requirements.txt` (not shown), but code imports indicate:
- PyTorch/transformers
- librosa (audio processing, ~10s initial load)
- soundfile (audio I/O)

## Code Structure
- `/zipvoice/luxvoice.py` contains main `LuxTTS` class
- Device support: CUDA, CPU (with threading), MPS (Apple Silicon)
- Python 100% codebase

## Batching/Concurrency
No explicit mention of batch processing or concurrent inference support.

## Performance Benchmarks
Only qualitative claims provided: "SOTA voice cloning on par with models 10x larger" — no quantitative metrics (MOS, WER, etc.) published.


Here's my analysis:

## LuxTTS Overview

- **Based on**: ZipVoice (distilled to 4 steps) — likely **flow-matching/diffusion** based
- **Vocoder**: 48kHz Vocos-based
- **VRAM**: <1GB, **150x realtime** on GPU
- **English only**, voice cloning with ~3s reference audio
- Apache 2.0 license

## Finetuning for Hindi/Modi

**Challenges:**
- No finetuning code/docs provided — you'd need to reverse-engineer the training pipeline from the ZipVoice base
- English-only currently — Hindi would need phoneme/text encoder changes
- ZipVoice itself is from k2-fsa and may have limited training documentation

**Feasibility:** Moderate effort. You'd need to:
1. Get the base ZipVoice training code
2. Adapt the text frontend for Hindi (Devanagari/IPA)
3. Finetune on your 18hr Modi dataset
4. Re-distill to 4 steps

## 500 Concurrency on A100 — Very Feasible

This is where LuxTTS shines:

| Factor | Details |
|---|---|
| Model VRAM | <1GB — A100 80GB can hold **many instances** |
| Speed | 150x realtime (fp32), ~300x with fp16 |
| 4-step generation | Very few forward passes per request |
| Bottleneck | Likely CPU-bound on text preprocessing, not GPU |

**Back-of-napkin math:**
- 10s of audio at 150x RT = ~67ms GPU time per request
- A100 can batch multiple requests simultaneously
- With dynamic batching (e.g., via Triton or vLLM-style serving): **500 concurrency is realistic** if average request is ~10s of audio
- You'd want: **Triton Inference Server** or **Ray Serve** with batched inference, fp16, and potentially 2-4 A100s for headroom

**The real question** is whether you need Hindi support. If English is fine, this is a strong candidate for high-concurrency serving. If you need Hindi, finetuning is doable but non-trivial given the lack of official training code.


can u check with english first what concurnecy we can get


Let me clone and test it.


Bash

nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader 2>/dev/null || echo "No GPU found"

Check GPU availability


NVIDIA A100-SXM4-80GB, 81920 MiB, 81156 MiB


A100 80GB with 81GB free. Let me set it up.


git clone https://github.com/ysharma3501/LuxTTS.git 2>&1 | tail -5

Clone LuxTTS repo


Cloning into 'LuxTTS'...


List repo contents


LICENSE
pyproject.toml
README.md
requirements.txt
zipvoice


     1→--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html
     2→
     3→torch
     4→torchaudio
     5→numpy
     6→lhotse
     7→huggingface_hub
     8→safetensors
     9→tensorboard
    10→vocos
    11→pydub
    12→onnxruntime
    13→git+https://github.com/ysharma3501/LinaCodec.git
    14→librosa
    15→transformers<=4.57.6
    16→
    17→# Normalization
    18→cn2an
    19→inflect
    20→
    21→# Tokenization
    22→jieba
    23→piper_phonemize
    24→pypinyin
    25→setuptools<81
    26→

--find-links https://k2-fsa.github.io/icefall/piper_phonemize.html

torch
torchaudio
numpy
lhotse
huggingface_hub
safetensors
tensorboard
vocos
pydub
onnxruntime
git+https://github.com/ysharma3501/LinaCodec.git
librosa
transformers<=4.57.6

# Normalization
cn2an
inflect

# Tokenization
jieba
piper_phonemize
pypinyin
setuptools<81


List zipvoice module


bin
__init__.py
luxvoice.py
modeling_utils.py
models
onnx_modeling.py
tokenizer
utils


cd /home/ubuntu/LuxTTS && pip install -e . 2>&1 | tail -10

Install LuxTTS and dependencies


vllm 0.16.0 requires protobuf!=6.30.*,!=6.31.*,!=6.32.*,!=6.33.0.*,!=6.33.1.*,!=6.33.2.*,!=6.33.3.*,!=6.33.4.*,>=5.29.6, but you have protobuf 4.25.8 which is incompatible.
voxcpm 1.5.0 requires datasets<4,>=3, but you have datasets 4.8.4 which is incompatible.
voxcpm 1.5.0 requires modelscope>=1.22.0, but you have modelscope 1.17.1 which is incompatible.
xcodec2 0.1.5 requires einops==0.8.0, but you have einops 0.8.2 which is incompatible.
xcodec2 0.1.5 requires numpy>=2.0.2, but you have numpy 1.26.4 which is incompatible.
xcodec2 0.1.5 requires torch==2.5.0, but you have torch 2.9.1 which is incompatible.
xcodec2 0.1.5 requires torchaudio==2.5.0, but you have torchaudio 2.9.1+cu128 which is incompatible.
xformers 0.0.35 requires torch>=2.10, but you have torch 2.9.1 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed Zipvoice-0.0.11 triton-3.5.1


     1→import torch
     2→from zipvoice.modeling_utils import process_audio, generate, load_models_gpu, load_models_cpu
     3→from zipvoice.onnx_modeling import generate_cpu
     4→
     5→class LuxTTS:
     6→    """
     7→    LuxTTS class for encoding prompt and generating speech on cpu/cuda/mps.
     8→    """
     9→
    10→    def __init__(self, model_path='YatharthS/LuxTTS', device='cuda', threads=4):
    11→        if model_path == 'YatharthS/LuxTTS':
    12→            model_path = None
    13→
    14→        # Auto-detect better device if cuda is requested but not available
    15→        if device == 'cuda' and not torch.cuda.is_available():
    16→            if torch.backends.mps.is_available():
    17→                print("CUDA not available, switching to MPS")
    18→                device = 'mps'
    19→            else:
    20→                print("CUDA not available, switching to CPU")
    21→                device = 'cpu'
    22→
    23→        if device == 'cpu':
    24→            model, feature_extractor, vocos, tokenizer, transcriber = load_models_cpu(model_path, threads)
    25→            print("Loading model on CPU")
    26→        else:
    27→            model, feature_extractor, vocos, tokenizer, transcriber = load_models_gpu(model_path, device=device)
    28→            print("Loading model on GPU")
    29→
    30→        self.model = model
    31→        self.feature_extractor = feature_extractor
    32→        self.vocos = vocos
    33→        self.tokenizer = tokenizer
    34→        self.transcriber = transcriber
    35→        self.device = device
    36→        self.vocos.freq_range = 12000
    37→
    38→
    39→
    40→    def encode_prompt(self, prompt_audio, duration=5, rms=0.001):
    41→        """encodes audio prompt according to duration and rms(volume control)"""
    42→        prompt_tokens, prompt_features_lens, prompt_features, prompt_rms = process_audio(prompt_audio, self.transcriber, self.tokenizer, self.feature_extractor, self.device, target_rms=rms, duration=duration)
    43→        encode_dict = {"prompt_tokens": prompt_tokens, 'prompt_features_lens': prompt_features_lens, 'prompt_features': prompt_features, 'prompt_rms': prompt_rms}
    44→
    45→        return encode_dict
    46→
    47→    def generate_speech(self, text, encode_dict, num_steps=4, guidance_scale=3.0, t_shift=0.5, speed=1.0, return_smooth=False):
    48→        """encodes text and generates speech using flow matching model according to steps, guidance scale, and t_shift(like temp)"""
    49→
    50→        prompt_tokens, prompt_features_lens, prompt_features, prompt_rms = encode_dict.values()
    51→
    52→        if return_smooth == True:
    53→            self.vocos.return_48k = False
    54→        else:
    55→            self.vocos.return_48k = True
    56→
    57→        if self.device == 'cpu':
    58→            final_wav = generate_cpu(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)
    59→        else:
    60→            final_wav = generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)
    61→
    62→        return final_wav.cpu()
    63→

import torch
from zipvoice.modeling_utils import process_audio, generate, load_models_gpu, load_models_cpu
from zipvoice.onnx_modeling import generate_cpu

class LuxTTS:
    """
    LuxTTS class for encoding prompt and generating speech on cpu/cuda/mps.
    """

    def __init__(self, model_path='YatharthS/LuxTTS', device='cuda', threads=4):
        if model_path == 'YatharthS/LuxTTS':
            model_path = None

        # Auto-detect better device if cuda is requested but not available
        if device == 'cuda' and not torch.cuda.is_available():
            if torch.backends.mps.is_available():
                print("CUDA not available, switching to MPS")
                device = 'mps'
            else:
                print("CUDA not available, switching to CPU")
                device = 'cpu'

        if device == 'cpu':
            model, feature_extractor, vocos, tokenizer, transcriber = load_models_cpu(model_path, threads)
            print("Loading model on CPU")
        else:
            model, feature_extractor, vocos, tokenizer, transcriber = load_models_gpu(model_path, device=device)
            print("Loading model on GPU")

        self.model = model
        self.feature_extractor = feature_extractor
        self.vocos = vocos
        self.tokenizer = tokenizer
        self.transcriber = transcriber
        self.device = device
        self.vocos.freq_range = 12000

    def encode_prompt(self, prompt_audio, duration=5, rms=0.001):
        """encodes audio prompt according to duration and rms(volume control)"""
        prompt_tokens, prompt_features_lens, prompt_features, prompt_rms = process_audio(prompt_audio, self.transcriber, self.tokenizer, self.feature_extractor, self.device, target_rms=rms, duration=duration)
        encode_dict = {"prompt_tokens": prompt_tokens, 'prompt_features_lens': prompt_features_lens, 'prompt_features': prompt_features, 'prompt_rms': prompt_rms}

        return encode_dict

    def generate_speech(self, text, encode_dict, num_steps=4, guidance_scale=3.0, t_shift=0.5, speed=1.0, return_smooth=False):
        """encodes text and generates speech using flow matching model according to steps, guidance scale, and t_shift(like temp)"""

        prompt_tokens, prompt_features_lens, prompt_features, prompt_rms = encode_dict.values()

        if return_smooth == True:
            self.vocos.return_48k = False
        else:
            self.vocos.return_48k = True

        if self.device == 'cpu':
            final_wav = generate_cpu(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)
        else:
            final_wav = generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)

        return final_wav.cpu()


Good — it's **flow matching** based. Let me write a concurrency benchmark.


     1→import argparse
     2→import datetime as dt
     3→import json
     4→import logging
     5→import os
     6→from pathlib import Path
     7→from typing import Optional
     8→
     9→import numpy as np
    10→import safetensors.torch
    11→import torch
    12→import librosa
    13→import torchaudio
    14→from transformers import pipeline
    15→from huggingface_hub import snapshot_download
    16→from lhotse.utils import fix_random_seed
    17→
    18→from zipvoice.models.zipvoice_distill import ZipVoiceDistill
    19→from zipvoice.tokenizer.tokenizer import EmiliaTokenizer
    20→from zipvoice.utils.checkpoint import load_checkpoint
    21→from zipvoice.utils.common import AttributeDict, str2bool
    22→from zipvoice.utils.feature import VocosFbank
    23→from zipvoice.utils.infer import rms_norm
    24→
    25→from dataclasses import dataclass, field
    26→from typing import Optional, List
    27→
    28→from linacodec.vocoder.vocos import Vocos
    29→from zipvoice.onnx_modeling import OnnxModel
    30→from torch.nn.utils import parametrize
    31→
    32→
    33→@dataclass
    34→class LuxTTSConfig:
    35→    # Model Setup
    36→    model_dir: Optional[str] = None
    37→    checkpoint_name: str = "model.pt"
    38→    vocoder_path: Optional[str] = None
    39→    trt_engine_path: Optional[str] = None
    40→
    41→    # Tokenizer & Language
    42→    tokenizer: str = "emilia"  # choices: ["emilia", "libritts", "espeak", "simple"]
    43→    lang: str = "en-us"
    44→
    45→
    46→@torch.inference_mode
    47→def process_audio(audio, transcriber, tokenizer, feature_extractor, device, target_rms=0.1, duration=4, feat_scale=0.1):
    48→    prompt_wav, sr = librosa.load(audio, sr=24000, duration=duration)
    49→    prompt_wav2, sr = librosa.load(audio, sr=16000, duration=duration)
    50→    prompt_text = transcriber(prompt_wav2)["text"]

import argparse
import datetime as dt
import json
import logging
import os
from pathlib import Path
from typing import Optional

import numpy as np
import safetensors.torch
import torch
import librosa
import torchaudio
from transformers import pipeline
from huggingface_hub import snapshot_download
from lhotse.utils import fix_random_seed

from zipvoice.models.zipvoice_distill import ZipVoiceDistill
from zipvoice.tokenizer.tokenizer import EmiliaTokenizer
from zipvoice.utils.checkpoint import load_checkpoint
from zipvoice.utils.common import AttributeDict, str2bool
from zipvoice.utils.feature import VocosFbank
from zipvoice.utils.infer import rms_norm

from dataclasses import dataclass, field
from typing import Optional, List

from linacodec.vocoder.vocos import Vocos
from zipvoice.onnx_modeling import OnnxModel
from torch.nn.utils import parametrize

@dataclass
class LuxTTSConfig:
    # Model Setup
    model_dir: Optional[str] = None
    checkpoint_name: str = "model.pt"
    vocoder_path: Optional[str] = None
    trt_engine_path: Optional[str] = None

    # Tokenizer & Language
    tokenizer: str = "emilia"  # choices: ["emilia", "libritts", "espeak", "simple"]
    lang: str = "en-us"

@torch.inference_mode
def process_audio(audio, transcriber, tokenizer, feature_extractor, device, target_rms=0.1, duration=4, feat_scale=0.1):
    prompt_wav, sr = librosa.load(audio, sr=24000, duration=duration)
    prompt_wav2, sr = librosa.load(audio, sr=16000, duration=duration)
    prompt_text = transcriber(prompt_wav2)["text"]


64:def generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, model, vocoder, tokenizer, num_step=4, guidance_scale=3.0, speed=1.0, t_shift=0.5, target_rms=0.1):


    64→def generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, model, vocoder, tokenizer, num_step=4, guidance_scale=3.0, speed=1.0, t_shift=0.5, target_rms=0.1):
    65→    tokens = tokenizer.texts_to_token_ids([text])
    66→    device = next(model.parameters()).device  # Auto-detect device
    67→
    68→    speed = speed * 1.3
    69→
    70→    with torch.inference_mode():
    71→        (pred_features, _, _, _) = model.sample(
    72→            tokens=tokens,
    73→            prompt_tokens=prompt_tokens,
    74→            prompt_features=prompt_features,
    75→            prompt_features_lens=prompt_features_lens,
    76→            speed=speed,
    77→            t_shift=t_shift,
    78→            duration='predict',
    79→            num_step=num_step,
    80→            guidance_scale=guidance_scale,
    81→        )
    82→
    83→    # Convert to waveform
    84→    pred_features = pred_features.permute(0, 2, 1) / 0.1
    85→    wav = vocoder.decode(pred_features).squeeze(1).clamp(-1, 1)
    86→
    87→    # Volume matching
    88→    if prompt_rms < target_rms:
    89→        wav = wav * (prompt_rms / target_rms)
    90→
    91→    return wav
    92→
    93→def load_models_gpu(model_path=None, device="cuda"):
    94→    params = LuxTTSConfig()
    95→    if model_path is None:
    96→        model_path = snapshot_download("YatharthS/LuxTTS")
    97→
    98→    token_file = f"{model_path}/tokens.txt"
    99→    model_ckpt = f"{model_path}/model.pt"
   100→    model_config = f"{model_path}/config.json"
   101→
   102→    transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-base", device=device)
   103→    tokenizer = EmiliaTokenizer(token_file=token_file)
   104→    tokenizer_config = {"vocab_size": tokenizer.vocab_size, "pad_id": tokenizer.pad_id}
   105→
   106→    with open(model_config, "r") as f:
   107→        model_config = json.load(f)
   108→
   109→    model = ZipVoiceDistill(
   110→        **model_config["model"],
   111→        **tokenizer_config,
   112→    )
   113→    load_checkpoint(filename=model_ckpt, model=model, strict=True)
   114→    params.device = torch.device(device, 0)
   115→
   116→    model = model.to(params.device).eval()
   117→    feature_extractor = VocosFbank()
   118→
   119→    vocos = Vocos.from_hparams(f'{model_path}/vocoder/config.yaml').to(device)
   120→    parametrize.remove_parametrizations(vocos.upsampler.upsample_layers[0], "weight")
   121→    parametrize.remove_parametrizations(vocos.upsampler.upsample_layers[1], "weight")
   122→    vocos.load_state_dict(torch.load(f'{model_path}/vocoder/vocos.bin', map_location=params.device))
   123→

def generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, model, vocoder, tokenizer, num_step=4, guidance_scale=3.0, speed=1.0, t_shift=0.5, target_rms=0.1):
    tokens = tokenizer.texts_to_token_ids([text])
    device = next(model.parameters()).device  # Auto-detect device

    speed = speed * 1.3

    with torch.inference_mode():
        (pred_features, _, _, _) = model.sample(
            tokens=tokens,
            prompt_tokens=prompt_tokens,
            prompt_features=prompt_features,
            prompt_features_lens=prompt_features_lens,
            speed=speed,
            t_shift=t_shift,
            duration='predict',
            num_step=num_step,
            guidance_scale=guidance_scale,
        )

    # Convert to waveform
    pred_features = pred_features.permute(0, 2, 1) / 0.1
    wav = vocoder.decode(pred_features).squeeze(1).clamp(-1, 1)

    # Volume matching
    if prompt_rms < target_rms:
        wav = wav * (prompt_rms / target_rms)

    return wav

def load_models_gpu(model_path=None, device="cuda"):
    params = LuxTTSConfig()
    if model_path is None:
        model_path = snapshot_download("YatharthS/LuxTTS")

    token_file = f"{model_path}/tokens.txt"
    model_ckpt = f"{model_path}/model.pt"
    model_config = f"{model_path}/config.json"

    transcriber = pipeline("automatic-speech-recognition", model="openai/whisper-base", device=device)
    tokenizer = EmiliaTokenizer(token_file=token_file)
    tokenizer_config = {"vocab_size": tokenizer.vocab_size, "pad_id": tokenizer.pad_id}

    with open(model_config, "r") as f:
        model_config = json.load(f)

    model = ZipVoiceDistill(
        **model_config["model"],
        **tokenizer_config,
    )
    load_checkpoint(filename=model_ckpt, model=model, strict=True)
    params.device = torch.device(device, 0)

    model = model.to(params.device).eval()
    feature_extractor = VocosFbank()

    vocos = Vocos.from_hparams(f'{model_path}/vocoder/config.yaml').to(device)
    parametrize.remove_parametrizations(vocos.upsampler.upsample_layers[0], "weight")
    parametrize.remove_parametrizations(vocos.upsampler.upsample_layers[1], "weight")
    vocos.load_state_dict(torch.load(f'{model_path}/vocoder/vocos.bin', map_location=params.device))


Now let me write and run the benchmark.


Write

"""
Benchmark LuxTTS concurrency on A100.
Tests: single request latency, then concurrent throughput.
"""
import torch
import time
import numpy as np
from concurrent.futures import ThreadPoolExecutor, as_completed
from zipvoice.luxvoice import LuxTTS
import soundfile as sf
import os

# --- Create a short reference audio if none exists ---
REF_AUDIO = "/home/ubuntu/LuxTTS/ref_audio.wav"
TEST_TEXT = "The quick brown fox jumps over the lazy dog. This is a test of the text to speech system for benchmarking purposes."
SHORT_TEXT = "Hello, this is a concurrency test."

def create_ref_audio():
    """Create a simple sine wave as reference audio."""
    if os.path.exists(REF_AUDIO):
        return
    sr = 24000
    duration = 5
    t = np.linspace(0, duration, sr * duration)
    # Simple tone
    audio = 0.3 * np.sin(2 * np.pi * 440 * t).astype(np.float32)
    sf.write(REF_AUDIO, audio, sr)
    print(f"Created reference audio: {REF_AUDIO}")

def benchmark():
    create_ref_audio()

    print("=" * 60)
    print("LuxTTS Concurrency Benchmark — A100 80GB")
    print("=" * 60)

    # Load model
    print("\n[1] Loading model...")
    t0 = time.time()
    tts = LuxTTS(device='cuda')
    load_time = time.time() - t0
    print(f"    Model loaded in {load_time:.1f}s")

    # Check GPU memory after load
    mem_used = torch.cuda.memory_allocated() / 1024**3
    mem_reserved = torch.cuda.memory_reserved() / 1024**3
    print(f"    GPU memory: {mem_used:.2f} GB allocated, {mem_reserved:.2f} GB reserved")

    # Encode prompt
    print("\n[2] Encoding reference prompt...")
    t0 = time.time()
    encoded = tts.encode_prompt(REF_AUDIO, duration=5)
    encode_time = time.time() - t0
    print(f"    Encoded in {encode_time:.2f}s")

    # --- Single request latency ---
    print("\n[3] Single request latency (warmup)...")
    for i in range(2):
        t0 = time.time()
        wav = tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
        elapsed = time.time() - t0
        audio_dur = len(wav[0]) / 48000
        rtf = elapsed / audio_dur
        print(f"    Run {i+1}: {elapsed:.3f}s wall, {audio_dur:.2f}s audio, RTF={rtf:.4f} ({1/rtf:.0f}x realtime)")

    # Save a sample
    sf.write("/home/ubuntu/LuxTTS/benchmark_sample.wav", wav[0].numpy(), 48000)

    # --- Longer text latency ---
    print("\n[4] Longer text latency...")
    t0 = time.time()
    wav = tts.generate_speech(TEST_TEXT, encoded, num_steps=4)
    elapsed = time.time() - t0
    audio_dur = len(wav[0]) / 48000
    rtf = elapsed / audio_dur
    print(f"    {elapsed:.3f}s wall, {audio_dur:.2f}s audio, RTF={rtf:.4f} ({1/rtf:.0f}x realtime)")

    mem_peak = torch.cuda.max_memory_allocated() / 1024**3
    print(f"    Peak GPU memory: {mem_peak:.2f} GB")

    # --- Sequential throughput baseline ---
    print("\n[5] Sequential throughput (10 requests)...")
    t0 = time.time()
    for _ in range(10):
        tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
    seq_time = time.time() - t0
    print(f"    10 requests in {seq_time:.2f}s = {10/seq_time:.1f} req/s")

    # --- Concurrent throughput with threads ---
    # Note: Python GIL + CUDA means threads serialize on GPU,
    # but this simulates real server concurrency patterns

    def do_request(idx):
        t0 = time.time()
        wav = tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
        elapsed = time.time() - t0
        audio_dur = len(wav[0]) / 48000
        return idx, elapsed, audio_dur

    for concurrency in [1, 5, 10, 25, 50, 100]:
        print(f"\n[6] Concurrent requests: {concurrency}...")
        torch.cuda.reset_peak_memory_stats()

        t0 = time.time()
        results = []
        with ThreadPoolExecutor(max_workers=concurrency) as executor:
            futures = [executor.submit(do_request, i) for i in range(concurrency)]
            for f in as_completed(futures):
                results.append(f.result())

        total_time = time.time() - t0
        latencies = [r[1] for r in results]
        audio_durs = [r[2] for r in results]
        total_audio = sum(audio_durs)
        peak_mem = torch.cuda.max_memory_allocated() / 1024**3

        print(f"    Total wall time: {total_time:.2f}s")
        print(f"    Throughput: {concurrency/total_time:.1f} req/s")
        print(f"    Audio generated: {total_audio:.1f}s total")
        print(f"    Latency — p50: {np.median(latencies):.3f}s, p95: {np.percentile(latencies, 95):.3f}s, max: {max(latencies):.3f}s")
        print(f"    Peak GPU mem: {peak_mem:.2f} GB")

    # --- Summary ---
    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"Model VRAM: {mem_used:.2f} GB")
    print(f"A100 total: 80 GB")
    print(f"Theoretical model copies: {int(80 / mem_used)} (if memory-bound)")
    print(f"Sequential req/s: {10/seq_time:.1f}")
    print("Note: True batched inference (not just threaded) would")
    print("need model code changes to batch the forward pass.")

if __name__ == "__main__":
    benchmark()


File created successfully at: /home/ubuntu/LuxTTS/benchmark_concurrency.py

"""
Benchmark LuxTTS concurrency on A100.
Tests: single request latency, then concurrent throughput.
"""
import torch
import time
import numpy as np
from concurrent.futures import ThreadPoolExecutor, as_completed
from zipvoice.luxvoice import LuxTTS
import soundfile as sf
import os

# --- Create a short reference audio if none exists ---
REF_AUDIO = "/home/ubuntu/LuxTTS/ref_audio.wav"
TEST_TEXT = "The quick brown fox jumps over the lazy dog. This is a test of the text to speech system for benchmarking purposes."
SHORT_TEXT = "Hello, this is a concurrency test."

def create_ref_audio():
    """Create a simple sine wave as reference audio."""
    if os.path.exists(REF_AUDIO):
        return
    sr = 24000
    duration = 5
    t = np.linspace(0, duration, sr * duration)
    # Simple tone
    audio = 0.3 * np.sin(2 * np.pi * 440 * t).astype(np.float32)
    sf.write(REF_AUDIO, audio, sr)
    print(f"Created reference audio: {REF_AUDIO}")

def benchmark():
    create_ref_audio()

    print("=" * 60)
    print("LuxTTS Concurrency Benchmark — A100 80GB")
    print("=" * 60)

    # Load model
    print("\n[1] Loading model...")
    t0 = time.time()
    tts = LuxTTS(device='cuda')
    load_time = time.time() - t0
    print(f"    Model loaded in {load_time:.1f}s")

    # Check GPU memory after load
    mem_used = torch.cuda.memory_allocated() / 1024**3
    mem_reserved = torch.cuda.memory_reserved() / 1024**3
    print(f"    GPU memory: {mem_used:.2f} GB allocated, {mem_reserved:.2f} GB reserved")

    # Encode prompt
    print("\n[2] Encoding reference prompt...")
    t0 = time.time()
    encoded = tts.encode_prompt(REF_AUDIO, duration=5)
    encode_time = time.time() - t0
    print(f"    Encoded in {encode_time:.2f}s")

    # --- Single request latency ---
    print("\n[3] Single request latency (warmup)...")
    for i in range(2):
        t0 = time.time()
        wav = tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
        elapsed = time.time() - t0
        audio_dur = len(wav[0]) / 48000
        rtf = elapsed / audio_dur
        print(f"    Run {i+1}: {elapsed:.3f}s wall, {audio_dur:.2f}s audio, RTF={rtf:.4f} ({1/rtf:.0f}x realtime)")

    # Save a sample
    sf.write("/home/ubuntu/LuxTTS/benchmark_sample.wav", wav[0].numpy(), 48000)

    # --- Longer text latency ---
    print("\n[4] Longer text latency...")
    t0 = time.time()
    wav = tts.generate_speech(TEST_TEXT, encoded, num_steps=4)
    elapsed = time.time() - t0
    audio_dur = len(wav[0]) / 48000
    rtf = elapsed / audio_dur
    print(f"    {elapsed:.3f}s wall, {audio_dur:.2f}s audio, RTF={rtf:.4f} ({1/rtf:.0f}x realtime)")

    mem_peak = torch.cuda.max_memory_allocated() / 1024**3
    print(f"    Peak GPU memory: {mem_peak:.2f} GB")

    # --- Sequential throughput baseline ---
    print("\n[5] Sequential throughput (10 requests)...")
    t0 = time.time()
    for _ in range(10):
        tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
    seq_time = time.time() - t0
    print(f"    10 requests in {seq_time:.2f}s = {10/seq_time:.1f} req/s")

    # --- Concurrent throughput with threads ---
    # Note: Python GIL + CUDA means threads serialize on GPU,
    # but this simulates real server concurrency patterns

    def do_request(idx):
        t0 = time.time()
        wav = tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
        elapsed = time.time() - t0
        audio_dur = len(wav[0]) / 48000
        return idx, elapsed, audio_dur

    for concurrency in [1, 5, 10, 25, 50, 100]:
        print(f"\n[6] Concurrent requests: {concurrency}...")
        torch.cuda.reset_peak_memory_stats()

        t0 = time.time()
        results = []
        with ThreadPoolExecutor(max_workers=concurrency) as executor:
            futures = [executor.submit(do_request, i) for i in range(concurrency)]
            for f in as_completed(futures):
                results.append(f.result())

        total_time = time.time() - t0
        latencies = [r[1] for r in results]
        audio_durs = [r[2] for r in results]
        total_audio = sum(audio_durs)
        peak_mem = torch.cuda.max_memory_allocated() / 1024**3

        print(f"    Total wall time: {total_time:.2f}s")
        print(f"    Throughput: {concurrency/total_time:.1f} req/s")
        print(f"    Audio generated: {total_audio:.1f}s total")
        print(f"    Latency — p50: {np.median(latencies):.3f}s, p95: {np.percentile(latencies, 95):.3f}s, max: {max(latencies):.3f}s")
        print(f"    Peak GPU mem: {peak_mem:.2f} GB")

    # --- Summary ---
    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"Model VRAM: {mem_used:.2f} GB")
    print(f"A100 total: 80 GB")
    print(f"Theoretical model copies: {int(80 / mem_used)} (if memory-bound)")
    print(f"Sequential req/s: {10/seq_time:.1f}")
    print("Note: True batched inference (not just threaded) would")
    print("need model code changes to batch the forward pass.")

if __name__ == "__main__":
    benchmark()


Bash

Run LuxTTS concurrency benchmark


Error: Exit code 127
/bin/bash: line 1: python: command not found


Run LuxTTS concurrency benchmark


Error: Exit code 1
WARNING:root:Failed import k2 with error No module named 'k2'. Swoosh functions will fallback to PyTorch implementation, leading to slower speed and higher memory consumption.
Created reference audio: /home/ubuntu/LuxTTS/ref_audio.wav
============================================================
LuxTTS Concurrency Benchmark — A100 80GB
============================================================

[1] Loading model...

Fetching 11 files:   0%|          | 0/11 [00:00<?, ?it/s]
Fetching 11 files:   9%|▉         | 1/11 [00:00<00:06,  1.45it/s]
Fetching 11 files:  36%|███▋      | 4/11 [00:08<00:15,  2.18s/it]
Fetching 11 files:  55%|█████▍    | 6/11 [00:08<00:06,  1.27s/it]
Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.30it/s]
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0d5fa590-1d4f-40c7-975a-4847b233c70e)')' thrown while requesting HEAD https://huggingface.co/openai/whisper-base/resolve/main/tokenizer.json
WARNING:huggingface_hub.utils._http:'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 0d5fa590-1d4f-40c7-975a-4847b233c70e)')' thrown while requesting HEAD https://huggingface.co/openai/whisper-base/resolve/main/tokenizer.json
Retrying in 1s [Retry 1/5].
WARNING:huggingface_hub.utils._http:Retrying in 1s [Retry 1/5].
Device set to use cuda
`return_token_timestamps` is deprecated for WhisperFeatureExtractor and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
Transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.
WARNING:root:No English or Chinese characters found,                             skipping segment of unknown language: ('…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………', 'other')
Loading model on GPU
    Model loaded in 26.6s
    GPU memory: 0.79 GB allocated, 0.88 GB reserved

[2] Encoding reference prompt...
............................................................................................................................................................................................................................................................................................................................................................................................................................................................
    Encoded in 4.52s

[3] Single request latency (warmup)...
Traceback (most recent call last):
  File "/home/ubuntu/LuxTTS/benchmark_concurrency.py", line 135, in <module>
    benchmark()
  File "/home/ubuntu/LuxTTS/benchmark_concurrency.py", line 60, in benchmark
    wav = tts.generate_speech(SHORT_TEXT, encoded, num_steps=4)
  File "/home/ubuntu/LuxTTS/zipvoice/luxvoice.py", line 60, in generate_speech
    final_wav = generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)
  File "/home/ubuntu/LuxTTS/zipvoice/modeling_utils.py", line 71, in generate
    (pred_features, _, _, _) = model.sample(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 425, in sample
    ) = self.forward_text_inference_ratio_duration(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 327, in forward_text_inference_ratio_duration
    text_condition, padding_mask = self.forward_text_condition(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 236, in forward_text_condition
    padding_mask = make_pad_mask(features_lens, max_len=num_frames)  # (B, T)
  File "/home/ubuntu/LuxTTS/zipvoice/utils/common.py", line 423, in make_pad_mask
    seq_range = torch.arange(0, max_len, device=lengths.device)
RuntimeError: upper bound and lower bound inconsistent with step sign


The sine wave reference isn't producing valid features. Let me use a real audio sample instead.


ls /home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav && soxi /home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav 2>/dev/null || python3 -c "import torchaudio; w,sr=torchaudio.load('/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav'); print(f'sr={sr}, dur={w.shape[1]/sr:.2f}s')"

Check real audio file


/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav

Input File     : '/home/ubuntu/lewm-tts/processed_data_100mel/audio/000000.wav'
Channels       : 1
Sample Rate    : 24000
Precision      : 16-bit
Duration       : 00:00:09.52 = 228480 samples ~ 714 CDDA sectors
File Size      : 457k
Bit Rate       : 384k
Sample Encoding: 16-bit Signed Integer PCM


That's Hindi audio — the model is English. Let me find/create a proper English reference.


# Generate English speech-like reference with espeak or gtts
python3 -c "
import numpy as np, soundfile as sf

# Generate a more speech-like signal (modulated noise + harmonics)
sr = 24000; dur = 5
t = np.linspace(0, dur, sr*dur)
# Fundamental + harmonics (speech-like)
f0 = 120  # male pitch
signal = np.zeros_like(t)
for h in range(1, 8):
    signal += (0.3/h) * np.sin(2*np.pi*f0*h*t)
# AM modulation (syllable-like)
mod = 0.5 + 0.5*np.sin(2*np.pi*3*t)
signal = (signal * mod * 0.5).astype(np.float32)
sf.write('/home/ubuntu/LuxTTS/ref_audio.wav', signal, sr)
print('wrote ref')
" 2>&1

Generate speech-like reference audio


wrote ref


python3 -c "
import torch, time
from zipvoice.luxvoice import LuxTTS

print('Loading model...')
tts = LuxTTS(device='cuda')
mem = torch.cuda.memory_allocated()/1024**3
print(f'GPU mem: {mem:.2f} GB')

print('Encoding prompt...')
encoded = tts.encode_prompt('ref_audio.wav', duration=5)

print('Generating speech...')
t0 = time.time()
wav = tts.generate_speech('Hello, this is a concurrency test for benchmarking.', encoded, num_steps=4)
elapsed = time.time() - t0
audio_dur = len(wav[0]) / 48000
print(f'Wall: {elapsed:.3f}s, Audio: {audio_dur:.2f}s, RTF: {elapsed/audio_dur:.4f}, Speed: {1/(elapsed/audio_dur):.0f}x realtime')
print(f'Peak mem: {torch.cuda.max_memory_allocated()/1024**3:.2f} GB')
" 2>&1

Test single LuxTTS inference


Error: Exit code 1
WARNING:root:Failed import k2 with error No module named 'k2'. Swoosh functions will fallback to PyTorch implementation, leading to slower speed and higher memory consumption.
Loading model...

Fetching 11 files:   0%|          | 0/11 [00:00<?, ?it/s]
Fetching 11 files: 100%|██████████| 11/11 [00:00<00:00, 128516.28it/s]
Device set to use cuda
`return_token_timestamps` is deprecated for WhisperFeatureExtractor and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
Transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.
WARNING:root:No English or Chinese characters found,                             skipping segment of unknown language: ('…………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………', 'other')
Traceback (most recent call last):
  File "<string>", line 15, in <module>
  File "/home/ubuntu/LuxTTS/zipvoice/luxvoice.py", line 60, in generate_speech
    final_wav = generate(prompt_tokens, prompt_features_lens, prompt_features, prompt_rms, text, self.model, self.vocos, self.tokenizer, num_step=num_steps, guidance_scale=guidance_scale, t_shift=t_shift, speed=speed)
  File "/home/ubuntu/LuxTTS/zipvoice/modeling_utils.py", line 71, in generate
    (pred_features, _, _, _) = model.sample(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 425, in sample
    ) = self.forward_text_inference_ratio_duration(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 327, in forward_text_inference_ratio_duration
    text_condition, padding_mask = self.forward_text_condition(
  File "/home/ubuntu/LuxTTS/zipvoice/models/zipvoice.py", line 236, in forward_text_condition
    padding_mask = make_pad_mask(features_lens, max_len=num_frames)  # (B, T)
  File "/home/ubuntu/LuxTTS/zipvoice/utils/common.py", line 423, in make_pad_mask
    seq_range = torch.arange(0, max_len, device=lengths.device)
RuntimeError: upper bound and lower bound inconsistent with step sign
Loading model on GPU
GPU mem: 0.79 GB
Encoding prompt...
............................................................................................................................................................................................................................................................................................................................................................................................................................................................
Generating speech...


The synthetic audio doesn't produce valid features. The Whisper transcriber outputs garbage, leading to 0-length features. Let me grab a real English speech sample.


# Use espeak to generate real English speech
espeak-ng "Hello, this is a test of the voice cloning system. We are benchmarking concurrency on the A100 GPU." -w /home/ubuntu/LuxTTS/ref_audio.wav --stdout 2>/dev/null | head -c 0
# Check if espeak is available
which espeak-ng 2>/dev/null || which espeak 2>/dev/null || echo "no espeak"
# Alternative: use piper or gtts
pip show gTTS 2>/dev/null | head -2 || echo "no gTTS"

Check TTS tools for generating reference


/usr/bin/espeak-ng
Name: gTTS
Version: 2.5.4


# Generate English reference with gTTS (Google TTS) - produces natural speech
python3 -c "
from gtts import gTTS
import io
from pydub import AudioSegment

tts = gTTS('Hello everyone, welcome to this demonstration of the text to speech system. We are testing voice quality and performance.', lang='en')
tts.save('/home/ubuntu/LuxTTS/ref_gtts.mp3')

# Convert to wav 24kHz mono
audio = AudioSegment.from_mp3('/home/ubuntu/LuxTTS/ref_gtts.mp3')
audio = audio.set_frame_rate(24000).set_channels(1)
audio.export('/home/ubuntu/LuxTTS/ref_audio.wav', format='wav')
print('Done')
" 2>&1

Generate English reference audio with gTTS


Done


Check reference audio details


Input File     : '/home/ubuntu/LuxTTS/ref_audio.wav'
Channels       : 1
Sample Rate    : 24000
Precision      : 16-bit
Duration       : 00:00:08.88 = 213120 samples ~ 666 CDDA sectors
File Size      : 426k
Bit Rate       : 384k
Sample Encoding: 16-bit Signed Integer PCM


Good, 8.88s of real English speech. Now let me run the benchmark.


Run LuxTTS concurrency benchmark


WARNING:root:Failed import k2 with error No module named 'k2'. Swoosh functions will fallback to PyTorch implementation, leading to slower speed and higher memory consumption.
============================================================
LuxTTS Concurrency Benchmark — A100 80GB
============================================================

[1] Loading model...

Fetching 11 files:   0%|          | 0/11 [00:00<?, ?it/s]
Fetching 11 files: 100%|██████████| 11/11 [00:00<00:00, 129599.28it/s]
Device set to use cuda
`return_token_timestamps` is deprecated for WhisperFeatureExtractor and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
Transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English. This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`. See https://github.com/huggingface/transformers/pull/28687 for more details.
/home/ubuntu/.local/lib/python3.10/site-packages/linacodec/vocoder/vocos.py:132: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
  with autocast(enabled=False):
Loading model on GPU
    Model loaded in 3.5s
    GPU memory: 0.79 GB allocated, 0.88 GB reserved

[2] Encoding reference prompt...
 Hello everyone. Welcome to this demonstration of the text to speech system.
    Encoded in 1.26s

[3] Single request latency (warmup)...
    Run 1: 0.630s wall, 1.75s audio, RTF=0.3602 (3x realtime)
    Run 2: 0.445s wall, 1.75s audio, RTF=0.2543 (4x realtime)

[4] Longer text latency...
    0.341s wall, 6.23s audio, RTF=0.0548 (18x realtime)
    Peak GPU memory: 1.02 GB

[5] Sequential throughput (10 requests)...
    10 requests in 2.17s = 4.6 req/s

[6] Concurrent requests: 1...
    Total wall time: 0.22s
    Throughput: 4.5 req/s
    Audio generated: 1.7s total
    Latency — p50: 0.220s, p95: 0.220s, max: 0.220s
    Peak GPU mem: 0.85 GB

[6] Concurrent requests: 5...
    Total wall time: 2.79s
    Throughput: 1.8 req/s
    Audio generated: 8.7s total
    Latency — p50: 2.778s, p95: 2.784s, max: 2.784s
    Peak GPU mem: 0.99 GB

[6] Concurrent requests: 10...
    Total wall time: 5.19s
    Throughput: 1.9 req/s
    Audio generated: 17.5s total
    Latency — p50: 5.159s, p95: 5.182s, max: 5.184s
    Peak GPU mem: 1.15 GB

[6] Concurrent requests: 25...
    Total wall time: 12.67s
    Throughput: 2.0 req/s
    Audio generated: 43.7s total
    Latency — p50: 12.607s, p95: 12.658s, max: 12.660s
    Peak GPU mem: 1.61 GB

[6] Concurrent requests: 50...
    Total wall time: 25.14s
    Throughput: 2.0 req/s
    Audio generated: 87.5s total
    Latency — p50: 24.987s, p95: 25.094s, max: 25.123s
    Peak GPU mem: 2.35 GB

[6] Concurrent requests: 100...
    Total wall time: 49.77s
    Throughput: 2.0 req/s
    Audio generated: 174.9s total
    Latency — p50: 49.368s, p95: 49.651s, max: 49.709s
    Peak GPU mem: 3.83 GB

============================================================
SUMMARY
============================================================
Model VRAM: 0.79 GB
A100 total: 80 GB
Theoretical model copies: 100 (if memory-bound)
Sequential req/s: 4.6
Note: True batched inference (not just threaded) would
need model code changes to batch the forward pass.


Here are the results:

## LuxTTS Benchmark Results — A100 80GB

### Single Request Performance
| Text Length | Wall Time | Audio Output | Realtime Factor |
|---|---|---|---|
| Short (~8 words) | 0.22s | 1.75s audio | **4x realtime** |
| Long (~20 words) | 0.34s | 6.23s audio | **18x realtime** |

### Concurrency (threaded, single model)
| Concurrent Reqs | Throughput | p50 Latency | Peak VRAM |
|---|---|---|---|
| 1 | 4.5 req/s | 0.22s | 0.85 GB |
| 10 | 1.9 req/s | 5.2s | 1.15 GB |
| 50 | 2.0 req/s | 25s | 2.35 GB |
| 100 | 2.0 req/s | 50s | 3.83 GB |

### Key Takeaways

**Single model = ~4.6 req/s** — threads serialize on the GPU (Python GIL + CUDA stream). Throughput stays flat at ~2 req/s with concurrent threads because they queue up.

### Getting to 500 Concurrency

The model uses only **0.79 GB VRAM**. The A100 has 80 GB. Two strategies:

1. **Multi-process model replicas** — Run ~20-40 independent model processes, each on its own CUDA stream. At 4.6 req/s per replica:
   - **20 replicas = ~90 req/s** (using ~16 GB VRAM)
   - **40 replicas = ~180 req/s** (using ~32 GB VRAM)

2. **True batched inference** — Modify `model.sample()` to accept batch>1 inputs. This would be the biggest win since the model's forward pass could process multiple requests simultaneously. Could potentially reach **50-100+ req/s per replica**.

3. **Multi-GPU** — With 2-4 A100s + replicas + batching, **500 concurrency with <1s latency is achievable**.

**Bottom line**: On a single A100 without code changes, you get ~5 req/s. With model replicas you can serve 500 concurrent with ~3-5s latency. With batched inference code changes, 500 concurrent with <1s latency becomes realistic on 1-2 A100s.

Want me to test multi-process replicas or look into batching the model's forward pass?


i want in a single gpu the mdoel mentine d150x speed


Good catch — we got 18x but the claim is 150x. The warning at the top is a big clue:

> `Failed import k2... Swoosh functions will fallback to PyTorch implementation, leading to slower speed and higher memory consumption`

Let me investigate what's needed.


zipvoice/utils/checkpoint.py:273:    pattern = re.compile(r"checkpoint-([0-9]+).pt")
zipvoice/utils/diagnostics.py:605:                torch.float16,
zipvoice/utils/diagnostics.py:615:                        torch.float16,
zipvoice/utils/diagnostics.py:627:                torch.float16,
zipvoice/utils/diagnostics.py:637:                        torch.float16,
zipvoice/utils/tensorrt.py:88:        x = x.to(torch.float16)
zipvoice/utils/tensorrt.py:89:        t = t.to(torch.float16)
zipvoice/utils/tensorrt.py:90:        padding_mask = padding_mask.to(torch.float16)
zipvoice/utils/tensorrt.py:92:            guidance_scale = guidance_scale.to(torch.float16)
zipvoice/bin/train_zipvoice_distill.py:29:    --use-fp16 1 \
zipvoice/bin/train_zipvoice_distill.py:44:    --use-fp16 1 \
zipvoice/bin/train_zipvoice_distill.py:271:        "--use-fp16",
zipvoice/bin/train_zipvoice_distill.py:274:        help="Whether to use half precision training.",
zipvoice/bin/train_zipvoice_distill.py:630:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_distill.py:693:        if params.batch_idx_train % 100 == 0 and params.use_fp16:
zipvoice/bin/train_zipvoice_distill.py:716:            cur_grad_scale = scaler._scale.item() if params.use_fp16 else 1.0
zipvoice/bin/train_zipvoice_distill.py:724:                + (f"grad_scale: {scaler._scale.item()}" if params.use_fp16 else "")
zipvoice/bin/train_zipvoice_distill.py:735:                if params.use_fp16:
zipvoice/bin/train_zipvoice_distill.py:821:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_distill.py:991:    scaler = create_grad_scaler(enabled=params.use_fp16)
zipvoice/bin/train_zipvoice_dialog_stereo.py:25:    --use-fp16 1 \
zipvoice/bin/train_zipvoice_dialog_stereo.py:250:        "--use-fp16",
zipvoice/bin/train_zipvoice_dialog_stereo.py:253:        help="Whether to use half precision training.",
zipvoice/bin/train_zipvoice_dialog_stereo.py:496:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_dialog_stereo.py:557:        if params.batch_idx_train % 100 == 0 and params.use_fp16:
zipvoice/bin/train_zipvoice_dialog_stereo.py:580:            cur_grad_scale = scaler._scale.item() if params.use_fp16 else 1.0
zipvoice/bin/train_zipvoice_dialog_stereo.py:588:                + (f"grad_scale: {scaler._scale.item()}" if params.use_fp16 else "")
zipvoice/bin/train_zipvoice_dialog_stereo.py:599:                if params.use_fp16:
zipvoice/bin/train_zipvoice_dialog_stereo.py:683:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_dialog_stereo.py:818:    scaler = create_grad_scaler(enabled=params.use_fp16)
zipvoice/bin/train_zipvoice_dialog.py:25:    --use-fp16 1 \
zipvoice/bin/train_zipvoice_dialog.py:252:        "--use-fp16",
zipvoice/bin/train_zipvoice_dialog.py:255:        help="Whether to use half precision training.",
zipvoice/bin/train_zipvoice_dialog.py:494:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_dialog.py:554:        if params.batch_idx_train % 100 == 0 and params.use_fp16:
zipvoice/bin/train_zipvoice_dialog.py:577:            cur_grad_scale = scaler._scale.item() if params.use_fp16 else 1.0
zipvoice/bin/train_zipvoice_dialog.py:585:                + (f"grad_scale: {scaler._scale.item()}" if params.use_fp16 else "")
zipvoice/bin/train_zipvoice_dialog.py:596:                if params.use_fp16:
zipvoice/bin/train_zipvoice_dialog.py:679:            with torch_autocast(dtype=torch.float16, enabled=params.use_fp16):
zipvoice/bin/train_zipvoice_dialog.py:815:    scaler = create_grad_scaler(enabled=params.use_fp16)

[Showing results with pagination = limit: 40]


Try installing k2


xcodec2 0.1.5 requires torch==2.5.0, but you have torch 1.13.1 which is incompatible.
xcodec2 0.1.5 requires torchaudio==2.5.0, but you have torchaudio 2.9.1+cu128 which is incompatible.
xformers 0.0.35 requires torch>=2.10, but you have torch 1.13.1 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed k2-1.23.4 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 torch-1.13.1


k2 installed but it pulled torch 1.13.1 which will conflict. Let me fix that and also check for TensorRT and torch.compile options.


# k2 pulled wrong torch version - reinstall correct torch
pip install torch==2.9.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu128 2>&1 | tail -3

Reinstall correct torch version


xformers 0.0.35 requires torch>=2.10, but you have torch 2.9.1+cu128 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed torch-2.9.1+cu128


     1→# Copyright         2025  Nvidia Corp.        (authors: Yuekai Zhang)
     2→#
     3→# See ../../../../LICENSE for clarification regarding multiple authors
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→
    17→"""
    18→This script provides utility functions for working with TensorRT in ZipVoice.
    19→"""
    20→
    21→import logging
    22→import os
    23→import queue
    24→from typing import Any, Tuple, Optional
    25→
    26→import torch
    27→import torch.nn as nn
    28→
    29→
    30→class TrtContextWrapper:
    31→    """A wrapper class for managing TensorRT execution contexts."""
    32→
    33→    def __init__(
    34→        self, trt_engine: Any, trt_concurrent: int = 1, device: str = "cuda:0"
    35→    ):
    36→        """
    37→        Initializes the TrtContextWrapper.
    38→
    39→        Args:
    40→            trt_engine (Any): The TensorRT engine.
    41→            trt_concurrent (int, optional): The number of concurrent contexts. Defaults to 1.
    42→            device (str, optional): The device to use. Defaults to 'cuda:0'.
    43→        """
    44→        self.trt_context_pool = queue.Queue(maxsize=trt_concurrent)
    45→        self.trt_engine = trt_engine
    46→        self.device = device
    47→        for _ in range(trt_concurrent):
    48→            trt_context = trt_engine.create_execution_context()
    49→            trt_stream = torch.cuda.stream(torch.cuda.Stream(torch.device(device)))
    50→            assert trt_context is not None, 'failed to create trt context, maybe not enough CUDA memory, try reduce current trt concurrent {}'.format(trt_concurrent)
    51→            self.trt_context_pool.put([trt_context, trt_stream])
    52→        assert self.trt_context_pool.empty() is False, 'no avaialbe estimator context'
    53→        self.feat_dim = 100
    54→
    55→    def acquire_estimator(self) -> Tuple[list, Any]:
    56→        """Acquires a TensorRT context from the pool."""
    57→        return self.trt_context_pool.get(), self.trt_engine
    58→
    59→    def release_estimator(self, context: Any, stream: Any):
    60→        """
    61→        Releases a TensorRT context back to the pool.
    62→
    63→        Args:
    64→            context (Any): The TensorRT context.
    65→            stream (Any): The CUDA stream.
    66→        """
    67→        self.trt_context_pool.put([context, stream])
    68→
    69→    def __call__(
    70→        self,
    71→        x: torch.Tensor,
    72→        t: torch.Tensor,
    73→        padding_mask: torch.Tensor,
    74→        guidance_scale: Optional[torch.Tensor] = None,
    75→    ) -> torch.Tensor:
    76→        """
    77→        Executes the TensorRT engine.
    78→
    79→        Args:
    80→            x (torch.Tensor): The input tensor.
    81→            t (torch.Tensor): The time tensor.
    82→            padding_mask (torch.Tensor): The padding mask tensor.
    83→            guidance_scale (torch.Tensor): The guidance scale tensor.
    84→
    85→        Returns:
    86→            torch.Tensor: The output tensor.
    87→        """
    88→        x = x.to(torch.float16)
    89→        t = t.to(torch.float16)
    90→        padding_mask = padding_mask.to(torch.float16)
    91→        if guidance_scale is not None:
    92→            guidance_scale = guidance_scale.to(torch.float16)
    93→        [estimator, stream], trt_engine = self.acquire_estimator()
    94→        # NOTE need to synchronize when switching stream
    95→        torch.cuda.current_stream().synchronize()
    96→        batch_size = x.size(0)
    97→        seq_len = x.size(1)
    98→
    99→        # Create output tensor with shape (N, T, 100)
   100→        output = torch.empty(batch_size, seq_len, self.feat_dim, dtype=x.dtype, device=x.device)
   101→
   102→        with stream:
   103→            estimator.set_input_shape('x', (batch_size, x.size(1), x.size(2)))
   104→            estimator.set_input_shape('t', (batch_size,))
   105→            estimator.set_input_shape('padding_mask', (batch_size, padding_mask.size(1)))
   106→            if guidance_scale is not None:
   107→                estimator.set_input_shape('guidance_scale', (batch_size,))
   108→
   109→            # Set input tensor addresses
   110→            input_data_ptrs = [x.contiguous().data_ptr(), t.contiguous().data_ptr(), padding_mask.contiguous().data_ptr()]
   111→            if guidance_scale is not None:
   112→                input_data_ptrs.append(guidance_scale.contiguous().data_ptr())
   113→            for i, j in enumerate(input_data_ptrs):
   114→                estimator.set_tensor_address(trt_engine.get_tensor_name(i), j)
   115→
   116→            # Set output tensor address
   117→            # The output tensor name should be the last tensor name in the engine
   118→            num_tensors = trt_engine.num_io_tensors
   119→            output_tensor_name = trt_engine.get_tensor_name(num_tensors - 1)  # Last tensor is output
   120→            estimator.set_tensor_address(output_tensor_name, output.contiguous().data_ptr())
   121→
   122→            # run trt engine
   123→            assert estimator.execute_async_v3(torch.cuda.current_stream().cuda_stream) is True
   124→            torch.cuda.current_stream().synchronize()
   125→        self.release_estimator(estimator, stream)
   126→        return output.to(torch.float32)
   127→
   128→def load_trt(model: nn.Module, trt_model: str, trt_concurrent: int = 1):
   129→    """
   130→    Loads a TensorRT engine and replaces the model's fm_decoder with a TrtContextWrapper.
   131→
   132→    Args:
   133→        model (nn.Module): The model to modify.
   134→        trt_model (str): The path to the TensorRT engine file.
   135→        trt_concurrent (int, optional): The number of concurrent contexts. Defaults to 1.
   136→    """
   137→    assert os.path.exists(trt_model), f"Please export trt model first."
   138→    import tensorrt as trt
   139→    with open(trt_model, 'rb') as f:
   140→        estimator_engine = trt.Runtime(trt.Logger(trt.Logger.INFO)).deserialize_cuda_engine(f.read())
   141→    assert estimator_engine is not None, 'failed to load trt {}'.format(trt_model)
   142→    del model.fm_decoder
   143→    model.fm_decoder = TrtContextWrapper(estimator_engine, trt_concurrent=trt_concurrent, device='cuda')
   144→

# Copyright         2025  Nvidia Corp.        (authors: Yuekai Zhang)
#
# See ../../../../LICENSE for clarification regarding multiple authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""
This script provides utility functions for working with TensorRT in ZipVoice.
"""

import logging
import os
import queue
from typing import Any, Tuple, Optional

import torch
import torch.nn as nn

class TrtContextWrapper:
    """A wrapper class for managing TensorRT execution contexts."""

    def __init__(
        self, trt_engine: Any, trt_concurrent: int = 1, device: str = "cuda:0"
    ):
        """
        Initializes the TrtContextWrapper.

        Args:
            trt_engine (Any): The TensorRT engine.
            trt_concurrent (int, optional): The number of concurrent contexts. Defaults to 1.
            device (str, optional): The device to use. Defaults to 'cuda:0'.
        """
        self.trt_context_pool = queue.Queue(maxsize=trt_concurrent)
        self.trt_engine = trt_engine
        self.device = device
        for _ in range(trt_concurrent):
            trt_context = trt_engine.create_execution_context()
            trt_stream = torch.cuda.stream(torch.cuda.Stream(torch.device(device)))
            assert trt_context is not None, 'failed to create trt context, maybe not enough CUDA memory, try reduce current trt concurrent {}'.format(trt_concurrent)
            self.trt_context_pool.put([trt_context, trt_stream])
        assert self.trt_context_pool.empty() is False, 'no avaialbe estimator context'
        self.feat_dim = 100

    def acquire_estimator(self) -> Tuple[list, Any]:
        """Acquires a TensorRT context from the pool."""
        return self.trt_context_pool.get(), self.trt_engine

    def release_estimator(self, context: Any, stream: Any):
        """
        Releases a TensorRT context back to the pool.

        Args:
            context (Any): The TensorRT context.
            stream (Any): The CUDA stream.
        """
        self.trt_context_pool.put([context, stream])

    def __call__(
        self,
        x: torch.Tensor,
        t: torch.Tensor,
        padding_mask: torch.Tensor,
        guidance_scale: Optional[torch.Tensor] = None,
    ) -> torch.Tensor:
        """
        Executes the TensorRT engine.

        Args:
            x (torch.Tensor): The input tensor.
            t (torch.Tensor): The time tensor.
            padding_mask (torch.Tensor): The padding mask tensor.
            guidance_scale (torch.Tensor): The guidance scale tensor.

        Returns:
            torch.Tensor: The output tensor.
        """
        x = x.to(torch.float16)
        t = t.to(torch.float16)
        padding_mask = padding_mask.to(torch.float16)
        if guidance_scale is not None:
            guidance_scale = guidance_scale.to(torch.float16)
        [estimator, stream], trt_engine = self.acquire_estimator()
        # NOTE need to synchronize when switching stream
        torch.cuda.current_stream().synchronize()
        batch_size = x.size(0)
        seq_len = x.size(1)

        # Create output tensor with shape (N, T, 100)
        output = torch.empty(batch_size, seq_len, self.feat_dim, dtype=x.dtype, device=x.device)

        with stream:
            estimator.set_input_shape('x', (batch_size, x.size(1), x.size(2)))
            estimator.set_input_shape('t', (batch_size,))
            estimator.set_input_shape('padding_mask', (batch_size, padding_mask.size(1)))
            if guidance_scale is not None:
                estimator.set_input_shape('guidance_scale', (batch_size,))

            # Set input tensor addresses
            input_data_ptrs = [x.contiguous().data_ptr(), t.contiguous().data_ptr(), padding_mask.contiguous().data_ptr()]
            if guidance_scale is not None:
                input_data_ptrs.append(guidance_scale.contiguous().data_ptr())
            for i, j in enumerate(input_data_ptrs):
                estimator.set_tensor_address(trt_engine.get_tensor_name(i), j)

            # Set output tensor address
            # The output tensor name should be the last tensor name in the engine
            num_tensors = trt_engine.num_io_tensors
            output_tensor_name = trt_engine.get_tensor_name(num_tensors - 1)  # Last tensor is output
            estimator.set_tensor_address(output_tensor_name, output.contiguous().data_ptr())

            # run trt engine
            assert estimator.execute_async_v3(torch.cuda.current_stream().cuda_stream) is True
            torch.cuda.current_stream().synchronize()
        self.release_estimator(estimator, stream)
        return output.to(torch.float32)

def load_trt(model: nn.Module, trt_model: str, trt_concurrent: int = 1):
    """
    Loads a TensorRT engine and replaces the model's fm_decoder with a TrtContextWrapper.

    Args:
        model (nn.Module): The model to modify.
        trt_model (str): The path to the TensorRT engine file.
        trt_concurrent (int, optional): The number of concurrent contexts. Defaults to 1.
    """
    assert os.path.exists(trt_model), f"Please export trt model first."
    import tensorrt as trt
    with open(trt_model, 'rb') as f:
        estimator_engine = trt.Runtime(trt.Logger(trt.Logger.INFO)).deserialize_cuda_engine(f.read())
    assert estimator_engine is not None, 'failed to load trt {}'.format(trt_model)
    del model.fm_decoder
    model.fm_decoder = TrtContextWrapper(estimator_engine, trt_concurrent=trt_concurrent, device='cuda')


Now I see it — the 150x claim is likely with **TensorRT + fp16**. Let me also check the model's sample/forward to see where time is spent and try fp16 + torch.compile.


zipvoice/models/zipvoice_distill.py:35:            "fm_decoder_downsampling_factor",
zipvoice/models/zipvoice_distill.py:36:            "fm_decoder_num_layers",
zipvoice/models/zipvoice_distill.py:37:            "fm_decoder_cnn_module_kernel",
zipvoice/models/zipvoice_distill.py:38:            "fm_decoder_dim",
zipvoice/models/zipvoice_distill.py:39:            "fm_decoder_feedforward_dim",
zipvoice/models/zipvoice_distill.py:40:            "fm_decoder_num_heads",
zipvoice/models/zipvoice_distill.py:52:        self.fm_decoder = TTSZipformer(
zipvoice/models/zipvoice_distill.py:55:            downsampling_factor=kwargs["fm_decoder_downsampling_factor"],
zipvoice/models/zipvoice_distill.py:56:            num_encoder_layers=kwargs["fm_decoder_num_layers"],
zipvoice/models/zipvoice_distill.py:57:            cnn_module_kernel=kwargs["fm_decoder_cnn_module_kernel"],
zipvoice/models/zipvoice_distill.py:58:            encoder_dim=kwargs["fm_decoder_dim"],
zipvoice/models/zipvoice_distill.py:59:            feedforward_dim=kwargs["fm_decoder_feedforward_dim"],
zipvoice/models/zipvoice_distill.py:60:            num_heads=kwargs["fm_decoder_num_heads"],
zipvoice/models/zipvoice_distill.py:69:        self.solver = DistillEulerSolver(self, func_name="forward_fm_decoder")
zipvoice/models/zipvoice_distill.py:71:    def forward(
zipvoice/models/zipvoice.py:40:        fm_decoder_downsampling_factor: List[int] = [1, 2, 4, 2, 1],
zipvoice/models/zipvoice.py:41:        fm_decoder_num_layers: List[int] = [2, 2, 4, 4, 4],
zipvoice/models/zipvoice.py:42:        fm_decoder_cnn_module_kernel: List[int] = [31, 15, 7, 15, 31],
zipvoice/models/zipvoice.py:43:        fm_decoder_feedforward_dim: int = 1536,
zipvoice/models/zipvoice.py:44:        fm_decoder_num_heads: int = 4,
zipvoice/models/zipvoice.py:45:        fm_decoder_dim: int = 512,
zipvoice/models/zipvoice.py:65:            fm_decoder_downsampling_factor: List of downsampling factors for each layer
zipvoice/models/zipvoice.py:67:            fm_decoder_num_layers: List of the number of layers for each block in the
zipvoice/models/zipvoice.py:69:            fm_decoder_cnn_module_kernel: List of kernel sizes for CNN modules in the
zipvoice/models/zipvoice.py:71:            fm_decoder_feedforward_dim: Dimension of the feedforward network in the
zipvoice/models/zipvoice.py:73:            fm_decoder_num_heads: Number of attention heads in the flow-matching
zipvoice/models/zipvoice.py:75:            fm_decoder_dim: Hidden dimension of the flow-matching decoder.
zipvoice/models/zipvoice.py:95:        self.fm_decoder = TTSZipformer(
zipvoice/models/zipvoice.py:98:            downsampling_factor=fm_decoder_downsampling_factor,
zipvoice/models/zipvoice.py:99:            num_encoder_layers=fm_decoder_num_layers,
zipvoice/models/zipvoice.py:100:            cnn_module_kernel=fm_decoder_cnn_module_kernel,
zipvoice/models/zipvoice.py:101:            encoder_dim=fm_decoder_dim,
zipvoice/models/zipvoice.py:102:            feedforward_dim=fm_decoder_feedforward_dim,
zipvoice/models/zipvoice.py:103:            num_heads=fm_decoder_num_heads,
zipvoice/models/zipvoice.py:133:        self.solver = EulerSolver(self, func_name="forward_fm_decoder")
zipvoice/models/zipvoice.py:135:    def forward_fm_decoder(
zipvoice/models/zipvoice.py:180:            vt = self.fm_decoder(
zipvoice/models/zipvoice.py:184:            vt = self.fm_decoder(x=xt, t=t, padding_mask=padding_mask)
zipvoice/models/zipvoice.py:187:    def forward_text_embed(
zipvoice/models/zipvoice.py:214:    def forward_text_condition(
zipvoice/models/zipvoice.py:253:    def forward_text_train(
zipvoice/models/zipvoice.py:270:    def forward_text_inference_gt_duration(
zipvoice/models/zipvoice.py:290:    def forward_text_inference_ratio_duration(
zipvoice/models/zipvoice.py:332:    def forward(
zipvoice/models/zipvoice.py:375:        vt = self.forward_fm_decoder(
zipvoice/models/zipvoice.py:388:    def sample(
zipvoice/models/zipvoice.py:488:    def sample_intermediate(
zipvoice/models/zipvoice_dialog.py:33:        fm_decoder_downsampling_factor: List[int] = [1, 2, 4, 2, 1],
zipvoice/models/zipvoice_dialog.py:34:        fm_decoder_num_layers: List[int] = [2, 2, 4, 4, 4],
zipvoice/models/zipvoice_dialog.py:35:        fm_decoder_cnn_module_kernel: List[int] = [31, 15, 7, 15, 31],
zipvoice/models/zipvoice_dialog.py:36:        fm_decoder_feedforward_dim: int = 1536,
zipvoice/models/zipvoice_dialog.py:37:        fm_decoder_num_heads: int = 4,
zipvoice/models/zipvoice_dialog.py:38:        fm_decoder_dim: int = 512,
zipvoice/models/zipvoice_dialog.py:60:            fm_decoder_downsampling_factor: List of downsampling factors for each layer
zipvoice/models/zipvoice_dialog.py:62:            fm_decoder_num_layers: List of the number of layers for each block in the
zipvoice/models/zipvoice_dialog.py:64:            fm_decoder_cnn_module_kernel: List of kernel sizes for CNN modules in the
zipvoice/models/zipvoice_dialog.py:66:            fm_decoder_feedforward_dim: Dimension of the feedforward network in the
zipvoice/models/zipvoice_dialog.py:68:            fm_decoder_num_heads: Number of attention heads in the flow-matching
zipvoice/models/zipvoice_dialog.py:70:            fm_decoder_dim: Hidden dimension of the flow-matching decoder.
zipvoice/models/zipvoice_dialog.py:91:            fm_decoder_downsampling_factor=fm_decoder_downsampling_factor,
zipvoice/models/zipvoice_dialog.py:92:            fm_decoder_num_layers=fm_decoder_num_layers,
zipvoice/models/zipvoice_dialog.py:93:            fm_decoder_cnn_module_kernel=fm_decoder_cnn_module_kernel,
zipvoice/models/zipvoice_dialog.py:94:            fm_decoder_feedforward_dim=fm_decoder_feedforward_dim,
zipvoice/models/zipvoice_dialog.py:95:            fm_decoder_num_heads=fm_decoder_num_heads,
zipvoice/models/zipvoice_dialog.py:96:            fm_decoder_dim=fm_decoder_dim,
zipvoice/models/zipvoice_dialog.py:127:    def forward_text_embed(
zipvoice/models/zipvoice_dialog.py:161:    def forward(
zipvoice/models/zipvoice_dialog.py:204:        vt = self.forward_fm_decoder(
zipvoice/models/zipvoice_dialog.py:224:            "fm_decoder_downsampling_factor",
zipvoice/models/zipvoice_dialog.py:225:            "fm_decoder_num_layers",
zipvoice/models/zipvoice_dialog.py:226:            "fm_decoder_cnn_module_kernel",
zipvoice/models/zipvoice_dialog.py:227:            "fm_decoder_dim",
zipvoice/models/zipvoice_dialog.py:228:            "fm_decoder_feedforward_dim",
zipvoice/models/zipvoice_dialog.py:229:            "fm_decoder_num_heads",
zipvoice/models/zipvoice_dialog.py:241:        self.fm_decoder = TTSZipformerTwoStream(
zipvoice/models/zipvoice_dialog.py:244:            downsampling_factor=kwargs["fm_decoder_downsampling_factor"],
zipvoice/models/zipvoice_dialog.py:245:            num_encoder_layers=kwargs["fm_decoder_num_layers"],
zipvoice/models/zipvoice_dialog.py:246:            cnn_module_kernel=kwargs["fm_decoder_cnn_module_kernel"],
zipvoice/models/zipvoice_dialog.py:247:            encoder_dim=kwargs["fm_decoder_dim"],
zipvoice/models/zipvoice_dialog.py:248:            feedforward_dim=kwargs["fm_decoder_feedforward_dim"],
zipvoice/models/zipvoice_dialog.py:249:            num_heads=kwargs["fm_decoder_num_heads"],
zipvoice/models/zipvoice_dialog.py:258:    def forward(
zipvoice/models/zipvoice_dialog.py:303:        vt = self.forward_fm_decoder(
zipvoice/models/modules/zipformer.py:249:    def forward(
zipvoice/models/modules/zipformer.py:496:    def forward(
zipvoice/models/modules/zipformer.py:709:    def forward(
zipvoice/models/modules/zipformer.py:805:    def forward(self, src_orig: Tensor, src: Tensor):
zipvoice/models/modules/zipformer.py:830:    def forward(
zipvoice/models/modules/zipformer.py:894:    def forward(self, src: Tensor) -> Tensor:
zipvoice/models/modules/zipformer.py:932:    def forward(self, src: Tensor) -> Tensor:
zipvoice/models/modules/zipformer.py:1041:    def forward(self, x: Tensor, left_context_len: int = 0) -> Tensor:
zipvoice/models/modules/zipformer.py:1156:    def forward(
zipvoice/models/modules/zipformer.py:1366:    def forward(
zipvoice/models/modules/zipformer.py:1440:    def forward(self, x: Tensor):
zipvoice/models/modules/zipformer.py:1506:    def forward(
zipvoice/models/modules/zipformer.py:1645:    def forward(
zipvoice/models/modules/scaling.py:321:    def forward(ctx, x: Tensor, dim: int):
zipvoice/models/modules/scaling.py:359:    def forward(
zipvoice/models/modules/scaling.py:458:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:514:    def forward(
zipvoice/models/modules/scaling.py:661:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:807:    def forward(ctx, x: Tensor, module: nn.Module) -> Tensor:
zipvoice/models/modules/scaling.py:895:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:921:    def forward(ctx, x: Tensor, y: Tensor, name: str):
zipvoice/models/modules/scaling.py:944:    def forward(ctx, x: Tensor, min: float, max: float):
zipvoice/models/modules/scaling.py:993:    def forward(self, x):
zipvoice/models/modules/scaling.py:1004:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1013:    def forward(ctx, x, y, alpha):
zipvoice/models/modules/scaling.py:1036:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1054:    def forward(ctx, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1104:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1119:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1133:    def forward(ctx, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1180:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1195:    def forward(self, x: Tensor) -> Tensor:
zipvoice/models/modules/scaling.py:1226:    def forward(
zipvoice/models/modules/scaling.py:1351:    def forward(self, x: Tensor):
zipvoice/models/modules/zipformer_two_stream.py:219:    def forward(
zipvoice/models/modules/solver.py:33:        func_name: str = "forward_fm_decoder",
zipvoice/models/modules/solver.py:40:    def forward(
zipvoice/models/modules/solver.py:123:        func_name: str = "forward_fm_decoder",
zipvoice/models/modules/solver.py:127:    def forward(
zipvoice/models/modules/solver.py:172:        func_name: str = "forward_fm_decoder",
zipvoice/models/modules/solver.py:181:    def sample(
zipvoice/models/modules/solver.py:243:        func_name: str = "forward_fm_decoder",


Now let me run a proper benchmark with fp16 autocast and torch.compile: