type: user

role: user

content: hi

permissionMode: bypassPermissions

userType: external

entrypoint: cli

gitBranch: HEAD


type: message

role: assistant

type: text

text: Hi! How can I help you today?

service_tier: standard

speed: standard

type: assistant


content: what the fuck are u dng ..


text: Not much — just waiting for you to tell me what to work on! What do you need?


content: how are u


text: Doing good, thanks! What can I help you with?


content: i want u to go all in anc heck all about this model https://huggingface.co/YatharthS/LuxTTS


type: thinking

thinking: The user wants me to research a model on HuggingFace. Let me fetch the page to learn about it.


ToolSearch

query: select:WebFetch


tool_name: WebFetch

query: select:WebFetch


thinking: Let me fetch the HuggingFace page for this model and also search for any related info.


WebFetch

prompt: Extract ALL information about this model: architecture, parameters, training details, features, usage instructions, code examples, dependencies, model card content, any linked papers or repos, license, performance claims, supported languages, audio quality details, inference code, and anything else available. Be extremely thorough.


prompt: List ALL files in this repository with their sizes. Include every file and folder visible.


content: There's an issue with the selected model (claude-haiku-4-5@20251001). It may not exist or you may not have access to it. Run /model to pick a different model.

codeText: OK


result: There's an issue with the selected model (claude-haiku-4-5@20251001). It may not exist or you may not have access to it. Run /model to pick a different model.


thinking: WebFetch failed due to model issue. Let me try using an Agent with WebFetch/WebSearch instead.


text: Let me try a different approach.


Agent

description: Research LuxTTS model thoroughly

prompt:
Research the model at https://huggingface.co/YatharthS/LuxTTS thoroughly.

Use WebFetch to grab these URLs and extract ALL information:
1. https://huggingface.co/YatharthS/LuxTTS — main model card (architecture, params, training, features, usage, code examples, license, etc.)
2. https://huggingface.co/YatharthS/LuxTTS/tree/main — list all files and sizes
3. https://huggingface.co/YatharthS/LuxTTS/blob/main/README.md — full README content

Also use WebSearch to find any related papers, blog posts, GitHub repos, or discussions about LuxTTS.

Be extremely thorough — extract every detail: architecture type, parameter count, training data, supported languages, audio quality, inference code, dependencies, any linked papers or repos, license, performance benchmarks, and anything else available.


text:
Now I have comprehensive information. Here is the full research report.

---

## LuxTTS -- Complete Research Report

### Overview

**LuxTTS** is a lightweight, high-speed text-to-speech model designed for zero-shot voice cloning. It is built on top of **ZipVoice** (by Xiaomi/k2-fsa), distilled down to 4 inference steps, and paired with a custom 48kHz vocoder for high-clarity output. It was created by **Yatharth Sharma** (GitHub: ysharma3501, email: yatharthsharma350@gmail.com).

- **HuggingFace**: https://huggingface.co/YatharthS/LuxTTS
- **GitHub**: https://github.com/ysharma3501/LuxTTS (3,405 stars, 423 forks as of today)
- **Colab**: https://colab.research.google.com/drive/1cDaxtbSDLRmu6tRV_781Of_GSjHSo1Cu
- **HF Space**: https://huggingface.co/spaces/YatharthS/LuxTTS
- **License**: Apache-2.0
- **Downloads**: 6,811 | **Likes**: 182
- **Created**: 2026-01-22 | **Last Updated**: 2026-01-23
- **Language**: English only (`en`)

---

### Architecture

LuxTTS is a **flow-matching TTS** model using the **Zipformer** backbone (from k2-fsa/icefall). Key architectural components:

1. **Text Encoder** -- A Zipformer-based encoder
   - `text_encoder_dim`: 192
   - `text_encoder_num_layers`: 4
   - `text_encoder_feedforward_dim`: 512
   - `text_encoder_num_heads`: 4
   - `text_encoder_cnn_module_kernel`: 9
   - `text_embed_dim`: 192

2. **Flow Matching (FM) Decoder** -- A larger Zipformer with time embeddings and guidance scale embeddings, using a **multi-resolution U-Net-like downsampling/upsampling structure**:
   - `fm_decoder_dim`: 512
   - `fm_decoder_feedforward_dim`: 1536
   - `fm_decoder_num_heads`: 4
   - `fm_decoder_num_layers`: [2, 2, 4, 4, 4] (5 stages, 16 layers total)
   - `fm_decoder_downsampling_factor`: [1, 2, 4, 2, 1] (hourglass structure)
   - `fm_decoder_cnn_module_kernel`: [31, 15, 7, 15, 31]
   - `query_head_dim`: 32, `value_head_dim`: 12, `pos_head_dim`: 4, `pos_dim`: 48
   - `time_embed_dim`: 192
   - `feat_dim`: 100 (mel spectrogram features)

3. **Distillation**: Inherits from `ZipVoice` but replaces the solver with `DistillEulerSolver`, enabling generation in only **4 steps** (vs. the original 16+ steps). Uses an anchor-based ODE update scheme.

4. **Vocoder** -- A modified **Vocos** vocoder (from gemelo-ai) with a custom upsampler (from **LinaCodec**) to output 48kHz audio:
   - Backbone: VocosBackbone, dim=512, intermediate_dim=1536, 8 layers
   - Head: ISTFTHead (n_fft=1024, hop_length=256)
   - Upsampler: UpSamplerBlock from LinaCodec with upsample_factors [2, 1]
   - Mel features: 100 mels, sample_rate=24000 internally, upsampled to 48kHz output

5. **Tokenizer**: Uses `EmiliaTokenizer` with **360 phoneme-level tokens** (IPA symbols including click consonants, vowels, diacritics, etc.) processed via `piper_phonemize`.

6. **Prompt Encoder**: Uses **Whisper** (whisper-base for GPU, whisper-tiny for CPU) to transcribe the reference audio, then extracts mel features using `VocosFbank` at 24kHz.

---

### Parameter Count & File Sizes

Total storage: **~1.19 GB**

| File | Size | Description |
|---|---|---|
| `model.pt` | 491 MB | Full PyTorch model (float32) |
| `fm_decoder.onnx` | 478 MB | FM decoder ONNX (float32) |
| `fm_decoder_int8.onnx` | 125 MB | FM decoder ONNX (int8 quantized) |
| `text_encoder.onnx` | 17.6 MB | Text encoder ONNX (float32) |
| `text_encoder_int8.onnx` | 5.6 MB | Text encoder ONNX (int8 quantized) |
| `vocoder/vocos.bin` | 64 MB | Vocos vocoder weights |
| `vocoder/config.yaml` | 752 B | Vocoder config |
| `config.json` | 697 B | Model config |
| `tokens.txt` | 2.6 KB | Tokenizer vocabulary |

From the file sizes, the FM decoder dominates (~478MB float32 ONNX). The full model.pt at 491MB in float32 suggests roughly **~120-130M parameters** total (491MB / 4 bytes per param). The text encoder is quite small (~4.4M params from the 17.6MB ONNX file). VRAM usage is stated to be under **1GB**.

---

### Key Features

- **Voice Cloning**: Zero-shot, SOTA quality claimed on par with models 10x larger
- **48kHz Output**: Clear, high-fidelity speech (vs. 24kHz from most TTS models)
- **150x Realtime Speed**: On a single GPU; also faster-than-realtime on CPU
- **Under 1GB VRAM**: Fits on any local GPU
- **Multi-device**: Supports CUDA, MPS (Apple Silicon), and CPU (via ONNX)
- **Distilled to 4 Steps**: From ZipVoice's original multi-step flow matching

---

### Inference API & Parameters

```python
from zipvoice.luxvoice import LuxTTS
lux_tts = LuxTTS('YatharthS/LuxTTS', device='cuda')  # or 'cpu', 'mps'

encoded_prompt = lux_tts.encode_prompt('reference.wav', duration=5, rms=0.01)
final_wav = lux_tts.generate_speech(
    text="Hello world!",
    encode_dict=encoded_prompt,
    num_steps=4,         # Higher = better quality but slower (3-4 best for efficiency)
    guidance_scale=3.0,  # Classifier-free guidance scale
    t_shift=0.5,         # Sampling temperature-like param (higher = better quality but worse WER)
    speed=1.0,           # Speech rate (lower = slower)
    return_smooth=False   # True = smoother but less clean (uses 24kHz path)
)
```

**Tips from the authors**:
- Minimum 3-second reference audio for voice cloning
- Use `return_smooth=True` if you hear metallic sounds
- Lower `t_shift` reduces pronunciation errors but degrades quality

---

### Dependencies

Core: `torch`, `torchaudio`, `numpy`, `onnxruntime`, `librosa`, `vocos`, `lhotse`, `huggingface_hub`, `safetensors`, `transformers<=4.57.6`, `pydub`

Custom: `linacodec` (from `https://github.com/ysharma3501/LinaCodec.git`) -- provides the 48kHz upsampler for Vocos

Tokenization: `piper_phonemize`, `pypinyin`, `jieba`, `inflect`, `cn2an`

Runtime: Whisper (openai/whisper-base or whisper-tiny) for automatic prompt transcription

---

### Upstream Lineage

- **ZipVoice** (k2-fsa/ZipVoice, 928 stars): "Fast and High-Quality Zero-Shot Text-to-Speech with Flow Matching" -- https://zipvoice.github.io/ -- Originally by Xiaomi (Wei Kang, Han Zhu). LuxTTS was forked from `johnwick123f/ZipVoice` which itself is a fork of `k2-fsa/ZipVoice`.
- **Vocos** (gemelo-ai/vocos): The base vocoder architecture
- **LinaCodec** (ysharma3501/LinaCodec): Custom codec providing the 48kHz upsampling block

---

### Codebase Structure (GitHub)

```
ysharma3501/LuxTTS/
  zipvoice/
    luxvoice.py              -- Main user-facing LuxTTS class
    modeling_utils.py         -- GPU inference, model loading
    onnx_modeling.py          -- CPU/ONNX inference path
    models/
      zipvoice.py             -- Base ZipVoice model (20KB)
      zipvoice_distill.py     -- Distilled variant used by LuxTTS
      zipvoice_dialog.py      -- Dialog variant (unused by LuxTTS)
      modules/
        zipformer.py          -- Core Zipformer architecture (63KB)
        scaling.py            -- Scaling utilities (58KB)
        solver.py             -- Euler/Distill solvers for flow matching
        zipformer_two_stream.py
    tokenizer/
      tokenizer.py            -- EmiliaTokenizer, EspeakTokenizer, etc.
      normalizer.py           -- Text normalization
    utils/                    -- Feature extraction, checkpointing, inference helpers
    bin/                      -- Training, inference, ONNX/TensorRT export scripts
  requirements.txt
  pyproject.toml
```

---

### Community Projects

- **LuxTTS-Gradio** (NidAll/LuxTTS-Gradio): Gradio web UI
- **OptiSpeech** (ycharfi09/OptiClone): Clean UI app
- **LuxTTS-Comfyui** (DragonDiffusionbyBoyo/BoyoLuxTTS-Comfyui): ComfyUI nodes
- **26 HuggingFace Spaces** duplicating/using the model

---

### Roadmap

- [x] Release model and code
- [x] HuggingFace Spaces demo
- [x] MPS support (contributor: @builtbybasit)
- [ ] Release LuxTTS v1.5
- [ ] Float16 inference code (expected ~2x speedup)

---

### Key Technical Observations

1. **The flow matching operates in mel spectrogram space** (100-dim mel features at 24kHz), not in a latent/token space. The vocoder then converts mels to 48kHz waveform.

2. **The distillation approach** uses `DistillEulerSolver` with an anchor-based ODE update: at each step it predicts both x_0 (noise) and x_1 (clean data), interpolates for the next step, and snaps to x_1 on the final step.

3. **The 48kHz output is achieved by upsampling** -- the model internally operates at 24kHz mel resolution, and the custom Vocos vocoder with LinaCodec's `UpSamplerBlock` (factors [2,1]) produces 48kHz waveform from the ISTFT head.

4. **No training data is disclosed** in the model card or README. The upstream ZipVoice likely trained on Emilia/LibriTTS (given the tokenizer names), but LuxTTS does not specify its training data.

5. **The codebase includes training scripts** (`train_zipvoice.py`, `train_zipvoice_distill.py`) suggesting the distillation can be reproduced, though no training instructions are provided in the README.

text:
agentId: a1811137bb0257102 (use SendMessage with to: 'a1811137bb0257102' to continue this agent)
<usage>total_tokens: 35666
tool_uses: 29
duration_ms: 149746</usage>

status: completed

prompt:
Research the model at https://huggingface.co/YatharthS/LuxTTS thoroughly.

Use WebFetch to grab these URLs and extract ALL information:
1. https://huggingface.co/YatharthS/LuxTTS — main model card (architecture, params, training, features, usage, code examples, license, etc.)
2. https://huggingface.co/YatharthS/LuxTTS/tree/main — list all files and sizes
3. https://huggingface.co/YatharthS/LuxTTS/blob/main/README.md — full README content

Also use WebSearch to find any related papers, blog posts, GitHub repos, or discussions about LuxTTS.

Be extremely thorough — extract every detail: architecture type, parameter count, training data, supported languages, audio quality, inference code, dependencies, any linked papers or repos, license, performance benchmarks, and anything else available.

agentType: general-purpose


thinking: Great, I got a comprehensive report. Let me summarize this for the user.


text:
Here's the full breakdown on **LuxTTS**:

---

## LuxTTS — Full Deep Dive

**Creator:** Yatharth Sharma (@ysharma3501)
**License:** Apache-2.0 | **Language:** English only
**GitHub:** 3,405 stars, 423 forks
**HF Downloads:** 6,811 | **Likes:** 182

---

### What It Is

A **lightweight, zero-shot voice cloning TTS** model built on top of **ZipVoice** (by Xiaomi/k2-fsa). It's been distilled down to **4 inference steps** and paired with a custom **48kHz vocoder** for high-fidelity output.

### Architecture

1. **Text Encoder** — Zipformer-based, dim=192, 4 layers (~4.4M params)
2. **Flow Matching Decoder** — Larger Zipformer with U-Net-like hourglass structure, dim=512, 16 layers across 5 stages with downsampling factors [1,2,4,2,1]. This is the heavy part.
3. **Vocoder** — Modified **Vocos** (gemelo-ai) + **LinaCodec** upsampler to go from 24kHz mel space → 48kHz waveform output
4. **Prompt Encoder** — Uses **Whisper** (base/tiny) to transcribe reference audio + mel feature extraction
5. **Tokenizer** — EmiliaTokenizer with 360 IPA phoneme tokens via `piper_phonemize`

**Total size:** ~491MB (float32), roughly **~120-130M parameters**

### Key Claims

- **150x realtime** on a single GPU
- **Under 1GB VRAM**
- **48kHz output** (most TTS models do 24kHz)
- Zero-shot voice cloning "on par with models 10x larger"
- Works on **CUDA, MPS (Apple Silicon), and CPU** (ONNX path)

### How It Works Technically

- Flow matching operates in **100-dim mel spectrogram space** at 24kHz — not in a latent/token space
- Uses `DistillEulerSolver` with anchor-based ODE updates — predicts both x_0 (noise) and x_1 (clean), interpolates, snaps to x_1 on final step
- The 48kHz is achieved via **upsampling** — the core model runs at 24kHz, then LinaCodec's UpSamplerBlock (factors [2,1]) in the vocoder produces 48kHz waveform

### Files on HF

| File | Size |
|---|---|
| `model.pt` | 491 MB |
| `fm_decoder.onnx` | 478 MB |
| `fm_decoder_int8.onnx` | 125 MB |
| `text_encoder.onnx` | 17.6 MB |
| `text_encoder_int8.onnx` | 5.6 MB |
| `vocoder/vocos.bin` | 64 MB |

### Usage

```python
from zipvoice.luxvoice import LuxTTS

lux_tts = LuxTTS('YatharthS/LuxTTS', device='cuda')
encoded_prompt = lux_tts.encode_prompt('reference.wav', duration=5, rms=0.01)
final_wav = lux_tts.generate_speech(
    text="Hello world!",
    encode_dict=encoded_prompt,
    num_steps=4,
    guidance_scale=3.0,
    t_shift=0.5,
    speed=1.0,
    return_smooth=False  # True if you hear metallic artifacts
)
```

### Lineage

- Forked from **ZipVoice** (k2-fsa) → "Fast and High-Quality Zero-Shot TTS with Flow Matching"
- Vocoder based on **Vocos** (gemelo-ai)
- 48kHz upsampler from author's own **LinaCodec**
- **No training data disclosed** — upstream ZipVoice likely used Emilia/LibriTTS

### What's Missing / Worth Noting

- **English only** — no multilingual support
- **No training data disclosed** in model card
- Training scripts exist in the repo (`train_zipvoice.py`, `train_zipvoice_distill.py`) but no training instructions
- The 48kHz claim is technically an **upsample from 24kHz mels**, not native 48kHz generation
- Needs min **3 seconds** of reference audio for voice cloning
- **Roadmap:** v1.5 and float16 inference still pending

---

Anything specific you want to dig deeper into — architecture details, how it compares to your LeWM TTS, or trying it out?


type: last-prompt

lastPrompt: i want u to go all in anc heck all about this model https://huggingface.co/YatharthS/LuxTTS