<external_links>
### Potentially Relevant Websearch Results

You should respond as if these information are known to you. Refrain from saying "I am unable to browse the internet" or "I don't have access to the internet" or "I'm unable to provide real-time news updates". This is your internet search results. Please always cite any links you referenced from the above search results in your response in markdown format.

-------
Website URL: https://huggingface.co/fishaudio/s2-pro
Website Title: fishaudio/s2-pro · Hugging Face
Website Content:
Hugging Face's logo Hugging Face

* Models
* Datasets
* Spaces
* Buckets new
* Docs
* Enterprise
* Pricing
* ---
* Log In
* Sign Up

#

fishaudio

/

s2-pro

like 385

Follow

Fish Audio 1.15k

Text-to-Speech Safetensors

83 languages

fish\_qwen3\_omni instruction-following multilingual

arxiv: 2603.08823

License: fish-audio-research-license

 Model card Files Files and versions xet  Community 11

* Fish Audio S2 Pro
   * Architecture
   * Fine-Grained Inline Control
   * Supported Languages
   * Production Streaming Performance
   * Links
   * Technical Report
   * License

#  Fish Audio S2 Pro

Fish Audio S2 Pro overview — fine-grained control, multi-speaker multi-turn generation, low-latency streaming, and long-context inference.

**Technical Report** | **GitHub** | **Playground**

**Fish Audio S2 Pro** is a leading text-to-speech (TTS) model with fine-grained inline control of prosody and emotion. Trained on over 10M+ hours of audio data across 80+ languages, the system combines reinforcement learning alignment with a dual-autoregressive architecture. The release includes model weights, fine-tuning code, and an SGLang-based streaming inference engine.

##  Architecture

S2 Pro builds on a decoder-only transformer combined with an RVQ-based audio codec (10 codebooks, \~21 Hz frame rate) using a **Dual-Autoregressive (Dual-AR)** architecture:

* **Slow AR** (4B parameters): Operates along the time axis and predicts the primary semantic codebook.
* **Fast AR** (400M parameters): Generates the remaining 9 residual codebooks at each time step, reconstructing fine-grained acoustic detail.

This asymmetric design keeps inference efficient while preserving audio fidelity. Because the Dual-AR architecture is structurally isomorphic to standard autoregressive LLMs, it inherits all LLM-native serving optimizations from SGLang — including continuous batching, paged KV cache, CUDA graph replay, and RadixAttention-based prefix caching.

##  Fine-Grained Inline Control

S2 Pro enables localized control over speech generation by embedding natural-language instructions directly within the text using `[tag]` syntax. Rather than relying on a fixed set of predefined tags, S2 Pro accepts **free-form textual descriptions** — such as `[whisper in small voice]`, `[professional broadcast tone]`, or `[pitch up]` — allowing open-ended expression control at the word level.

**Common tags (15,000+ unique tags supported):**

`[pause]` `[emphasis]` `[laughing]` `[inhale]` `[chuckle]` `[tsk]` `[singing]` `[excited]` `[laughing tone]` `[interrupting]` `[chuckling]` `[excited tone]` `[volume up]` `[echo]` `[angry]` `[low volume]` `[sigh]` `[low voice]` `[whisper]` `[screaming]` `[shouting]` `[loud]` `[surprised]` `[short pause]` `[exhale]` `[delight]` `[panting]` `[audience laughter]` `[with strong accent]` `[volume down]` `[clearing throat]` `[sad]` `[moaning]` `[shocked]`

##  Supported Languages

S2 Pro supports 80+ languages.

**Tier 1:** Japanese (ja), English (en), Chinese (zh)

**Tier 2:** Korean (ko), Spanish (es), Portuguese (pt), Arabic (ar), Russian (ru), French (fr), German (de)

**Other supported languages:** sv, it, tr, no, nl, cy, eu, ca, da, gl, ta, hu, fi, pl, et, hi, la, ur, th, vi, jw, bn, yo, xsl, cs, sw, nn, he, ms, uk, id, kk, bg, lv, my, tl, sk, ne, fa, af, el, bo, hr, ro, sn, mi, yi, am, be, km, is, az, sd, br, sq, ps, mn, ht, ml, sr, sa, te, ka, bs, pa, lt, kn, si, hy, mr, as, gu, fo, and more.

##  Production Streaming Performance

On a single NVIDIA H200 GPU:

* **Real-Time Factor (RTF):** 0.195
* **Time-to-first-audio:** \~100 ms
* **Throughput:** 3,000+ acoustic tokens/s while maintaining RTF below 0.5

##  Links

* Fish Speech GitHub
* Fish Audio Playground
* Blog & Tech Report

##  Technical Report

If you find our work useful, please consider citing our report:

```bibtex
@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report},
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823},
}

```

##  License

This model is licensed under the Fish Audio Research License. Research and non-commercial use is permitted free of charge. Commercial use requires a separate license from Fish Audio — contact business@fish.audio.

Downloads last month

3,964

Safetensors

Model size

5B params

Tensor type

BF16

·

Chat template

Files info

Inference Providers NEW

Text-to-Speech

This model isn't deployed by any Inference Provider. 🙋 9 Ask for provider support

##  Model tree for fishaudio/s2-pro

Finetunes

2 models

##  Spaces usingfishaudio/s2-pro 4

🐟 artificialguybr/fish-s2-pro-zero 🐟 mcuo/fish-s2-pro-zero 🐟 fguilleme/fish-s2-pro-zero 🐟 MAYA-AI/fish-s2-pro-zero

##  Collection includingfishaudio/s2-pro

Fish Audio S2  Collection 1 item • Updated4 days ago •  7

##  Paper forfishaudio/s2-pro

Fish Audio S2 Technical Report  Paper• 2603.08823 • Published5 days ago •  22

 System theme

Company

TOS Privacy About Careers

Website

Models Datasets Spaces Pricing Docs
____

</external_links>
<user_query>
https://huggingface.co/fishaudio/s2-pro

Let's check whether we can fine-tune this model on a specific language.
</user_query>


Let me look into the fine-tuning capabilities for Fish Audio S2 Pro. I'll check the GitHub repository and the model's details, as well as your existing project context.

The user wants to explore whether they can fine-tune the Fish Audio S2 Pro model on a specific language. Let me gather more information about this model, particularly around fine-tuning capabilities. The HuggingFace page mentions "fine-tuning code" is included in the release, and links to a GitHub repository. Let me check the GitHub repo and any available documentation.

Let me also check what the user has been working on recently - they have a `download_all.py` file in `/home/ubuntu/modi/` that was recently viewed.


Let me dig deeper into the fine-tuning documentation and technical report.

Let me get more details from the Fish Speech GitHub repository, particularly about fine-tuning documentation. Let me also check the technical report and any fine-tuning guides.


Let me search for the fine-tuning documentation and check the repo structure more closely.

Let me try to find the fine-tuning documentation more specifically. The model page mentions "fine-tuning code" is included. Let me look at the GitHub repo structure and find the relevant docs.


Let me read the technical report and the official fine-tuning guide.

Let me read the technical report and the fine-tuning documentation page.


Let me check what you've already downloaded in the `modi` directory to understand the language context.

Now I have a comprehensive picture. Let me also check what's in the user's `modi` directory to understand what language they might be working with, and see if I can determine from the YouTube URLs what kind of content they're downloading.

The user has a `download_all.py` script that downloads YouTube audio as WAV files. This looks like it could be dataset preparation for TTS fine-tuning. The videos could be in a specific language - let me check if there are any metadata files already downloaded.

Let me also check if the fine-tuning docs mention using the s2-pro model specifically or just the s1-mini.


Now I have the full picture. Here's a comprehensive analysis of fine-tuning Fish Audio S2 Pro for your use case.

---

## Fine-Tuning Fish Audio S2 Pro - Assessment

### Your Data

You have **130 "Mann Ki Baat" episodes** (PM Modi's radio addresses) downloaded as WAV files in `/home/ubuntu/modi/`. The audio is in **Hindi** (`"language": "hi"`), with episodes averaging ~30 minutes each. That's roughly **65+ hours of raw audio** -- a substantial dataset.

### Can You Fine-Tune S2 Pro?

**Yes, technically it's possible**, but there are important caveats:

#### 1. Hindi is Already Supported (but not Tier 1)

Hindi (`hi`) falls in the "Other supported languages" tier -- not Tier 1 (ja, en, zh) or Tier 2 (ko, es, pt, ar, ru, fr, de). This means fine-tuning on Hindi data could genuinely improve quality for this language, since the base model has less Hindi training data relative to top-tier languages.

#### 2. Official Warning About RL-Trained Models

The fine-tuning docs carry an explicit warning:

> *"We highly do not recommend users to do fine-tuning on an RL trained model. Fine-tuning a model after RL can shift the model distribution, which may lead to degraded performance."*

S2 Pro **is** RL-trained (via GRPO). This means LoRA fine-tuning may undo some of the alignment gains. However, for a language that's underrepresented in training, the trade-off could still be net positive.

#### 3. Fine-Tuning Code References `openaudio-s1-mini`, Not `s2-pro`

The current fine-tuning documentation and tooling use `openaudio-s1-mini` checkpoints. The S2 Pro model uses the same Dual-AR architecture (built on Qwen3-4B) and the same codec, so the pipeline should work with S2 Pro weights, but you'd need to point the checkpoint paths to `s2-pro` instead of `s1-mini`.

### What You'd Need to Do

The pipeline would be:

**Step 1 - Segment & Transcribe** your long-form audio into short utterances (5-30 seconds each). You'd need:
- Voice Activity Detection (VAD) to split 30-min episodes into clips
- Hindi ASR transcription for each clip (e.g., Whisper large-v3)
- Speaker diarization (Mann Ki Baat has one primary speaker but occasionally others)

**Step 2 - Extract semantic tokens** using the codec:
```bash
python tools/vqgan/extract_vq.py data \
    --num-workers 1 --batch-size 16 \
    --config-name "modded_dac_vq" \
    --checkpoint-path "checkpoints/s2-pro/codec.pth"
```

**Step 3 - Pack dataset** into protobuf format:
```bash
python tools/llama/build_dataset.py \
    --input "data" --output "data/protos" \
    --text-extension .lab --num-workers 16
```

**Step 4 - LoRA fine-tuning**:
```bash
python fish_speech/train.py --config-name text2semantic_finetune \
    project=modi_hindi \
    +lora@model.model.lora_config=r_8_alpha_16
```

### Hardware Requirements

- The S2 Pro model is **4B parameters** (Slow AR) + 400M (Fast AR)
- For inference alone, you need **24GB+ VRAM**
- For LoRA fine-tuning, you'd likely need an **A100 80GB** or similar (or an H100)

### Practical Recommendation

Before committing to fine-tuning, it would be worth:

1. **Testing zero-shot first** -- S2 Pro supports voice cloning from a short reference sample (10-30 sec). Try using a Modi speech clip as a reference prompt and see if the Hindi output quality is acceptable without fine-tuning.
2. **If quality is lacking**, then fine-tuning makes sense, particularly since your dataset is large and focused on a single speaker/language.

Would you like to proceed with setting up the fine-tuning pipeline? The first concrete step would be segmenting and transcribing your Mann Ki Baat audio files.

Now I have the full picture. The user is downloading "Mann Ki Baat" episodes from PM Modi's YouTube channel. These are in Hindi (language: "hi"). The user has 130 WAV files and 130 metadata JSON files downloaded. They want to fine-tune Fish Audio S2 Pro on Hindi.

Let me provide a comprehensive assessment:

1. **Can it be done?** Yes, technically. Fish Audio S2 Pro supports Hindi ("hi" is listed in the supported languages), and the fine-tuning code is openly available.

2. **Key considerations:**
   - The fine-tuning docs reference `openaudio-s1-mini` checkpoints, not `s2-pro` directly.

I need to verify whether the fine-tuning pipeline actually supports the s2-pro model, since the official documentation warns that fine-tuning on RL-trained models like S2 Pro can degrade performance. Hindi is classified as a Tier 2+ language with room for improvement, but the user's data—long-form Mann Ki Baat episodes of 30+ minutes each—would require segmentation and corresponding transcription files (.lab format) to work with the pipeline.


<external_links>
### Potentially Relevant Websearch Results

You should respond as if these information are known to you. Refrain from saying "I am unable to browse the internet" or "I don't have access to the internet" or "I'm unable to provide real-time news updates". This is your internet search results. Please always cite any links you referenced from the above search results in your response in markdown format.

-------
Website URL: https://github.com/fishaudio/fish-speech
Website Title: GitHub - fishaudio/fish-speech: SOTA Open Source TTS · GitHub
Website Content:
Skip to content

## Navigation Menu

Toggle navigation

 Sign in

Appearance settings

* Platform
   * AI CODE CREATION
         * GitHub CopilotWrite better code with AI
         * GitHub SparkBuild and deploy intelligent apps
         * GitHub ModelsManage and compare prompts
         * MCP RegistryNewIntegrate external tools
   * DEVELOPER WORKFLOWS
         * ActionsAutomate any workflow
         * CodespacesInstant dev environments
         * IssuesPlan and track work
         * Code ReviewManage code changes
   * APPLICATION SECURITY
         * GitHub Advanced SecurityFind and fix vulnerabilities
         * Code securitySecure your code as you build
         * Secret protectionStop leaks before they start
   * EXPLORE
         * Why GitHub
         * Documentation
         * Blog
         * Changelog
         * Marketplace
View all features
* Solutions
   * BY COMPANY SIZE
         * Enterprises
         * Small and medium teams
         * Startups
         * Nonprofits
   * BY USE CASE
         * App Modernization
         * DevSecOps
         * DevOps
         * CI/CD
         * View all use cases
   * BY INDUSTRY
         * Healthcare
         * Financial services
         * Manufacturing
         * Government
         * View all industries
View all solutions
* Resources
   * EXPLORE BY TOPIC
         * AI
         * Software Development
         * DevOps
         * Security
         * View all topics
   * EXPLORE BY TYPE
         * Customer stories
         * Events & webinars
         * Ebooks & reports
         * Business insights
         * GitHub Skills
   * SUPPORT & SERVICES
         * Documentation
         * Customer support
         * Community forum
         * Trust center
         * Partners
View all resources
* Open Source
   * COMMUNITY
         * GitHub SponsorsFund open source developers
   * PROGRAMS
         * Security Lab
         * Maintainer Community
         * Accelerator
         * Archive Program
   * REPOSITORIES
         * Topics
         * Trending
         * Collections
* Enterprise
   * ENTERPRISE SOLUTIONS
         * Enterprise platformAI-powered developer platform
   * AVAILABLE ADD-ONS
         * GitHub Advanced SecurityEnterprise-grade security features
         * Copilot for BusinessEnterprise-grade AI features
         * Premium SupportEnterprise-grade 24/7 support
* Pricing

Search or jump to...

# Search code, repositories, users, issues, pull requests...

 Search

Clear

Search syntax tips

#  Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

 Cancel  Submit feedback

#  Saved searches

## Use saved searches to filter your results more quickly

Name

Query

 To see all available qualifiers, see our documentation.

 Cancel  Create saved search

 Sign in

 Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert

{{ message }}

 fishaudio / **fish-speech** Public

* Notifications You must be signed in to change notification settings
* Fork2.3k
* Star 27k

* Code
* Issues 13
* Pull requests 15
* Discussions
* Actions
* Security 0
* Insights

Additional navigation options

* Code
* Issues
* Pull requests
* Discussions
* Actions
* Security
* Insights

# fishaudio/fish-speech

main

BranchesTags

Go to file

Code

Open more actions menu

## Folders and files

| Name                                                                                | Name                                                                                | Last commit message | Last commit date |
| ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- |
| Latest commit History[710 Commits](/fishaudio/fish-speech/commits/main/)710 Commits                              |                                                                                |                     |                  |
| [.github](/fishaudio/fish-speech/tree/main/.github ".github")                                                    | [.github](/fishaudio/fish-speech/tree/main/.github ".github")                                                    |                     |                  |
| [docker](/fishaudio/fish-speech/tree/main/docker "docker")                                                       | [docker](/fishaudio/fish-speech/tree/main/docker "docker")                                                       |                     |                  |
| [docs](/fishaudio/fish-speech/tree/main/docs "docs")                                                             | [docs](/fishaudio/fish-speech/tree/main/docs "docs")                                                             |                     |                  |
| [fish\_speech](/fishaudio/fish-speech/tree/main/fish%5Fspeech "fish_speech")                                     | [fish\_speech](/fishaudio/fish-speech/tree/main/fish%5Fspeech "fish_speech")                                     |                     |                  |
| [tools](/fishaudio/fish-speech/tree/main/tools "tools")                                                          | [tools](/fishaudio/fish-speech/tree/main/tools "tools")                                                          |                     |                  |
| [.dockerignore](/fishaudio/fish-speech/blob/main/.dockerignore ".dockerignore")                                  | [.dockerignore](/fishaudio/fish-speech/blob/main/.dockerignore ".dockerignore")                                  |                     |                  |
| [.gitignore](/fishaudio/fish-speech/blob/main/.gitignore ".gitignore")                                           | [.gitignore](/fishaudio/fish-speech/blob/main/.gitignore ".gitignore")                                           |                     |                  |
| [.pre-commit-config.yaml](/fishaudio/fish-speech/blob/main/.pre-commit-config.yaml ".pre-commit-config.yaml")    | [.pre-commit-config.yaml](/fishaudio/fish-speech/blob/main/.pre-commit-config.yaml ".pre-commit-config.yaml")    |                     |                  |
| [.project-root](/fishaudio/fish-speech/blob/main/.project-root ".project-root")                                  | [.project-root](/fishaudio/fish-speech/blob/main/.project-root ".project-root")                                  |                     |                  |
| [.readthedocs.yaml](/fishaudio/fish-speech/blob/main/.readthedocs.yaml ".readthedocs.yaml")                      | [.readthedocs.yaml](/fishaudio/fish-speech/blob/main/.readthedocs.yaml ".readthedocs.yaml")                      |                     |                  |
| [API\_FLAGS.txt](/fishaudio/fish-speech/blob/main/API%5FFLAGS.txt "API_FLAGS.txt")                               | [API\_FLAGS.txt](/fishaudio/fish-speech/blob/main/API%5FFLAGS.txt "API_FLAGS.txt")                               |                     |                  |
| [FishAudioS2TecReport.pdf](/fishaudio/fish-speech/blob/main/FishAudioS2TecReport.pdf "FishAudioS2TecReport.pdf") | [FishAudioS2TecReport.pdf](/fishaudio/fish-speech/blob/main/FishAudioS2TecReport.pdf "FishAudioS2TecReport.pdf") |                     |                  |
| [LICENSE](/fishaudio/fish-speech/blob/main/LICENSE "LICENSE")                                                    | [LICENSE](/fishaudio/fish-speech/blob/main/LICENSE "LICENSE")                                                    |                     |                  |
| [README.md](/fishaudio/fish-speech/blob/main/README.md "README.md")                                              | [README.md](/fishaudio/fish-speech/blob/main/README.md "README.md")                                              |                     |                  |
| [compose.base.yml](/fishaudio/fish-speech/blob/main/compose.base.yml "compose.base.yml")                         | [compose.base.yml](/fishaudio/fish-speech/blob/main/compose.base.yml "compose.base.yml")                         |                     |                  |
| [compose.yml](/fishaudio/fish-speech/blob/main/compose.yml "compose.yml")                                        | [compose.yml](/fishaudio/fish-speech/blob/main/compose.yml "compose.yml")                                        |                     |                  |
| [dockerfile.dev](/fishaudio/fish-speech/blob/main/dockerfile.dev "dockerfile.dev")                               | [dockerfile.dev](/fishaudio/fish-speech/blob/main/dockerfile.dev "dockerfile.dev")                               |                     |                  |
| [entrypoint.sh](/fishaudio/fish-speech/blob/main/entrypoint.sh "entrypoint.sh")                                  | [entrypoint.sh](/fishaudio/fish-speech/blob/main/entrypoint.sh "entrypoint.sh")                                  |                     |                  |
| [inference.ipynb](/fishaudio/fish-speech/blob/main/inference.ipynb "inference.ipynb")                            | [inference.ipynb](/fishaudio/fish-speech/blob/main/inference.ipynb "inference.ipynb")                            |                     |                  |
| [mkdocs.yml](/fishaudio/fish-speech/blob/main/mkdocs.yml "mkdocs.yml")                                           | [mkdocs.yml](/fishaudio/fish-speech/blob/main/mkdocs.yml "mkdocs.yml")                                           |                     |                  |
| [pyproject.toml](/fishaudio/fish-speech/blob/main/pyproject.toml "pyproject.toml")                               | [pyproject.toml](/fishaudio/fish-speech/blob/main/pyproject.toml "pyproject.toml")                               |                     |                  |
| [pyrightconfig.json](/fishaudio/fish-speech/blob/main/pyrightconfig.json "pyrightconfig.json")                   | [pyrightconfig.json](/fishaudio/fish-speech/blob/main/pyrightconfig.json "pyrightconfig.json")                   |                     |                  |
| [uv.lock](/fishaudio/fish-speech/blob/main/uv.lock "uv.lock")                                                    | [uv.lock](/fishaudio/fish-speech/blob/main/uv.lock "uv.lock")                                                    |                     |                  |
| View all files                                                                                |                                                                                |                     |                  |

## Repository files navigation

* README
* License

# Fish Speech

**English** | 简体中文 | Portuguese | 日本語 | 한국어 | العربية

Fish Audio S1 - Expressive Voice Cloning and Text-to-Speech | Product Hunt fishaudio%2Ffish-speech | Trendshift

Discord Docker QQ Channel

HuggingFace Model Fish Audio Blog Paper | Technical Report

Important

**License Notice**
This codebase and its associated model weights are released under **FISH AUDIO RESEARCH LICENSE**. Please refer to LICENSE for more details. We will take action against any violation of the license.

Warning

**Legal Disclaimer**
We do not hold any responsibility for any illegal usage of the codebase. Please refer to your local laws about DMCA and other related laws.

## Quick Start

### For Human

Here are the official documents for Fish Audio S2, follow the instructions to get started easily.

* Installation
* Command Line Inference
* WebUI Inference
* Server Inference
* Docker Setup

Important

**For SGLang server, please read SGLang-Omni README.**

### For LLM Agent

```
Install and configure Fish-Audio S2 by following the instructions here: https://speech.fish.audio/install/

```

## Fish Audio S2

**Best text-to-speech system among both open source and closed source**

Fish Audio S2 is the latest model developed by Fish Audio. Trained on over 10 million hours of audio across approximately 50 languages, S2 combines reinforcement learning alignment with a Dual-Autoregressive architecture to generate speech that sounds natural, realistic, and emotionally rich.

S2 supports fine-grained inline control of prosody and emotion using natural-language tags like `[laugh]`, `[whispers]`, and `[super happy]`, as well as native multi-speaker and multi-turn generation.

Visit the Fish Audio website for live playground. Read the blog post and technical report for more details.

### Model Variants

| Model  | Size          | Availability                                           | Description                                                     |
| ------ | ------------- | ------------------------------------------------------ | --------------------------------------------------------------- |
| S2-Pro | 4B parameters | [HuggingFace](https://huggingface.co/fishaudio/s2-pro) | Full-featured flagship model with maximum quality and stability |

More details of the model can be found in the technical report.

## Benchmark Results

| Benchmark                                 | Fish Audio S2                |
| ----------------------------------------- | ---------------------------- |
| Seed-TTS Eval — WER (Chinese)             | **0.54%** (best overall)     |
| Seed-TTS Eval — WER (English)             | **0.99%** (best overall)     |
| Audio Turing Test (with instruction)      | **0.515** posterior mean     |
| EmergentTTS-Eval — Win Rate               | **81.88%** (highest overall) |
| Fish Instruction Benchmark — TAR          | **93.3%**                    |
| Fish Instruction Benchmark — Quality      | **4.51 / 5.0**               |
| Multilingual (MiniMax Testset) — Best WER | **11 of 24** languages       |
| Multilingual (MiniMax Testset) — Best SIM | **17 of 24** languages       |

On Seed-TTS Eval, S2 achieves the lowest WER among all evaluated models including closed-source systems: Qwen3-TTS (0.77/1.24), MiniMax Speech-02 (0.99/1.90), Seed-TTS (1.12/2.25). On the Audio Turing Test, 0.515 surpasses Seed-TTS (0.417) by 24% and MiniMax-Speech (0.387) by 33%. On EmergentTTS-Eval, S2 achieves particularly strong results in paralinguistics (91.61% win rate), questions (84.41%), and syntactic complexity (83.39%).

## Highlights

### Fine-Grained Inline Control via Natural Language

S2 enables localized control over speech generation by embedding natural-language instructions directly at specific word or phrase positions within the text. Rather than relying on a fixed set of predefined tags, S2 accepts free-form textual descriptions — such as `[whisper in small voice]`, `[professional broadcast tone]`, or `[pitch up]` — allowing open-ended expression control at the word level.

### Dual-Autoregressive Architecture

S2 builds on a decoder-only transformer combined with an RVQ-based audio codec (10 codebooks, \~21 Hz frame rate). The Dual-AR architecture splits generation into two stages:

* **Slow AR** operates along the time axis and predicts the primary semantic codebook.
* **Fast AR** generates the remaining 9 residual codebooks at each time step, reconstructing fine-grained acoustic detail.

This asymmetric design — 4B parameters along the time axis, 400M parameters along the depth axis — keeps inference efficient while preserving audio fidelity.

### Reinforcement Learning Alignment

S2 uses Group Relative Policy Optimization (GRPO) for post-training alignment. The same models used to filter and annotate training data are directly reused as reward models during RL — eliminating distribution mismatch between pre-training data and post-training objectives. The reward signal combines semantic accuracy, instruction adherence, acoustic preference scoring, and timbre similarity.

### Production Streaming via SGLang

Because the Dual-AR architecture is structurally isomorphic to standard autoregressive LLMs, S2 directly inherits all LLM-native serving optimizations from SGLang — including continuous batching, paged KV cache, CUDA graph replay, and RadixAttention-based prefix caching.

On a single NVIDIA H200 GPU:

* **Real-Time Factor (RTF):** 0.195
* **Time-to-first-audio:** \~100 ms
* **Throughput:** 3,000+ acoustic tokens/s while maintaining RTF below 0.5

### Multilingual Support

S2 supports high-quality multilingual text-to-speech without requiring phonemes or language-specific preprocessing. Including:

**English, Chinese, Japanese, Korean, Arabics, German, French...**

**AND MORE!**

The list is constantly expanding, check Fish Audio for the latest releases.

### Native Multi-Speaker Generation

Fish Audio S2 allows users to upload reference audio with multi-speaker, the model will deal with every speaker's feature via `<|speaker:i|>` token. Then you can control the model's performance with the speaker id token, allowing a single generation to include multiple speakers. You no longer need to upload reference audio separately for each speaker.

### Multi-Turn Generation

Thanks to the expansion of the model context, our model can now use previous information to improve the expressiveness of subsequent generated content, thereby increasing the naturalness of the content.

### Rapid Voice Cloning

## Fish Audio S2 supports accurate voice cloning using a short reference sample (typically 10–30 seconds). The model captures timbre, speaking style, and emotional tendencies, producing realistic and consistent cloned voices without additional fine-tuning. Please refer to SGLang-Omni README to use the SGLang server.

## Credits

* VITS2 (daniilrobnikov)
* Bert-VITS2
* GPT VITS
* MQTTS
* GPT Fast
* GPT-SoVITS
* Qwen3

## Tech Report

@misc{fish-speech-v1.4,
      title={Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis},
      author={Shijia Liao and Yuxuan Wang and Tianyu Li and Yifan Cheng and Ruoyi Zhang and Rongzhi Zhou and Yijin Xing},
      year={2024},
      eprint={2411.01156},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2411.01156},
}

@misc{liao2026fishaudios2technical,
      title={Fish Audio S2 Technical Report},
      author={Shijia Liao and Yuxuan Wang and Songting Liu and Yifan Cheng and Ruoyi Zhang and Tianyu Li and Shidong Li and Yisheng Zheng and Xingwei Liu and Qingzheng Wang and Zhizhuo Zhou and Jiahua Liu and Xin Chen and Dawei Han},
      year={2026},
      eprint={2603.08823},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.08823},
}

## About

 SOTA Open Source TTS

speech.fish.audio

### Topics

 tts  transformer  llama  valle  vqvae  vits  vqgan

### Resources

 Readme

### License

 View license

###  Uh oh!

There was an error while loading. Please reload this page.

Activity

Custom properties

### Stars

**27k** stars

### Watchers

**147** watching

### Forks

**2.3k** forks

 Report repository

## Releases14

V1.5.1  Latest May 31, 2025

\+ 13 releases

## Contributors

###  Uh oh!

There was an error while loading. Please reload this page.

## Languages

* Python 95.0%
* Dockerfile 3.6%
* Jupyter Notebook 1.4%

## Footer

 © 2026 GitHub, Inc.

### Footer navigation

* Terms
* Privacy
* Security
* Status
* Community
* Docs
* Contact
* Manage cookies
* Do not share my personal information

 You can’t perform that action at this time.
____

</external_links>
<user_query>
https://github.com/fishaudio/fish-speech

check this
</user_query>


Let me explore the repository structure in detail, particularly the fine-tuning code and tools.

The user wants me to check the Fish Speech GitHub repository more thoroughly. Let me explore the repo structure, particularly the fine-tuning code, tools, and configuration files to understand what's available for fine-tuning S2 Pro.


Let me dig into the actual fine-tuning code, configs, and tools.

Let me explore the key files more - the fine-tuning config, the training script, tools, and the model structure.


Now let me pull the key source files for the fine-tuning pipeline.

Now let me get the key source files to understand the fine-tuning pipeline better - particularly the LoRA config, the model loading, and the build_dataset tool.


Now let me check the installation docs and the HuggingFace model files to see if the fine-tuning pipeline works with S2 Pro specifically.

Now I have a comprehensive understanding of the repo. Let me also check the install docs to understand the setup requirements.


Here's the full breakdown of the [Fish Speech GitHub repo](https://github.com/fishaudio/fish-speech) and what it means for your Hindi fine-tuning project:

---

## Repository Structure Overview

The repo is well-organized for both inference and fine-tuning:

| Path | Purpose |
|------|---------|
| `fish_speech/models/text2semantic/llama.py` | Core Dual-AR model (Qwen3-4B backbone) |
| `fish_speech/models/text2semantic/lora.py` | LoRA implementation |
| `fish_speech/models/text2semantic/lit_module.py` | PyTorch Lightning training module |
| `fish_speech/models/dac/` | Audio codec (tokenizer) |
| `fish_speech/configs/text2semantic_finetune.yaml` | Fine-tune config |
| `fish_speech/configs/lora/r_8_alpha_16.yaml` | LoRA hyperparams (r=8, alpha=16) |
| `fish_speech/train.py` | Training entry point (Hydra + Lightning) |
| `tools/vqgan/extract_vq.py` | Semantic token extraction |
| `tools/llama/build_dataset.py` | Dataset packing to protobuf |
| `tools/llama/merge_lora.py` | LoRA weight merging post-training |

## Key Findings

**1. Fine-tuning pipeline is complete and functional.** The repo provides the full chain: data prep, token extraction, dataset building, LoRA training, and LoRA merging.

**2. The fine-tune config defaults to `openaudio-s1-mini`, not `s2-pro`.** In `text2semantic_finetune.yaml`:

```yaml
pretrained_ckpt_path: checkpoints/openaudio-s1-mini
```

You'd need to change this to `checkpoints/s2-pro` to fine-tune the full S2 Pro model. The architecture is the same (Dual-AR based on `BaseTransformer`), so the training code should work -- but it will require significantly more VRAM (4B vs a smaller S1-mini).

**3. LoRA config is lightweight.** The default `r_8_alpha_16.yaml` is:

```yaml
r: 8
lora_alpha: 16
lora_dropout: 0.01
```

This is quite conservative. For a language adaptation task, you might want `r: 16` or `r: 32` to give the model more capacity to learn Hindi-specific patterns.

**4. Training uses PyTorch Lightning + Hydra.** Defaults are:
- `max_steps: 10000`
- `batch_size: 4`
- `lr: 1e-4` with AdamW
- Checkpoints every 100 steps
- Precision: `bf16-true`
- Strategy: DDP with NCCL

**5. Dataset format requires `.npy` semantic tokens + `.lab` text files**, organized by speaker directory. The `build_dataset.py` tool reads text from `.lab` (or `.txt`) files and pairs them with pre-extracted `.npy` semantic token arrays.

**6. System requirements:**
- Linux (or WSL)
- Python 3.12
- PyTorch 2.8.0
- 24GB VRAM minimum for inference; fine-tuning S2 Pro with LoRA will likely need **40-80GB VRAM**
- `portaudio19-dev`, `libsox-dev`, `ffmpeg` as system dependencies

## What You'd Need for Hindi Fine-Tuning

Your current state: 130 long-form WAV files (~65+ hours) of Mann Ki Baat in Hindi. To get from here to a fine-tuned model, you need:

1. **Segment audio** -- VAD to split 30-min episodes into 5-30s clips
2. **Transcribe** -- Hindi ASR (Whisper large-v3 works well for Hindi) to produce `.lab` files
3. **Clone the repo and install** -- `git clone`, `pip install -e .[cu129]`
4. **Download S2 Pro weights** -- `huggingface-cli download fishaudio/s2-pro --local-dir checkpoints/s2-pro`
5. **Extract semantic tokens** -- `tools/vqgan/extract_vq.py` with S2 Pro codec
6. **Build protobuf dataset** -- `tools/llama/build_dataset.py`
7. **LoRA fine-tune** -- `fish_speech/train.py` with modified config pointing to S2 Pro
8. **Merge LoRA** -- `tools/llama/merge_lora.py`

Want me to start building out the data preparation pipeline (segmentation + transcription) for your Mann Ki Baat audio?


<user_query>
I mean, why do I want to fine-tune it? I mean, kid, I'm not talking about LORA; I am talking about fine-tuning. They mentioned it can do indic languages cloning, but the indic language cloning is like shit. If we give, it is only speaking the reference text correctly, but the remaining is shitting over that, so unable to get that. I think he's not trained on more English-speaking things, so I thought of doing a fine-tuning on that on the indic languages so it can do better results.
</user_query>


Got it -- you're talking about **full fine-tuning**, not LoRA. And the core problem is clear:

**S2 Pro's Indic language support is weak in practice.** It can parrot back the reference audio's text, but when you ask it to generate *new* Hindi text, the quality falls apart -- bad pronunciation, broken prosody, garbled output. The model's text-to-semantic mapping for Hindi is undertrained compared to English/Chinese/Japanese.

Full fine-tuning is the right instinct for this. LoRA wouldn't be enough -- you need to actually update the model's understanding of how Hindi text maps to speech tokens, which means training the full 4B parameter Slow AR backbone.

Let me check what hardware you're working with, because that's the critical constraint.

The user is clarifying their intent:

1. They don't want LoRA fine-tuning -- they want **full fine-tuning** of the model
2. The problem: S2 Pro claims to support Indic languages (Hindi etc.), but in practice:
   - When you give it reference audio for voice cloning, it only reproduces the reference text correctly
   - When generating new text in Hindi, the quality is terrible ("shitting over that")
   - They believe the model is undertrained on Indic languages
3. Their goal: Full fine-tuning on Indic language data to improve the model's Hindi/Indic TTS quality

4. Full fine-tuning is a much heavier lift than LoRA -- it'll need multiple high-end GPUs and significant changes to the existing LoRA-focused fine-tuning setup, though the data preparation steps remain the same

.

Looking at what they're actually dealing with—the model can clone voices from reference audio but struggles with generating new Hindi text—it seems the real bottleneck is the weak Hindi language understanding rather than the voice cloning itself. Full fine-tuning on Hindi speech data could help, though I should be practical about what that entails and what hardware they'd actually need to make it work.


You have an **A100 80GB** -- that's excellent. Full fine-tuning of the 4B model is absolutely doable on this hardware with gradient checkpointing + bf16. No need for multi-GPU hacks.

Let me check disk usage since you're at 83% full.

They have:
- 1x NVIDIA A100 80GB SXM4 -- this is great for full fine-tuning!
- 118GB RAM
- 16 CPU cores
- 166GB free disk space (that's a concern -- model weights + training data + checkpoints could eat that up)

With a single A100 80GB, full fine-tuning of a 4B model is feasible with:
- DeepSpeed ZeRO-2 or ZeRO-3
- Gradient checkpointing
- bf16 training
- Gradient accumulation

A 4B parameter model in bf16 takes ~8GB for weights alone. With optimizer states (AdamW: 2x model size for momentum + variance), that's another ~16GB. Gradients are another ~8GB.

So roughly 32GB for the core training state, and with gradient checkpointing to reduce activation memory, the A100 should handle this comfortably. The existing fine-tuning setup uses PyTorch Lightning with DDP, so I just need to adjust the configuration for single-GPU training.

For the actual task, they want to fine-tune S2 Pro on their Hindi data to boost Indic language performance. The workflow starts with preparing their 65+ hours of Mann Ki Baat recordings through segmentation and transcription, then cloning the fish-speech repo, downloading the S2 Pro weights, and extracting semantic tokens from the audio.

From there, I'd build the protobuf dataset, modify the training config to remove LoRA and potentially reduce the learning rate while enabling gradient checkpointing, then run the training. The main constraint is disk space—they've got 166GB free, but the raw audio alone is around 45GB, plus the model weights at 8-10GB, segmented clips, semantic tokens, and checkpoints will add up quickly. I should recommend converting everything to mono 44.1kHz WAV first since that's what the model's codec expects.

For the actual fine-tuning setup with their A100 80GB, it's straightforward—disable LoRA, drop the learning rate to 1e-5 or lower, enable gradient checkpointing if needed, and run single-GPU training without DDP. But I need to verify the disk usage of their existing data first and figure out a practical approach to the space constraints.


Interesting -- you already have `modi_processed/` and `kani-tts-2-pretrain/`. Let me understand what you've already done.

Interesting! The user already has:
- `modi_processed/` (26GB) -- likely already segmented audio
- `transcripts/` (222GB) -- a large transcription dataset
- `soprano_data/` (90GB) -- some other audio dataset
- `kani-tts-2-pretrain/` (88GB) -- a TTS pretraining dataset!

This tells me the user has experience with TTS and has been working on this. They have a `kani-tts-2-pretrain` directory which suggests they've done TTS pretraining before. Let me check what's in `modi_processed` and `kani-tts-2-pretrain` to understand their current state better.

With only 166GB free, I'm concerned about disk space -- the S2 Pro weights alone would take 10-15GB, and training checkpoints will consume more. They might need to clean up some existing data or mount an external volume. Let me examine what's actually in the processed data and their existing TTS setup.


You've already been through this -- segmented data, transcriptions, and even attempted both LoRA and full fine-tuning already. Let me see what happened with your previous attempts.

Now I see the full picture:

1. The user has already:
   - Downloaded 130 Mann Ki Baat episodes (43GB raw WAV)
   - Segmented them into utterances (`/home/ubuntu/modi_processed/segments/`)
   - Transcribed them with Hindi ASR (the `dataset.jsonl` has text + audio path pairs)
   - Already attempted voice cloning with S2 Pro (the `final_samples/` directory has clone results)
   - Already done LoRA fine-tuning (`lora_output/`, `lora_output_v2/`)
   - Tried full fine-tuning as well (`full_finetune_output/`)
   - Built their own TTS model called "KaniTTS" (88GB, a 380M model with NanoCodec)

2. They've already been through the pain of trying to make Fish Audio S2 Pro work for Hindi—voice cloning gave poor results, LoRA didn't help enough, and even full fine-tuning didn't solve it. So they've moved on to their own custom TTS model and are now trying to figure out the next steps.

Let me look at what their previous fine-tuning attempts revealed.


<user_query>
I don't know why you're checking our directives. We don't care about the things. We have so many things in this machine, which is static for now. Those are like experiments. We don't touch those. Coming to the dataset, we have so much of dataset. You don't have to worry about that actually. We have like 10k hours of data for Hindi, Telugu, and some other languages. I am just asking you like how we can proceed. I want you to research on this to provide fish audio. Like, why the fuck are you searching this locally?
</user_query>


You're right, I apologize for wasting time digging through your local files. You have the data -- 10k hours across Hindi, Telugu, and other Indic languages. The question is purely about **how to do full fine-tuning of Fish Audio S2 Pro** (not LoRA) to improve Indic language quality.

Let me research this properly.


Key findings emerging. Let me dig into the specific GitHub issues about Hindi fine-tuning failures and the data requirements.

This is very revealing. Let me dig into those specific GitHub issues, especially #682 (Hindi finetuning producing noisy results) and #843 (how to continue training). Also the data quality issue #598. And the key finding: "No more plan for supporting hindi" - this is important context.


Here's what the research reveals. Let me be direct about it.

---

## Research Summary: Full Fine-Tuning Fish Audio S2 Pro for Indic Languages

### The Hard Truth from Fish Audio

1. **Fish Audio has explicitly abandoned Hindi support.** In [issue #682](https://github.com/fishaudio/fish-speech/issues/682), someone trained for **239,400 steps** on 48 hours of clean Hindi data and still got noisy garbage. A maintainer responded with *"No more plan for supporting hindi"* and closed the issue.

2. **The only documented fine-tuning path is LoRA.** There is zero documentation for full fine-tuning. The config has `lora_config: null` as a placeholder, but no one at Fish Audio has documented or supported the full fine-tuning path.

3. **They explicitly warn against fine-tuning RL-trained models** (which S2 Pro is). Their stated reason: it shifts model distribution and degrades performance.

### Why Indic Languages Fail

The problem is architectural. S2 Pro has two components:

- **Audio Codec (446M params)** -- Converts waveforms to/from discrete tokens. This is **language-agnostic** and should handle Hindi/Telugu phonetics fine. No need to touch this.

- **Slow AR (Qwen3-4B)** -- Maps text to semantic tokens. **This is where the problem is.** The Qwen3-4B backbone was pre-trained on 10M+ hours, but overwhelmingly on English/Chinese/Japanese. It simply doesn't know how to map Hindi/Telugu text to the correct semantic tokens. When you give it reference audio, it can parrot the reference because it's just copying tokens. But for new text, it doesn't know the text-to-speech mapping for Indic scripts.

- **Fast AR (400M)** -- Reconstructs acoustic details from semantic tokens. Language-agnostic, no need to touch.

### What Full Fine-Tuning Actually Means Here

You'd be doing **continued pre-training of the Slow AR** on Indic language text-speech pairs. Specifically:

1. **Freeze the codec** -- Don't touch the audio tokenizer
2. **Train the Slow AR (4B)** -- All parameters, no LoRA, on your 10K hours of Indic data
3. **Optionally train the Fast AR (400M)** -- Though this is less critical

### Data Requirements (from Fish Audio devs)

Per [issue #598](https://github.com/fishaudio/fish-speech/issues/598), a dev stated:
> *"If you want to fine-tune for a new language, you may need about 2K hours of low quality data, and about 100h (the more, the better) high quality data (44.1kHz with accurate labels)."*

You have 10K hours -- that exceeds their recommendation by 5x, which is good.

### The Practical Challenges

**1. The RL alignment will be destroyed.** Full fine-tuning will undo the GRPO alignment. But since that alignment doesn't work for Hindi anyway, this is an acceptable trade-off. You'd ideally want to do your own SFT stage after pre-training on Indic data.

**2. Catastrophic forgetting.** If you only train on Indic data, the model will lose English/Chinese/Japanese capabilities. If you want to preserve multilingual ability, you'd need to mix in some English/Chinese data during training.

**3. Training time with 1x A100 80GB.** Full fine-tuning of 4B params on 10K hours of data is essentially continued pre-training. With gradient checkpointing + bf16 + batch size ~4, you're looking at **weeks of training** on a single A100. Multiple GPUs would significantly help.

**4. Audio must be 44.1kHz.** The S2 Pro codec operates at 44.1kHz. Your current pipeline resamples to 24kHz (from your pipeline.py). All data would need to be at 44.1kHz for S2 Pro.

**5. Segments should be ~30 seconds max** (confirmed by devs).

### How to Actually Do It

The training command without LoRA is simply:

```bash
python fish_speech/train.py --config-name text2semantic_finetune \
    project=indic_full_finetune \
    pretrained_ckpt_path=checkpoints/s2-pro
```

No `+lora@model.model.lora_config=r_8_alpha_16` flag. Since `lora_config` defaults to `null`, this runs full fine-tuning. But you'd need to modify the config significantly:

- Lower learning rate (`1e-5` or `5e-6` instead of `1e-4`)
- Add gradient checkpointing (critical for 4B on 1 GPU)
- Increase `max_steps` massively (this isn't a 10K step job)
- Point to your Indic dataset protos
- Consider DeepSpeed ZeRO-2 for memory optimization

### Honest Assessment

With 10K hours of Indic data and your experience (you've already built KaniTTS), full fine-tuning is technically feasible. But the **issue #682 failure** (noisy output after 239K steps on 48 hours) is a red flag -- it suggests something beyond just "not enough data" might be going on. It could be that the codec's semantic distillation (trained on w2v-BERT 2.0, which is primarily English) doesn't produce good semantic tokens for Indic languages in the first place. If that's the case, even full fine-tuning of the Slow AR won't fix it -- you'd need to retrain the codec too, which is a much bigger undertaking.

Want me to dig deeper into whether the codec is the bottleneck, or do you want to proceed with setting up the full fine-tuning pipeline for the Slow AR?

Now I have a comprehensive picture. Let me also check the Fish Audio blog post about S2 and the discussions/issues about multi-language training.

Key findings from all the research:

1. **Fish Audio's official position on Hindi**: In issue #682, a maintainer (Stardust-minus) literally said "No more plan for supporting hindi" and closed the issue. Someone tried fine-tuning with 48 hours of Hindi data on fish-speech-1.4, trained for 239,400 steps, and still got noisy output.

2. **Official fine-tuning only supports LoRA**: The documentation and codebase only document LoRA fine-tuning. Full fine-tuning is technically possible by just not passing the LoRA config flag, but it's not documented or supported.

3. **The S2 Pro model is RL-trained**: The team explicitly warns against fine-tuning RL-trained models, which is a significant concern for full fine-tuning too. 4. **Data requirements for new language**: According to the devs, you'd need roughly 2,000 hours of low quality data plus 100 hours of high quality data at 44.1kHz with accurate labels, with segments cut to around 30 seconds. 5. **The architecture**: S2 Pro uses a Dual-AR architecture with a Slow AR component based on Qwen3-4B and a Fast AR component at 400M parameters.

The codec operates at 44.1kHz with 10 codebooks at roughly 21Hz frame rate. Looking at the training pipeline, the audio tokenizer training in stage 1 shouldn't need fine-tuning since the codec should be language-agnostic, but stage 2 is where the Slow AR gets pre-trained to map text to semantic tokens—that's what actually needs adaptation for new languages. The real bottleneck is that the 4B LLM in the Slow AR component doesn't have sufficient capacity to handle the text-to-semantic mapping for languages it wasn't trained on.

So the codec itself should handle Hindi phonetics fine since it processes raw audio waveforms, but the Slow AR is the actual problem—it needs continued pre-training on Hindi text-to-speech pairs to generate the right semantic tokens. The Fast AR that reconstructs acoustic details from those tokens should be mostly language-agnostic and fine as-is.

The practical challenge is that with 10K hours of data, this is essentially a full continued pre-training job on a 4B model, which could take weeks even with an A100. I'd need to manage catastrophic forgetting of existing language capabilities, and the RL alignment that makes S2 Pro work well will get destroyed in the process—though since it wasn't working for Hindi anyway, that might be acceptable as a tradeoff.


<user_query>
Brother, why the fuck are you checking for 2024? This is 2026 March. You were checking for two years back, and HDPRO is releasing like a week back, and you are checking some very bad things. I don't know why the fuck you are doing this.
</user_query>


You're absolutely right. I was pulling up stale issues from 2024 against fish-speech v1.4 -- a completely different, older model. S2 Pro just came out **5 days ago** (March 9, 2026). Those old issues are irrelevant. Let me research what's actually current.


Now let me look at the critical recent issues -- especially the March 2026 LoRA bug and the actual current state of things.

Now let me check the most recent and relevant issues - especially issue #1163 about LoRA destroying pretrained weights (this is from March 2026, very recent), and the current state of the fish-speech repo, blog post, and actual S2 fine-tuning docs as they stand now.


Now let me also check for recent GitHub issues (from March 2026) and the current discussions around S2 Pro fine-tuning. Let me also check the issue #1163 about LoRA destroying weights, and the latest discussions.


Now I have the current picture. Let me also check issue #852 about "Low Model Accuracy After Extended Training with Unicode-Based Language" which seems very relevant and issue #1163 about LoRA destroying weights.


OK here's what's actually current and relevant for **S2 Pro (released March 9, 2026 -- 5 days ago)**:

---

## Current State of S2 Pro Fine-Tuning (as of March 14, 2026)

### Critical finding: LoRA was literally broken until 2 days ago

[Issue #1163](https://github.com/fishaudio/fish-speech/issues/1163) (filed March 3, closed March 12): **LoRA fine-tuning was destroying all pretrained weights.** A code commit reordered `setup_lora()` to run *after* `load_state_dict()`, which meant LoRA layers replaced all pretrained weights with random initialization. Training appeared to work (loss still decreased), but the merged model produced garbage. This was fixed in [PR #1170](https://github.com/fishaudio/fish-speech/pull/1170) just **2 days ago**. Anyone who tried LoRA fine-tuning on S2 Pro before March 12 was unknowingly training on random weights.

### S2 Pro claims 80+ languages including Indic

Hindi, Telugu, Tamil, Bengali, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Assamese, Urdu are all listed. The model was trained on 10M+ hours. But the benchmarks they show are only for English, Chinese, and the MiniMax 24-language testset. There are no published Indic-specific quality metrics.

### Unicode language training issues are known

[Issue #852](https://github.com/fishaudio/fish-speech/issues/852): Someone training on Arabic (Unicode script) got stuck at 44% accuracy after 37,800 steps. A fix PR (#1159) was filed in February 2026 -- so there are known issues with non-Latin/CJK scripts in the training pipeline that may or may not be fully resolved.

### Full fine-tuning path

The existing training config supports full fine-tuning by simply **not passing the LoRA flag**. The config already has `lora_config: null` as default. On your A100 80GB:

- 4B params in bf16 = ~8GB weights
- AdamW states = ~16GB
- Gradients = ~8GB
- With gradient checkpointing, activations are manageable
- **Total ~40-50GB -- fits on a single A100 80GB**

### What would actually need to happen

Since S2 Pro is 5 days old, there's essentially **no community experience** with fine-tuning it yet (and the LoRA path was broken until 2 days ago). You'd be among the first doing this. The path would be:

1. **Clone latest fish-speech** (make sure you have the March 12 fixes)
2. **Download S2 Pro weights** from HuggingFace
3. **Prepare your 10K hours**: audio at 44.1kHz, segments ~30s, with `.lab` transcription files
4. **Extract semantic tokens** using S2 Pro's codec
5. **Build protobuf dataset**
6. **Run full fine-tuning** (no LoRA flag, lower LR ~1e-5 to 5e-6, gradient checkpointing, long training run)

The open question is whether the **codec itself** handles Indic languages well. The codec uses semantic distillation from w2v-BERT 2.0, and if that model's representations are weak for Hindi/Telugu, then even perfect Slow AR training won't help -- the bottleneck would be at the tokenizer level. That's something you'd discover quickly by checking the quality of extracted semantic tokens on your Indic data.

Want me to start setting up the pipeline -- clone the repo, pull the weights, and test semantic token extraction on some of your Hindi audio first?

Now I have the complete current picture. Let me summarize what I've found that's actually relevant to S2 Pro (March 2026):

1. **S2 Pro was released March 9, 2026** -- just 5 days ago. It's brand new.

2. **S2 Pro officially supports 80+ languages including Hindi, Telugu, Tamil, Bengali, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Assamese, Urdu** -- but user's experience is that the quality for Indic languages is poor (can clone reference text but new text generation is garbage).

3. **There was a critical LoRA fine-tuning bug that just got fixed** -- it was corrupting the pretrained weights, so anyone who tried fine-tuning before March 12 was essentially training on broken weights, which could account for a lot of the failures people are seeing.

4. **Unicode-based languages like Arabic are struggling with accuracy** -- even after extensive training runs, they're only hitting 44-46% top-5 accuracy, and there's been a PR attempt to address this since February.

5. The fine-tuning documentation appears to be incomplete or still being updated.

6. The S2 Pro uses a Qwen3-4B model as its main component (the Slow AR), which is what actually needs to be trained for new languages, alongside smaller Fast AR and codec modules.

7. Full fine-tuning on a single A100 80GB GPU is actually feasible -- the weights take about 8GB in bf16, the optimizer states need roughly 16GB, gradients another 8GB, and with gradient checkpointing the activations fit within the remaining headroom, totaling around 40-50GB.


<user_query>
what about Magpie TTS
</user_query>


Here's what Magpie TTS is and how it compares to S2 Pro for your use case:

---

## Magpie TTS (NVIDIA, NeMo Framework)

**Released**: v2602 on **March 3, 2026** -- also brand new, 11 days ago. **Hindi support was just added in this release.**

### What It Is

- **357M parameters** (241M trainable) -- encoder-decoder transformer
- By NVIDIA, part of the [NeMo](https://github.com/NVIDIA/NeMo) speech framework
- Uses [NanoCodec](https://huggingface.co/nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps) (22kHz, 8 codebooks)
- **Anti-hallucination by design** -- uses monotonic alignment (CTC loss + attention priors) to prevent skipping/repeating words
- Trained on **50K hours** of audio
- 9 languages: En, Es, De, Fr, Vi, It, Zh, **Hi**, Ja

### Hindi Training Data

They used [AI4bharat Kathbath](https://huggingface.co/datasets/ai4bharat/Kathbath) for Hindi. That's a relatively small Hindi dataset. With your 10K hours of Indic data, you could train something significantly better.

### Fine-Tuning: Fully Documented

This is the big difference. NVIDIA has documented the **entire training pipeline**:

- **Full training** from scratch or continued pre-training
- **DPO** (Direct Preference Optimization) for quality refinement
- **GRPO** (same as Fish Audio uses) for online preference optimization
- All through Hydra configs with clear commands

Training command:
```bash
python examples/tts/magpietts.py \
    --config-name=magpietts \
    max_epochs=100 \
    batch_size=16 \
    model.codecmodel_path=/path/to/codec.nemo \
    train_ds_meta.dataset.manifest_path=/path/to/manifest.json
```

GRPO alignment:
```bash
python examples/tts/magpietts.py \
    +init_from_ptl_ckpt=/path/to/checkpoint \
    +mode="onlinepo_train" \
    +model.loss_type="grpo" \
    +model.cer_reward_weight=0.33 \
    +model.ssim_reward_weight=0.33
```

### Head-to-Head: Magpie TTS vs Fish Audio S2 Pro

| | **Magpie TTS** | **Fish Audio S2 Pro** |
|---|---|---|
| **Size** | 357M | 4.4B (4B + 400M) |
| **Hindi support** | Official (March 2026) | Listed but quality poor |
| **Full fine-tuning docs** | Complete (train/DPO/GRPO) | None (only LoRA documented) |
| **Training on 1x A100 80GB** | Easy, fast | Tight, needs gradient checkpointing |
| **Training speed (10K hrs data)** | Days | Weeks |
| **Audio quality** | 22kHz | 44.1kHz |
| **Voice cloning** | Fixed speakers in multilingual; zeroshot English-only | Zero-shot in all languages |
| **Hallucination handling** | Monotonic alignment (CTC) | No explicit mechanism |
| **RL alignment** | GRPO documented | GRPO done but fine-tuning warned against |
| **License** | NVIDIA Open Model (commercial OK) | Fish Audio Research License (commercial needs separate license) |
| **Codec** | NanoCodec (22kHz) | Modified DAC (44.1kHz) |
| **Inline control** | No | Yes ([tag] syntax) |
| **Known bugs** | None reported yet | LoRA was broken until 2 days ago |

### The Key Trade-offs

**Magpie TTS advantages for your case:**
- 12x smaller model = way faster to train and iterate
- Full training pipeline is documented and supported by NVIDIA
- Already has Hindi in the training mix, so you're extending, not starting from scratch
- Monotonic alignment prevents the hallucination/garbage output issue
- GRPO post-training is documented so you can do proper alignment after fine-tuning
- Commercial license

**Fish Audio S2 Pro advantages:**
- Much larger model = potentially more expressive
- 44.1kHz output quality vs 22kHz
- Zero-shot voice cloning for any language (if it works)
- Inline prosody/emotion control via natural language tags
- Larger pre-training data (10M hours vs 50K hours)

### The Catch with Magpie TTS

The multilingual model has **5 fixed speakers** and the zero-shot cloning is **English-only** in the current release. If your goal is Hindi voice cloning of arbitrary speakers, you'd need to either:
1. Fine-tune with your target speakers in the training data
2. Extend the zeroshot/flow architecture to support Hindi (more work)

---

**Bottom line**: If your primary goal is high-quality Hindi/Telugu TTS with your 10K hours and you want a well-documented, trainable pipeline, Magpie TTS is the more practical path. If you specifically need zero-shot voice cloning in Hindi with inline emotion control, S2 Pro is the only option but the fine-tuning path is undocumented and risky.

What's your priority -- voice cloning for arbitrary speakers, or general high-quality Indic TTS?

Now I have a comprehensive picture of Magpie TTS. Let me compare it with Fish Audio S2 Pro for the user's use case.

Key findings about Magpie TTS:

1. **By NVIDIA** -- part of the NeMo framework
2. **357M parameters** -- much smaller than S2 Pro's 4.4B
3. **Supports Hindi** -- added in v2602 (March 3, 2026), so very recent
4. **Hindi training data**: Uses [AI4bharat Kathbath](https://huggingface.co/datasets/ai4bharat/Kathbath) dataset

5. **Full fine-tuning is well-documented** with training and reinforcement learning approaches, uses NanoCodec for efficient audio encoding at 22kHz, and employs monotonic alignment to prevent hallucinations. It supports voice cloning through audio context conditioning and comes with a commercial-friendly license, offering five pre-built speaker voices. The 357M parameter size makes it significantly easier to fine-tune on consumer hardware like an A100 80GB, and it was trained on 50K hours of data.

For the user's specific needs, Magpie TTS appears more practical than Fish Audio S2 Pro—it's a much smaller model with official Hindi support and fully documented fine-tuning paths, whereas Fish Audio's larger 4.4B parameter model has poor Indic quality, recently fixed LoRA support, and warnings against fine-tuning due to its RL training. With 10K hours of Indic data and a single A100, Magpie could be fully trained much faster, and its design specifically mitigates hallucination issues through monotonic alignment.

The tradeoff is that Magpie's multilingual variant sacrifices some capabilities—it's smaller and potentially less expressive, supports only 9 languages instead of 80+, outputs at 22kHz rather than 44.1kHz, and critically, it removed zero-shot voice cloning in favor of fixed speakers. But looking at the NVIDIA NIM documentation, there are actually separate variants: a multilingual model with fixed voices, plus English-only zeroshot and flow models that do support voice cloning. So the zero-shot capability exists, just not in the multilingual version.


<user_query>
I wanted to explain more about Magpie TTS.
</user_query>


Now I have a very deep understanding of Magpie TTS. Let me also get the Koel-TTS paper details since that's the research paper behind Magpie TTS.


OK, here's the deep dive on Magpie TTS.

---

## Magpie TTS In-Depth

### Origin: Koel-TTS Research

Magpie TTS is the productized version of NVIDIA's research paper **[Koel-TTS](https://arxiv.org/abs/2502.05236)** (ICML Workshop on Machine Learning for Audio, 2025). "Koel" is a bird known for its distinctive call -- the name signals their focus on natural-sounding speech. The paper directly describes the model architecture, training methodology, and preference alignment techniques that Magpie TTS uses.

### Architecture Deep Dive

**Encoder-Decoder Transformer** (not decoder-only like Fish Audio S2 Pro):

- **Text Encoder**: 6-layer NAR (non-autoregressive) transformer with causal masking. Takes text tokens and produces contextual representations. Supports two tokenization modes:
  - **IPA phonemes** via grapheme-to-phoneme conversion (used for English, German, Spanish)
  - **Byte-level character tokenizer** for languages without G2P (used for Hindi, Japanese, etc.)

- **Audio Decoder**: 12-layer AR (autoregressive) transformer that generates discrete audio tokens, attending to the encoded text via cross-attention. Predicts **all 8 codebooks in parallel** at each timestep (not sequentially like Fish Audio's Dual-AR). This is possible because NanoCodec uses Finite Scalar Quantization (FSQ) where codebooks are independent.

- **Context Encoder**: For voice cloning, they explored 3 approaches. The released multilingual model uses **Decoder Context** -- reference audio tokens are prepended to target audio tokens in the decoder, so the model conditions on them via self-attention. This was found to generalize best to unseen speakers.

### The Anti-Hallucination System

This is Magpie's key differentiator. LLM-based TTS models (including Fish Audio S2) hallucinate -- they skip words, repeat phrases, or drift out of alignment. Magpie fixes this with:

1. **CTC Alignment Loss**: Forces the cross-attention between decoder and encoder to follow a monotonic left-to-right pattern. The model literally cannot skip ahead or loop back in the text.

2. **Beta-Binomial Attention Prior**: During the first 10K training steps, the cross-attention is biased toward a diagonal pattern. This is annealed over the next 5K steps, then removed. The model internalizes monotonic behavior.

3. **Classifier-Free Guidance (CFG)**: During training, text and audio conditioning are randomly dropped out (10% probability). At inference, conditional and unconditional logits are interpolated with a scale factor (gamma=2.5 optimal), which sharpens adherence to the input text and speaker identity.

### Audio Codec: NanoCodec

- **62M parameters**
- **22kHz sample rate**, 12.5 frames per second
- **8 codebooks** using Finite Scalar Quantization (FSQ)
- **1.89 kbps** bitrate
- **Partially causal**: non-causal encoder + causal decoder (enables streaming)
- 3 discriminators: multi-period, multi-band multi-scale STFT, and WavLM-based

The low frame rate (12.5 FPS vs Fish Audio's 21 FPS) means fewer autoregressive steps needed, making inference faster. But 22kHz output is notably lower quality than Fish Audio's 44.1kHz.

### Training Pipeline

**Supervised training** on ~50K hours:
- English: LibriTTS, HiFiTTS, LibriVox MLS, internal data (~18K hrs)
- German: CML-TTS (~1.5K hrs)
- Spanish, French, Italian, Dutch: CML-TTS + internal data
- Hindi: AI4bharat Kathbath dataset
- Japanese, Vietnamese, Mandarin: internal + public datasets

**Preference Alignment** (the post-training secret sauce):

Two options, both fully documented:

**DPO (Direct Preference Optimization)**:
1. Curate text-context pairs (mix of challenging tongue-twisters + regular text)
2. Generate 6 audio samples per pair
3. Score each with ASR (CER) + Speaker Verification (SSIM)
4. Pareto-optimal ranking to select chosen/rejected pairs
5. Fine-tune with DPO loss (beta=0.01, LR=2e-7, 4K iterations)

**GRPO (Group Relative Policy Optimization)** -- recommended:
1. Generate 12 candidate outputs per training example on-the-fly
2. Compute rewards: CER (0.45), SSIM (0.45), PESQ (0.1)
3. Optimize via policy gradient
4. Uses Whisper for multilingual CER computation
5. LR=1e-7, no dropout, no attention priors during GRPO

The results are dramatic. DPO + CFG on the multi-encoder model:
- CER dropped from 2.56% to 0.58% (unseen speakers)
- SSIM improved from 0.601 to 0.678

### Hindi Specifics

Hindi was added in v2602 (March 3, 2026). Key details:
- Uses **byte-level character tokenizer** (not IPA phonemes) -- so no need for Hindi G2P
- Training data: [AI4bharat Kathbath](https://huggingface.co/datasets/ai4bharat/Kathbath) -- this is a relatively small dataset
- Uses Whisper for multilingual ASR reward computation during GRPO
- 5 fixed speakers can speak Hindi (Sofia, Aria, Jason, Leo, John)

### What This Means for Your 10K Hours of Indic Data

**Magpie TTS is far more practical to fine-tune than Fish Audio S2 Pro for your use case:**

1. **357M params** -- full fine-tuning takes hours, not weeks, on your A100 80GB
2. **Complete training pipeline** -- supervised training, DPO, GRPO all documented with exact commands
3. **NeMo framework** -- battle-tested, NVIDIA-supported, not a fresh release with broken LoRA
4. **Manifest format** is simple JSON-lines -- easy to create from your existing dataset.jsonl
5. **Byte-level tokenizer** for Hindi means no phoneme conversion needed
6. **GRPO post-training** is documented and would let you do proper RL alignment on your Indic data after fine-tuning

**The gap you could fill**: Their Hindi training data (Kathbath) is small. With 10K hours, you could make the definitive Indic language Magpie TTS model. The pipeline would be:

1. Convert your data to NeMo manifest format
2. Resample to 22kHz (NanoCodec requirement)
3. Pre-compute audio codec tokens
4. Full supervised training (continued from their multilingual checkpoint)
5. GRPO alignment with Whisper-based CER rewards

**The trade-off**: 22kHz output, fixed speakers (no zero-shot cloning in multilingual mode). If you need voice cloning for arbitrary Hindi speakers, you'd need to adapt the zeroshot architecture for multilingual.

Want me to dig into anything specific -- the GRPO setup, the codec details, or how to adapt the zeroshot model for Hindi?


<user_query>
First, I want you to explain what is this decoder-only transformer, encoder and decoder transformers, and these things. Don't need to take these examples; you can explain to me normally.
</user_query>


Sure. Let me explain these architecture patterns from first principles.

---

## Transformer Architectures for TTS

### The Original Transformer (Encoder-Decoder)

The transformer was introduced for machine translation -- converting a sentence in one language to another. It has two halves:

**Encoder**: Reads the entire input at once. Every token can attend to every other token (bidirectional). It builds a rich representation of the input. Think of it as the "understanding" half -- it reads the full sentence, understands context, and produces a set of hidden states.

**Decoder**: Generates the output one token at a time, left to right. It can look at everything the encoder produced (via cross-attention), but it can only look at what it has generated so far (causal/masked self-attention). Think of it as the "speaking" half -- it produces one word at a time, peeking at the encoder's understanding of the input while doing so.

The **cross-attention** is the bridge between them. At each step, the decoder asks: "Given what I've generated so far, which parts of the encoder's representation should I focus on?" For TTS, this means: "I'm generating audio frame 47 -- which word in the text should I be pronouncing right now?"

### Decoder-Only Transformer

This throws away the encoder entirely. Everything -- input and output -- goes into a single sequence fed to the decoder. The model just predicts the next token given all previous tokens.

For TTS, a decoder-only model like Fish Audio S2 Pro does this: it takes the text tokens, appends audio tokens, and treats it all as one long sequence. The model predicts the next audio token based on all the text tokens and all previous audio tokens. There's no explicit encoder or cross-attention -- the model figures out the text-to-audio mapping implicitly through self-attention over the concatenated sequence.

**Why it works**: This is how GPT-style LLMs work. Since the Slow AR in S2 Pro is literally a Qwen3-4B language model, they can reuse all LLM infrastructure -- KV caching, continuous batching, SGLang optimizations. It's architecturally simple.

**The problem**: There's no explicit mechanism forcing the model to align text with audio monotonically. The model has to learn on its own that "token 5 in the text corresponds to audio frames 30-45." If it doesn't learn this well (which happens for low-resource languages like Hindi), it hallucinates -- skips words, repeats phrases, or generates garbage.

### Encoder-Decoder for TTS (what Magpie uses)

Magpie TTS uses the original encoder-decoder structure. The text goes through the encoder. The audio goes through the decoder. Cross-attention connects them.

**Why this matters for your problem**: The cross-attention creates an explicit alignment matrix between text positions and audio positions. You can literally visualize it -- it's a 2D grid where each cell says "how much should audio frame X attend to text token Y." In natural speech, this matrix should look like a diagonal line going from top-left to bottom-right, because speech follows text left to right.

Magpie enforces this diagonal pattern with CTC loss and attention priors. This is why it doesn't hallucinate -- it mechanically cannot skip a word or loop back, because the alignment is constrained to be monotonic.

A decoder-only model has no such alignment matrix. The text-audio relationship is buried in self-attention across a long concatenated sequence. There's nothing to constrain or regularize.

### Why This Distinction Matters for Indic Languages

When a model is well-trained on a language (English, Chinese), decoder-only works great -- it has learned the text-to-audio mapping from millions of hours of data. The implicit alignment in self-attention is sufficient.

For underrepresented languages like Hindi or Telugu, the model hasn't seen enough data to learn this mapping properly. A decoder-only model fails silently -- it generates something that sounds like speech but doesn't follow the input text. The reference audio part works because it's just copying tokens, but new text generation falls apart.

An encoder-decoder model with explicit alignment (like Magpie) degrades more gracefully. Even if the model hasn't seen much Hindi, the monotonic cross-attention constraint ensures it at least tries to pronounce each word in order. The output quality may be lower, but it won't produce incoherent garbage.

### The Dual-AR Trick (Fish Audio specific)

Fish Audio S2 Pro has an additional wrinkle. Their audio codec produces 10 codebooks per timestep. If you flatten all of them into the sequence, it becomes 10x longer, which is brutal for a decoder-only model.

Their solution: split into two models:
- **Slow AR (4B)**: Predicts only the first codebook (the "semantic" one) along the time axis. This is the main language model.
- **Fast AR (400M)**: At each timestep, takes the Slow AR's hidden state and generates the remaining 9 codebooks (the "acoustic detail" ones) depth-wise.

Magpie avoids this entirely because NanoCodec's 8 codebooks are independent (FSQ), so the decoder predicts all 8 in parallel at each timestep with a single linear layer. No second model needed.

---

The short version: **encoder-decoder with cross-attention gives you an explicit, constrainable alignment between text and audio. Decoder-only is simpler and scales better but has no alignment guarantee, which is why it fails on low-resource languages.**


<user_query>
how magpie tts zeroshot working how they trained and hwoe we cantrain
</user_query>


OK here's how Magpie TTS zero-shot voice cloning works, from the ground up.

---

## How Magpie TTS Zero-Shot Works

### The Core Idea

Zero-shot voice cloning means: give the model a short audio clip (3-10 seconds) of someone it has **never heard during training**, and it generates new speech in that person's voice. No fine-tuning, no retraining -- just one audio clip at inference time.

### Three Approaches They Built

The Koel-TTS paper explored three different ways to condition the model on a reference voice. Understanding all three helps explain why they chose what they chose.

---

**Approach 1: SV Conditioned (Speaker Verification Embedding)**

This is the simplest approach.

1. Take the reference audio
2. Pass it through a pre-trained speaker verification model (TitaNet-Large, a separate model that produces a 128-dimensional vector representing "who this person is")
3. Project that 128-dim vector to the transformer's hidden dimension
4. Repeat it across time and **add** it to the text encoder's output
5. The decoder sees text representations that are "colored" by the speaker identity

Think of it like: you take a fingerprint of the voice, then stamp that fingerprint onto every text token before the decoder sees it.

**Problem**: The speaker embedding is a compressed summary. It captures *who* is speaking (timbre, pitch range) but loses *how* they speak (pacing, accent, style, breath patterns). The cloned voice sounds like the right person but doesn't capture their mannerisms.

---

**Approach 2: Decoder Context (What the Multilingual Model Uses)**

This is what the released multilingual model uses.

1. Take the reference audio
2. Convert it to audio codec tokens using NanoCodec (the same codec used for output)
3. **Prepend** those reference audio tokens before the target audio tokens in the decoder

So the decoder input looks like:

```
[REF_TOKEN_1] [REF_TOKEN_2] ... [REF_TOKEN_N] [TARGET_TOKEN_1] [TARGET_TOKEN_2] ...
```

The decoder uses **self-attention** -- meaning every target token can attend to every reference token. The model learns: "The reference tokens show me what this voice sounds like. I should generate target tokens that sound like the same person."

During training, they randomly sample a 5-second slice from the same speaker's other utterances as the reference. The model learns to extract speaker characteristics from the reference and apply them to the new text.

**Why it works well for zero-shot**: The decoder sees the actual raw audio tokens of the reference, not a compressed embedding. It has access to all the fine-grained information -- timbre, speaking rate, breath patterns, accent, background characteristics. The self-attention mechanism figures out what to copy (voice identity) and what to generate fresh (new content).

**Why it generalizes best**: In their experiments, this approach had the best speaker similarity on **unseen** speakers. Because it's directly attending to the raw audio tokens, it doesn't need to have "learned" about a specific speaker during training. It just pattern-matches from whatever reference you give it.

---

**Approach 3: Multi-Encoder (Separate Context Encoder)**

1. Take the reference audio tokens
2. Pass them through a **dedicated context encoder** (a separate NAR transformer)
3. The decoder has **alternating cross-attention layers**:
   - Odd layers attend to the text encoder output
   - Even layers attend to the context encoder output

So the decoder alternates between asking "what should I say?" (text cross-attention) and "how should I sound?" (context cross-attention).

**Problem**: This performed best on **seen** speakers but worst on **unseen** speakers. It memorized training voices rather than learning a general voice-cloning capability. It overfits.

---

### The Zero-Shot Model (NIM Release)

The **Magpie TTS Zeroshot** model available on NVIDIA NIM uses a variant specific to zero-shot:

- **English + Spanish + French only** (not multilingual like the 357M model)
- Uses IPA phoneme tokenization for English/Spanish, character-based for French
- Uses the **21 FPS Low Frame Rate Speech Codec** (slightly different from the 12.5 FPS NanoCodec in the multilingual model -- higher frame rate = more detail)
- Has 11 built-in voices with emotions (neutral, calm, happy, fearful, angry, sad, disgusted)
- Supports **streaming** inference
- Has a `quality` parameter (1-40) that controls how closely the output matches the reference voice vs. speed

The **Magpie TTS Flow** model is a second variant that also takes the **transcript** of the reference audio as input. Knowing what words are being said in the reference helps the model better separate "what" from "who" -- leading to higher fidelity cloning, but it requires you to provide the transcript and only works offline.

### How Training Works for Zero-Shot

The training is straightforward. You need:

1. **Multi-speaker dataset**: Thousands of speakers, each with multiple utterances
2. For each training sample, you create a triplet:
   - **Context audio**: A random utterance from speaker X (this becomes the reference)
   - **Text**: The transcript of a *different* utterance from the same speaker X
   - **Target audio**: The actual audio of that different utterance

The model learns: given the context audio and the text, generate the target audio that sounds like the same person as the context but says the text content.

In NeMo manifest format:

```json
{
  "audio_filepath": "/path/to/target.wav",
  "text": "the words to synthesize",
  "duration": 8.5,
  "speaker": 42,
  "context_audio_filepath": "/path/to/reference.wav",
  "context_audio_duration": 5.0
}
```

The key insight: the model never sees speaker 42's audio at test time. But because it was trained on thousands of other speakers with the same "reference -> generate" pattern, it learns the general skill of voice cloning. At inference, you give it speaker 9999 (never seen), and the pattern still works.

### How You Could Train This for Hindi/Telugu

Here's what you'd need:

**Step 1: Data preparation**

Your 10K hours of Indic data needs to be organized by speaker. For each speaker, you need multiple utterances. The training pipeline will randomly pair utterances from the same speaker as (context, target) pairs.

Manifest format:
```json
{"audio_filepath": "/data/speaker_001/utt_001.wav", "text": "नमस्ते दोस्तों", "duration": 6.2, "speaker": 1}
{"audio_filepath": "/data/speaker_001/utt_002.wav", "text": "आज हम बात करेंगे", "duration": 4.8, "speaker": 1}
{"audio_filepath": "/data/speaker_002/utt_001.wav", "text": "తెలుగు భాష చాలా", "duration": 5.1, "speaker": 2}
```

The more speakers you have, the better the zero-shot generalization. If your 10K hours is mostly one speaker (like Modi), that won't teach voice cloning -- it'll teach a very good Modi voice. For zero-shot, you want hundreds to thousands of distinct speakers.

**Step 2: Audio codec tokenization**

Pre-compute NanoCodec tokens for all your audio:
```bash
# NeMo handles this, or you pre-compute tokens
# Audio needs to be at 22kHz
```

**Step 3: Training**

Two options:

**Option A: Continue from multilingual checkpoint** (recommended)

The multilingual model already has some Hindi capability. You continue training on your larger Indic dataset:

```bash
python examples/tts/magpietts.py \
    --config-name=magpietts \
    +init_from_ptl_ckpt=/path/to/magpie_multilingual_checkpoint \
    max_epochs=50 \
    batch_size=16 \
    model.codecmodel_path=nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
    train_ds_meta.indic.manifest_path=/path/to/train_manifest.json \
    train_ds_meta.indic.audio_dir=/ \
    val_ds_meta.indic_val.manifest_path=/path/to/val_manifest.json \
    val_ds_meta.indic_val.audio_dir=/
```

**Option B: Train the zeroshot architecture from scratch for Indic**

If you want proper zero-shot cloning (not fixed speakers), you'd need to use the decoder-context architecture and train on multi-speaker Indic data. The code is all in NeMo:
- Model: `nemo.collections.tts.models.MagpieTTSModel`
- Config: `magpietts` (Hydra config)
- Dataset class: `MagpieTTSDataset` with `context_audio_filepath` field

**Step 4: GRPO alignment**

After supervised training, run GRPO with Whisper as the ASR reward model (Whisper handles Hindi well):

```bash
python examples/tts/magpietts.py \
    +init_from_ptl_ckpt=/path/to/your_trained_checkpoint \
    +mode="onlinepo_train" \
    +model.loss_type="grpo" \
    +model.reward_asr_model="whisper" \
    +model.cer_reward_weight=0.45 \
    +model.ssim_reward_weight=0.45 \
    +model.pesq_reward_weight=0.1 \
    model.optim.lr=1e-7
```

### The Limitation for Your Case

The **open-source multilingual model** (357M on HuggingFace) has 5 fixed speakers -- no zero-shot cloning for Hindi. The **zero-shot models** (Zeroshot and Flow) are only available through NVIDIA NIM and only support English/Spanish/French.

So if you want **Hindi zero-shot voice cloning**, you'd need to train it yourself using the decoder-context architecture on your multi-speaker Indic dataset. The code and training pipeline are all available in NeMo -- it's the architecture and data that you'd need to bring.

The question is: do you have multi-speaker Indic data (many different speakers), or is your 10K hours mostly single-speaker or limited-speaker data?

Now let me explain how the zero-shot voice cloning works in Magpie TTS. Let me also check if there's a way to get the open-source zeroshot model weights or if it's only available through NIM.


<user_query>
did they not released the cloning checkpoints ? or how to do
</user_query>


The user is asking whether NVIDIA released the zero-shot voice cloning checkpoints for Magpie TTS, or if they're only available through NIM (their inference microservice). Let me search for this.


This is very interesting. Let me look at the key statement from the HuggingFace model card:

> "We have removed zero-shot capabilities of this model for this release."

So NVIDIA explicitly removed zero-shot cloning from the open-source checkpoint. The model on HuggingFace only has 5 fixed speakers (Sofia, Aria, Jason, Leo, John Van Stan) and cannot clone voices.

However, the NeMo framework code likely still supports the architecture for zero-shot (decoder context conditioning), and the NIM API offers the zero-shot model. But the open checkpoint doesn't have it.

Let me check if there's a way to train the zero-shot capability or if there's any other open checkpoint that supports it.

I should look at the NeMo source code to understand how the context encoder works and what the zero-shot training setup requires.


Now I have a very clear picture. Let me look at the key line from the HuggingFace model card:

> "We have removed zero-shot capabilities of this model for this release."

So NVIDIA explicitly removed zero-shot voice cloning from the open-source checkpoint (`nvidia/magpie_tts_multilingual_357m`). The open model only has 5 fixed speakers.

However:
1. The **NeMo framework code** fully supports zero-shot training (context encoder, decoder_context_tts configuration, etc.)
2. The **NIM API** offers `magpie-tts-zeroshot` and `magpie-tts-flow` models for voice cloning (English only)

3. The zero-shot NIM models require enterprise approval and aren't directly downloadable, while the public HuggingFace checkpoint has this capability intentionally stripped out

So the real constraint is that NVIDIA hasn't released an open-source zero-shot checkpoint—the only way to access zero-shot voice cloning is through their proprietary NIM API with approval, or by training a custom model from scratch using the NeMo framework's decoder_context_tts configuration.

For the user's specific needs, they could either request NIM API access (though it's limited to English, Spanish, and French), train their own zero-shot model from scratch with multi-speaker data, or attempt to fine-tune the open checkpoint by adding back the context encoder component and retraining it. Let me dig into the NeMo source to see what training configurations are actually available.


Now I have a clear picture. Let me also check the NeMo GitHub for the actual config files to understand what the zero-shot training config looks like.


Now I have a complete picture. Let me summarize what I found:

1. **The open-source checkpoint (`nvidia/magpie_tts_multilingual_357m`) explicitly has zero-shot voice cloning REMOVED**. From the HuggingFace model card: *"We have removed zero-shot capabilities of this model for this release."*

2. **The NIM API models** (`magpie-tts-zeroshot`, `magpie-tts-flow`) have zero-shot cloning but:
   - Only support English, Spanish, French
   - Not downloadable - require enterprise access approval
   - Only work through the NIM API

3. **The NeMo training framework fully supports zero-shot training** with multiple model types including context encoders and configurable context duration windows. The default configuration already includes the necessary components for zero-shot capability.

4. **I can train a zero-shot model from scratch** using NeMo by selecting the appropriate model type like `decoder_context_tts` or `decoder_ce` and configuring the context parameters.

For the dataset, I need to structure the manifest with multi-speaker audio paired alongside context audio samples, ensuring each entry includes the `context_audio_filepath` and `context_audio_duration` fields. Since there's no pre-trained cloning checkpoint available, I could either train entirely from scratch with the 10k hours of Indic data or leverage the existing multilingual checkpoint as initialization and fine-tune the context encoder component.


Here's the definitive answer on the zero-shot cloning checkpoint situation:

---

**No, NVIDIA has NOT released the zero-shot voice cloning checkpoint.** They explicitly state on the HuggingFace model card:

> *"We have removed zero-shot capabilities of this model for this release."*

---

## What's Actually Available

| Model | Downloadable? | Zero-Shot Cloning? | Languages |
|-------|--------------|-------------------|-----------|
| `nvidia/magpie_tts_multilingual_357m` (HuggingFace) | Yes, open-source | **No** -- 5 fixed English speakers only | 9 (incl. Hindi) |
| `magpie-tts-zeroshot` (NIM API) | **No** -- enterprise access only | Yes | En, Es, Fr only |
| `magpie-tts-flow` (NIM API) | **No** -- enterprise access only | Yes (needs transcript) | En, Es, Fr only |

So the open checkpoint can speak Hindi but only in those 5 fixed voices (Sofia, Aria, Jason, Leo, John Van Stan). The zero-shot models that can clone arbitrary voices are locked behind the NIM API and don't support Hindi anyway.

---

## How You CAN Do It: Train Zero-Shot Yourself

The good news is **the entire training framework for zero-shot IS open-source** in NeMo. The default config file already has everything set up for zero-shot context conditioning. Here's the actual default config:

```yaml
model:
  model_type: "decoder_ce"        # supports zero-shot via context encoder
  context_duration_min: 5.0       # 5 seconds of reference audio
  context_duration_max: 5.0
  use_text_conditioning_encoder: true
```

The `MagpieTTSDataset` already supports these manifest fields for zero-shot:

```json
{
  "audio_filepath": "/data/spk_001/utt_001.wav",
  "text": "target transcript here",
  "duration": 6.2,
  "speaker": 1,
  "context_audio_filepath": "/data/spk_001/utt_042.wav",
  "context_audio_duration": 5.0
}
```

The `context_audio_filepath` must be a **different utterance from the same speaker** -- this is how the model learns to extract speaker identity from a short reference and generalize to unseen speakers at inference.

### Concrete Strategy for Your Indic Data

**Step 1: Partial initialization from the multilingual checkpoint**

The open multilingual checkpoint already has trained encoder + decoder weights that understand Hindi text-to-audio mapping. You can load those weights and train only the new context encoder (a small 1-layer transformer) from scratch:

```bash
python examples/tts/magpietts.py \
    --config-name=magpietts \
    +init_from_ptl_ckpt=nvidia/magpie_tts_multilingual_357m \
    model.model_type=decoder_ce \
    model.context_duration_min=5.0 \
    model.context_duration_max=5.0 \
    max_epochs=50 \
    batch_size=16 \
    model.codecmodel_path=nvidia/nemo-nano-codec-22khz-1.89kbps-21.5fps \
    train_ds_meta.indic.manifest_path=/path/to/train_manifest.json \
    train_ds_meta.indic.audio_dir=/ \
    val_ds_meta.indic_val.manifest_path=/path/to/val_manifest.json \
    val_ds_meta.indic_val.audio_dir=/
```

The context encoder architecture from the config:

```yaml
context_encoder:
  n_layers: 1       # lightweight -- only 1 layer
  d_model: 768
  d_ffn: 3072
  sa_n_heads: 12
  is_causal: false   # bidirectional attention over reference audio
```

**Step 2: Data preparation**

Your 10k hours of multi-speaker Indic data is ideal. The critical requirement is:
- **Multiple speakers** (the more the better -- hundreds to thousands)
- **Multiple utterances per speaker** (so you can pair context and target from the same speaker)
- Audio at **22kHz** (NanoCodec's native sample rate; NeMo can auto-resample)
- Each manifest line needs `context_audio_filepath` pointing to a different utterance from the same speaker

You'd write a script that, for each utterance, randomly picks another utterance from the same speaker as the context.

**Step 3: GRPO alignment (after supervised training converges)**

```bash
python examples/tts/magpietts.py \
    +init_from_ptl_ckpt=/path/to/your_trained_checkpoint \
    +mode="onlinepo_train" \
    +model.loss_type="grpo" \
    +model.reward_asr_model="whisper" \
    +model.cer_reward_weight=0.33 \
    +model.ssim_reward_weight=0.33 \
    +model.pesq_reward_weight=0.33 \
    model.optim.lr=1e-7
```

---

## Summary

NVIDIA intentionally withheld the zero-shot weights for the open release. The NIM zero-shot models are enterprise-only and English/Spanish/French-limited. But the **training code and architecture are fully open** -- you'd need to train the zero-shot capability yourself using your multi-speaker Indic data, starting from the multilingual checkpoint that already understands Hindi. The context encoder is only a 1-layer transformer, so the incremental training cost on top of the pretrained base isn't massive.

Would you like me to help write the data preparation script that creates the zero-shot manifest with context pairings from your existing data?

Now I have the complete picture. Let me also check the MagpieTTSDataset documentation to understand the manifest format for context audio:

From the data doc:
> **MagpieTTSDataset** — For Magpie-TTS. In addition to the manifest structure for TextToSpeechDataset, supports `context_audio_filepath`, `context_audio_duration`, `target_audio_codes_path`, `context_audio_codes_path`. If code paths are missing or `load_cached_codes_if_available=False`, audio is loaded and codes computed in the model. Supports codec frame rate, BOS/EOS ids, prior scaling, tokenizer config, 16 kHz audio for speaker verification, text conditioning tokenizer, context duration min/max, and text context remapping.

The manifest format includes `context_audio_filepath` and `context_audio_duration` fields for zero-shot training. Now I can give the user a clear answer: NVIDIA hasn't released the zero-shot cloning checkpoint—the model card explicitly states they removed zero-shot capabilities for this release. The open-source checkpoint only supports 5 fixed English speakers across 9 languages, with no voice cloning capability.

Zero-shot cloning is restricted to their NIM services, which require enterprise access and only cover English, Spanish, and French. However, the training code is fully open-source in NeMo, and the config already supports context encoders with the necessary manifest fields. The user can actually train their own zero-shot model from scratch using NeMo with their multi-speaker Indic dataset, choosing between the `decoder_context_tts` or `decoder_ce` model types and potentially initializing the encoder/decoder weights.

The best approach would be initializing from the multilingual checkpoint to leverage its existing Hindi knowledge, letting the text encoder and decoder weights transfer over while training just the context encoder from scratch—a small 1-layer transformer that learns speaker characteristics from the audio context.


<user_query>
i have a xcodec codec trained internally in comapny for indic langaiegs cna i sue that
</user_query>


The user is asking if they can use their own internally trained XCodec (audio codec) for Indic languages instead of NVIDIA's NanoCodec with Magpie TTS. Let me think about this.

Key considerations:
1. Magpie TTS operates on discrete audio codec tokens - the architecture is codec-agnostic in principle
2. The default codec is NanoCodec (22kHz, 8 codebooks, ~21.5 FPS frame rate)
3. XCodec is a different codec architecture - I need to check compatibility requirements

The main questions are:
- What does Magpie TTS need from a codec? (number of codebooks, frame rate, vocabulary size)

- Whether you can swap in a custom codec and what configuration changes would be required

I should look into the specifics of how Magpie TTS interfaces with its codec to understand what constraints exist.