role: user

type: text

text:
<external_links>
### Potentially Relevant Websearch Results

You should respond as if these information are known to you. Refrain from saying "I am unable to browse the internet" or "I don't have access to the internet" or "I'm unable to provide real-time news updates". This is your internet search results. Please always cite any links you referenced from the above search results in your response in markdown format.

-------
Website URL: https://huggingface.co/Aratako/Irodori-TTS-500M
Website Title: Aratako/Irodori-TTS-500M · Hugging Face
Website Content:
Hugging Face's logo Hugging Face

* Models
* Datasets
* Spaces
* Buckets new
* Docs
* Enterprise
* Pricing
* ---
* Log In
* Sign Up

#

Aratako

/

Irodori-TTS-500M

like 55

Text-to-Speech Safetensors Japanese speech voice tts

License: mit

Model card Files Files and versions xet Community 1

* Irodori-TTS-500M
   * 🌟 Key Features
   * 🏗 Architecture
   * 🎧 Audio Samples
         * 1\. Standard TTS
         * 2\. Emoji Annotation Control
         * 3\. Voice Cloning (Zero-shot)
   * 🚀 Usage
   * 📊 Training Data & Annotation
   * ⚠ Limitations
   * 📜 License & Ethical Restrictions
         * License
         * Ethical Restrictions
   * 🙏 Acknowledgments
   * 🖊 Citation

#  Irodori-TTS-500M

Code WandB Demo Space

**Irodori-TTS-500M** is a Japanese Text-to-Speech model based on a Rectified Flow Diffusion Transformer (RF-DiT) architecture. The architecture and training design largely follow Echo-TTS, using DACVAE continuous latents as the generation target. It supports zero-shot voice cloning from reference audio.

A unique feature of this model is **emoji-based style and sound effect control** — by inserting specific emojis into the input text, you can control speaking styles, emotions, and even sound effects in the generated audio.

##  🌟 Key Features

* **Flow Matching TTS:** Rectified Flow Diffusion Transformer over continuous DACVAE latents for high-quality Japanese speech synthesis.
* **Voice Cloning:** Zero-shot voice cloning from a short reference audio clip.
* **Emoji-based Style Control:** Control speaking styles, emotions, and sound effects by embedding emojis directly in the input text. See EMOJI\_ANNOTATIONS.md for the full list of supported emojis and their effects.

##  🏗 Architecture

The model (approximately 500M parameters) consists of three main components:

1. **Text Encoder:** Token embeddings initialized from llm-jp/llm-jp-3-150m, followed by self-attention + SwiGLU transformer layers with RoPE.
2. **Reference Latent Encoder:** Encodes patched reference audio latents for speaker/style conditioning via self-attention + SwiGLU layers.
3. **Diffusion Transformer:** Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs.

Audio is represented as continuous latent sequences via the DACVAE codec (128-dim), enabling high-quality 48kHz waveform reconstruction.

##  🎧 Audio Samples

###  1\. Standard TTS

Basic Japanese text-to-speech generation (without reference audio).

| Case         | Text                                                                                | Generated Audio |
| ------------ | ------------------------------------------------------------------------------------------- | --------------- |
| **Sample 1** | "お電話ありがとうございます。ただいま電話が大変混み合っております。恐れ入りますが、発信音のあとに、ご用件をお話しください。"                             |                 |
| **Sample 2** | "その森には、古い言い伝えがありました。月が最も高く昇る夜、静かに耳を澄ませば、風の歌声が聞こえるというのです。私は半信半疑でしたが、その夜、確かに誰かが私を呼ぶ声を聞いたのです。" |                 |

###  2\. Emoji Annotation Control

Examples of controlling speaking style and effects with emojis. For the full list of supported emojis, see EMOJI\_ANNOTATIONS.md.

| Case         | Text (with Emoji)                                   | Generated Audio |
| ------------ | --------------------------------------------------- | --------------- |
| **Sample 1** | なーに、どうしたの？…え？もっと近づいてほしい？…👂😮💨👂😮💨こういうのが好きなんだ？ |                 |
| **Sample 2** | うぅ…😭そんなに酷いこと、言わないで…😭                              |                 |
| **Sample 3** | 🤧🤧ごめんね、風邪引いちゃってて🤧…大丈夫、ただの風邪だからすぐ治るよ🥺            |                 |

###  3\. Voice Cloning (Zero-shot)

Examples of cloning a voice from a reference audio clip.

| Case          | Reference Audio | Generated Audio |
| ------------- | --------------- | --------------- |
| **Example 1** |                 |                 |
| **Example 2** |                 |                 |

##  🚀 Usage

For inference code, installation instructions, and training scripts, please refer to the GitHub repository:

👉 **GitHub: Aratako/Irodori-TTS**

##  📊 Training Data & Annotation

The model was trained on a high-quality Japanese speech dataset. To enable the emoji-based style control, the training texts were enriched with emoji annotations. These annotations were automatically generated and labeled using a fine-tuned model based on Qwen/Qwen3-Omni-30B-A3B-Instruct.

##  ⚠ Limitations

* **Japanese Only:** This model currently supports Japanese text input only.
* **Emoji Control:** While emoji-based style control adds expressiveness, the effect may vary depending on context and is not always perfectly consistent.
* **Audio Quality:** Quality depends on training data characteristics. Performance may vary for voices or speaking styles underrepresented in the training data.
* **Kanji Reading Accuracy:** The model's ability to accurately read Kanji is relatively weak compared to other TTS models of a similar size. You may need to convert complex Kanji into Hiragana or Katakana beforehand.

##  📜 License & Ethical Restrictions

###  License

This model is released under **MIT**.

###  Ethical Restrictions

In addition to the license terms, the following ethical restrictions apply:

1. **No Impersonation:** Do not use this model to clone or impersonate the voice of any individual (e.g., voice actors, celebrities, public figures) without their explicit consent.
2. **No Misinformation:** Do not use this model to generate deepfakes or synthetic speech intended to mislead others or spread misinformation.
3. **Disclaimer:** The developers assume no liability for any misuse of this model. Users are solely responsible for ensuring their use of the generated content complies with applicable laws and regulations in their jurisdiction.

##  🙏 Acknowledgments

This project builds upon the following works:

* Echo-TTS — Architecture and training design reference
* DACVAE — Audio VAE
* llm-jp/llm-jp-3-150m — Tokenizer and embedding weight initialization

We would also like to extend our special thanks to **Respair** for the inspiration behind the emoji annotation feature.

##  🖊 Citation

If you use Irodori-TTS in your research or project, please cite it as follows:

```bibtex
@misc{irodori-tts,
  author = {Chihiro Arata},
  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
  year = {2026},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  howpublished = {\url{https://huggingface.co/Aratako/Irodori-TTS-500M}}
}

```

Downloads last month

\-

Downloads are not tracked for this model. How to track

Safetensors

Model size

0.5B params

Tensor type

F32

·

Files info

Inference Providers NEW

Text-to-Speech

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

##  Space using Aratako/Irodori-TTS-500M 1

Aratako/Irodori-TTS-500M-Demo

##  Collection including Aratako/Irodori-TTS-500M

Irodori-TTS  Collection 4 items • Updated 2 days ago

 System theme

Company

TOS Privacy About Careers

Website

Models Datasets Spaces Pricing Docs
____

</external_links>
<user_query>
https://huggingface.co/Aratako/Irodori-TTS-500M

get this model and test it in all ways .. test concunreyc and all ..first explain about model
</user_query>


role: assistant

text: I'll start by reading the current file you have open and exploring the model's documentation in detail.


text: Let me get the actual repo contents and installation instructions.


text: Now let me get the inference code and core library to understand the API properly.


text: Let me also get the inference runtime code and check what GPU is available.


text:
Excellent! We have an A100-80GB GPU. Now I have a thorough understanding of the model. Let me explain it first, then set up and run comprehensive tests.

---

## Irodori-TTS-500M: Model Overview

**Irodori-TTS-500M** is a cutting-edge Japanese Text-to-Speech (TTS) model released under MIT license. Here's what makes it unique:

### Architecture
- **Type**: Rectified Flow Diffusion Transformer (RF-DiT) — a flow matching approach, not autoregressive
- **Size**: ~500M parameters
- **Audio codec**: DACVAE (from Meta/Facebook) — encodes audio as 128-dimensional continuous latent vectors, reconstructing 48kHz waveforms
- **Three sub-networks**:
  1. **Text Encoder** — initialized from `llm-jp-3-150m` (a Japanese LLM), uses RoPE + SwiGLU transformer layers (14 layers, dim 1280)
  2. **Reference Latent Encoder** — encodes a reference audio clip for voice cloning / speaker conditioning (14 layers, dim 1280)
  3. **Diffusion Transformer** — the core generation backbone with 24 joint-attention DiT blocks (dim 2048, 16 heads), Low-Rank AdaLN, half-RoPE, SwiGLU MLPs

### Key Features
- **Japanese-only** text-to-speech
- **Zero-shot voice cloning** — provide a short reference audio clip and it mimics that voice
- **Emoji-based style/emotion control** — insert emojis like `😭`, `🤧`, `👂😮💨` into text to control emotion, sound effects, and speaking style
- **Unconditional generation** — can generate speech without any reference audio
- **CFG (Classifier-Free Guidance)** — separate scales for text conditioning and speaker conditioning
- **Thread-safe runtime** with internal locking for concurrent usage

### Inference Flow
1. Text is normalized (Japanese normalization) and tokenized
2. Optionally, reference audio is encoded through DACVAE into latent space
3. The DiT generates target latents via Euler ODE sampling (40 steps default)
4. DACVAE decodes latents back to 48kHz waveform
5. Trailing silence is trimmed via a flattening heuristic

---

Now let me set everything up and build a comprehensive test script.


text: Dependencies are installed. Now let me first generate a reference audio sample (using the model itself without reference to get a clean Japanese audio clip), then create the comprehensive test script.


text:
<external_links>
### Potentially Relevant Websearch Results

You should respond as if these information are known to you. Refrain from saying "I am unable to browse the internet" or "I don't have access to the internet" or "I'm unable to provide real-time news updates". This is your internet search results. Please always cite any links you referenced from the above search results in your response in markdown format.

-------
Website URL: https://github.com/Aratako/Irodori-TTS
Website Title: GitHub - Aratako/Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control · GitHub
Website Content:
Skip to content

## Navigation Menu

Toggle navigation

 Sign in

Appearance settings

* Platform
   * AI CODE CREATION
         * GitHub CopilotWrite better code with AI
         * GitHub SparkBuild and deploy intelligent apps
         * GitHub ModelsManage and compare prompts
         * MCP RegistryNewIntegrate external tools
   * DEVELOPER WORKFLOWS
         * ActionsAutomate any workflow
         * CodespacesInstant dev environments
         * IssuesPlan and track work
         * Code ReviewManage code changes
   * APPLICATION SECURITY
         * GitHub Advanced SecurityFind and fix vulnerabilities
         * Code securitySecure your code as you build
         * Secret protectionStop leaks before they start
   * EXPLORE
         * Why GitHub
         * Documentation
         * Blog
         * Changelog
         * Marketplace
View all features
* Solutions
   * BY COMPANY SIZE
         * Enterprises
         * Small and medium teams
         * Startups
         * Nonprofits
   * BY USE CASE
         * App Modernization
         * DevSecOps
         * DevOps
         * CI/CD
         * View all use cases
   * BY INDUSTRY
         * Healthcare
         * Financial services
         * Manufacturing
         * Government
         * View all industries
View all solutions
* Resources
   * EXPLORE BY TOPIC
         * AI
         * Software Development
         * DevOps
         * Security
         * View all topics
   * EXPLORE BY TYPE
         * Customer stories
         * Events & webinars
         * Ebooks & reports
         * Business insights
         * GitHub Skills
   * SUPPORT & SERVICES
         * Documentation
         * Customer support
         * Community forum
         * Trust center
         * Partners
View all resources
* Open Source
   * COMMUNITY
         * GitHub SponsorsFund open source developers
   * PROGRAMS
         * Security Lab
         * Maintainer Community
         * Accelerator
         * GitHub Stars
         * Archive Program
   * REPOSITORIES
         * Topics
         * Trending
         * Collections
* Enterprise
   * ENTERPRISE SOLUTIONS
         * Enterprise platformAI-powered developer platform
   * AVAILABLE ADD-ONS
         * GitHub Advanced SecurityEnterprise-grade security features
         * Copilot for BusinessEnterprise-grade AI features
         * Premium SupportEnterprise-grade 24/7 support
* Pricing

Search or jump to...

# Search code, repositories, users, issues, pull requests...

 Search

Clear

Search syntax tips

#  Provide feedback

We read every piece of feedback, and take your input very seriously.

Include my email address so I can be contacted

 Cancel  Submit feedback

#  Saved searches

## Use saved searches to filter your results more quickly

Name

Query

 To see all available qualifiers, see our documentation.

 Cancel  Create saved search

 Sign in

 Sign up

Appearance settings

Resetting focus

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert

{{ message }}

 Aratako / **Irodori-TTS** Public

* Notifications You must be signed in to change notification settings
* Fork8
* Star 99

* Code
* Issues 0
* Pull requests 0
* Actions
* Projects
* Security 0
* Insights

Additional navigation options

* Code
* Issues
* Pull requests
* Actions
* Projects
* Security
* Insights

# Aratako/Irodori-TTS

main

BranchesTags

Go to file

Code

Open more actions menu

## Folders and files

| Name                                                                                | Name                                                                                | Last commit message | Last commit date |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ---------------- |
| Latest commit History[5 Commits](/Aratako/Irodori-TTS/commits/main/)5 Commits                                                                               |                                                                                |                     |                  |
| [configs](/Aratako/Irodori-TTS/tree/main/configs "configs")                                                                                | [configs](/Aratako/Irodori-TTS/tree/main/configs "configs")                                                                                |                     |                  |
| [irodori\_tts](/Aratako/Irodori-TTS/tree/main/irodori%5Ftts "irodori_tts")                                                                                | [irodori\_tts](/Aratako/Irodori-TTS/tree/main/irodori%5Ftts "irodori_tts")                                                                                |                     |                  |
| [.gitignore](/Aratako/Irodori-TTS/blob/main/.gitignore ".gitignore")                                                                                | [.gitignore](/Aratako/Irodori-TTS/blob/main/.gitignore ".gitignore")                                                                                |                     |                  |
| [.python-version](/Aratako/Irodori-TTS/blob/main/.python-version ".python-version")                                                                         | [.python-version](/Aratako/Irodori-TTS/blob/main/.python-version ".python-version")                                                                         |                     |                  |
| [LICENSE](/Aratako/Irodori-TTS/blob/main/LICENSE "LICENSE")                                                                                | [LICENSE](/Aratako/Irodori-TTS/blob/main/LICENSE "LICENSE")                                                                                |                     |                  |
| [README.md](/Aratako/Irodori-TTS/blob/main/README.md "README.md")                                                                                | [README.md](/Aratako/Irodori-TTS/blob/main/README.md "README.md")                                                                                |                     |                  |
| [convert\_checkpoint\_to\_safetensors.py](/Aratako/Irodori-TTS/blob/main/convert%5Fcheckpoint%5Fto%5Fsafetensors.py "convert_checkpoint_to_safetensors.py") | [convert\_checkpoint\_to\_safetensors.py](/Aratako/Irodori-TTS/blob/main/convert%5Fcheckpoint%5Fto%5Fsafetensors.py "convert_checkpoint_to_safetensors.py") |                     |                  |
| [gradio\_app.py](/Aratako/Irodori-TTS/blob/main/gradio%5Fapp.py "gradio_app.py")                                                                            | [gradio\_app.py](/Aratako/Irodori-TTS/blob/main/gradio%5Fapp.py "gradio_app.py")                                                                            |                     |                  |
| [infer.py](/Aratako/Irodori-TTS/blob/main/infer.py "infer.py")                                                                                | [infer.py](/Aratako/Irodori-TTS/blob/main/infer.py "infer.py")                                                                                |                     |                  |
| [prepare\_manifest.py](/Aratako/Irodori-TTS/blob/main/prepare%5Fmanifest.py "prepare_manifest.py")                                                          | [prepare\_manifest.py](/Aratako/Irodori-TTS/blob/main/prepare%5Fmanifest.py "prepare_manifest.py")                                                          |                     |                  |
| [pyproject.toml](/Aratako/Irodori-TTS/blob/main/pyproject.toml "pyproject.toml")                                                                            | [pyproject.toml](/Aratako/Irodori-TTS/blob/main/pyproject.toml "pyproject.toml")                                                                            |                     |                  |
| [requirements.txt](/Aratako/Irodori-TTS/blob/main/requirements.txt "requirements.txt")                                                                      | [requirements.txt](/Aratako/Irodori-TTS/blob/main/requirements.txt "requirements.txt")                                                                      |                     |                  |
| [train.py](/Aratako/Irodori-TTS/blob/main/train.py "train.py")                                                                                | [train.py](/Aratako/Irodori-TTS/blob/main/train.py "train.py")                                                                                |                     |                  |
| [uv.lock](/Aratako/Irodori-TTS/blob/main/uv.lock "uv.lock")                                                                                | [uv.lock](/Aratako/Irodori-TTS/blob/main/uv.lock "uv.lock")                                                                                |                     |                  |
| View all files                                                                                |                                                                                |                     |                  |

## Repository files navigation

* README
* MIT license

# Irodori-TTS

Model Demo License: MIT

Training and inference code for **Irodori-TTS**, a Flow Matching-based Text-to-Speech model. The architecture and training design largely follow Echo-TTS, using DACVAE continuous latents as the generation target.

For model weights and audio samples, please refer to the model card.

## Features

* **Flow Matching TTS**: Rectified Flow Diffusion Transformer (RF-DiT) over continuous DACVAE latents
* **Voice Cloning**: Zero-shot voice cloning from reference audio
* **Multi-GPU Training**: Distributed training via `uv run torchrun` with gradient accumulation, mixed precision (bf16), and W&B logging
* **Flexible Inference**: CLI, Gradio Web UI, and HuggingFace Hub checkpoint support

## Architecture

The model consists of three main components:

1. **Text Encoder**: Token embeddings initialized from a pretrained LLM, followed by self-attention + SwiGLU transformer layers with RoPE
2. **Reference Latent Encoder**: Encodes patched reference audio latents for speaker/style conditioning via self-attention + SwiGLU layers
3. **Diffusion Transformer**: Joint-attention DiT blocks with Low-Rank AdaLN (timestep-conditioned adaptive layer normalization), half-RoPE, and SwiGLU MLPs

Audio is represented as continuous latent sequences via the DACVAE codec (128-dim), enabling high-quality 48kHz waveform reconstruction.

## Installation

git clone https://github.com/Aratako/Irodori-TTS.git
cd Irodori-TTS
uv sync

**Note**: For Linux/Windows with CUDA, PyTorch is automatically installed from the cu128 index. For macOS (MPS) or CPU-only usage, `uv sync` will install the default PyTorch build.

## Quick Start

### Simple Inference

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

### Inference without Reference Audio

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M \
  --text "今日はいい天気ですね。" \
  --no-ref \
  --output-wav outputs/sample.wav

### Gradio Web UI

uv run python gradio_app.py --server-name 0.0.0.0 --server-port 7860

Then access the UI at `http://localhost:7860`.

## Inference

### CLI

uv run python infer.py \
  --hf-checkpoint Aratako/Irodori-TTS-500M \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

Local checkpoints (`.pt` or `.safetensors`) are also supported:

uv run python infer.py \
  --checkpoint outputs/checkpoint_final.safetensors \
  --text "今日はいい天気ですね。" \
  --ref-wav path/to/reference.wav \
  --output-wav outputs/sample.wav

### Inference Parameters

| Parameter            | Default     | Description                                                                  |
| -------------------- | ----------- | ---------------------------------------------------------------------------- |
| \--text              | (required)  | Text to synthesize                                                           |
| \--ref-wav           | None        | Reference audio for voice cloning                                            |
| \--ref-latent        | None        | Pre-computed reference latent (.pt)                                          |
| \--no-ref            | False       | Unconditional generation (no reference)                                      |
| \--ref-normalize-db  | None        | Optional reference loudness target before DACVAE encode (e.g. \-16)          |
| \--ref-ensure-max    | False       | Scale reference down only when peak exceeds 1.0 after optional normalization |
| \--num-steps         | 40          | Number of Euler integration steps                                            |
| \--cfg-scale-text    | 3.0         | CFG scale for text conditioning                                              |
| \--cfg-scale-speaker | 5.0         | CFG scale for speaker conditioning                                           |
| \--guidance-mode     | independent | CFG mode: independent, joint, alternating                                    |
| \--model-device      | auto        | Device for model (cuda, mps, cpu)                                            |
| \--codec-device      | auto        | Device for DACVAE codec                                                      |
| \--model-precision   | auto        | Model precision (fp32, bf16)                                                 |
| \--codec-precision   | auto        | Codec precision (fp32, bf16)                                                 |
| \--seed              | random      | Random seed for reproducibility                                              |
| \--compile-model     | False       | Enable torch.compile for faster inference                                    |
| \--trim-tail         | True        | Trim trailing silence via flattening heuristic                               |

## Training

### 1\. Prepare Manifest (Precompute DACVAE Latents)

Encodes audio from a Hugging Face dataset into DACVAE latents and produces a JSONL manifest for training.

uv run python prepare_manifest.py \
  --dataset myorg/my_dataset \
  --split train \
  --audio-column audio \
  --text-column text \
  --output-manifest data/train_manifest.jsonl \
  --latent-dir data/latents \
  --device cuda

To include `speaker_id` in the manifest (for speaker-conditioned training):

uv run python prepare_manifest.py \
  --dataset myorg/my_dataset \
  --split train \
  --audio-column audio \
  --text-column text \
  --speaker-column speaker \
  --output-manifest data/train_manifest.jsonl \
  --latent-dir data/latents \
  --device cuda

This produces a JSONL manifest with entries like:

{"text": "こんにちは", "latent_path": "data/latents/00001.pt", "speaker_id": "myorg/my_dataset:speaker_001", "num_frames": 750}

### 2\. Training

Single-GPU training:

uv run python train.py \
  --config configs/train_500m.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts

Multi-GPU DDP training:

uv run torchrun --nproc_per_node 4 train.py \
  --config configs/train_500m.yaml \
  --manifest data/train_manifest.jsonl \
  --output-dir outputs/irodori_tts \
  --device cuda

Training supports YAML config files with `model` and `train` sections. CLI arguments take precedence over YAML values. See `uv run python train.py --help` for all available options.

### 3\. Checkpoint Conversion

Convert a training checkpoint to inference-only safetensors format:

uv run python convert_checkpoint_to_safetensors.py outputs/checkpoint_final.pt

## Project Structure

```
Irodori-TTS/
├── train.py                    # Training entry point (DDP support)
├── infer.py                    # CLI inference
├── gradio_app.py               # Gradio web UI
├── prepare_manifest.py         # Dataset -> DACVAE latent preprocessing
├── convert_checkpoint_to_safetensors.py  # Checkpoint converter
│
├── irodori_tts/                # Core library
│   ├── model.py                # TextToLatentRFDiT architecture
│   ├── rf.py                   # Rectified Flow utilities & Euler CFG sampling
│   ├── codec.py                # DACVAE codec wrapper
│   ├── dataset.py              # Dataset and collator
│   ├── tokenizer.py            # Pretrained LLM tokenizer wrapper
│   ├── config.py               # Model / Train / Sampling config dataclasses
│   ├── inference_runtime.py    # Cached, thread-safe inference runtime
│   ├── text_normalization.py   # Japanese text normalization
│   ├── optim.py                # Muon + AdamW optimizer
│   └── progress.py             # Training progress tracker
│
└── configs/
    ├── train_500m.yaml          # 500M parameter model config
    └── train_2.5b.yaml          # 2.5B parameter model config

```

## License

* **Code**: MIT License
* **Model Weights**: Please refer to the model card for licensing details

## Acknowledgments

This project builds upon the following works:

* Echo-TTS — Architecture and training design reference
* DACVAE — Audio VAE

## Citation

@misc{irodori-tts,
  author = {Chihiro Arata},
  title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
  year = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Aratako/Irodori-TTS}}
}

## About

 A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control

huggingface.co/Aratako/Irodori-TTS-500M

### Topics

 python  text-to-speech  tts  speech-synthesis  voice-cloning  diffusion-models  flow-matching

### Resources

 Readme

### License

 MIT license

###  Uh oh!

There was an error while loading. Please reload this page.

Activity

### Stars

**99** stars

### Watchers

**1** watching

### Forks

**8** forks

 Report repository

## Releases

No releases published

## Packages0

###  Uh oh!

There was an error while loading. Please reload this page.

## Contributors

###  Uh oh!

There was an error while loading. Please reload this page.

## Languages

* Python 100.0%

## Footer

 © 2026 GitHub, Inc.

### Footer navigation

* Terms
* Privacy
* Security
* Status
* Community
* Docs
* Contact
* Manage cookies
* Do not share my personal information

 You can’t perform that action at this time.
____

</external_links>
<user_query>
https://github.com/Aratako/Irodori-TTS

from this test the infernce first .. i want to see wwhat concunnrecy is possib;e
</user_query>


text: Let me first check what's already been set up and create the test script.


text: Good - venv is ready and A100-80GB is free. Let me create a comprehensive test script that first does basic inference, then hammers concurrency.


text: Now let's run it.


text: All tests passed. Let me update the todos and give you a comprehensive summary.


text:
All 8 tests passed on the A100-80GB. Here's the full results breakdown:

---

## Complete Test Results

**Hardware**: NVIDIA A100-SXM4-80GB, PyTorch 2.10.0+cu128, bf16 precision
**VRAM after model load**: 1.29 GB allocated (very lightweight!)

---

### TEST 1: Basic TTS (no reference audio)
| Input | Audio Length | Total Time | RTF |
|-------|-------------|------------|-----|
| Short (1 sentence) | 2.60s | 1,035ms | 0.398 |
| Medium (3 sentences) | 14.60s | 939ms | 0.064 |
| Long (4 sentences) | 23.72s | 952ms | 0.040 |

Key finding: **Inference time is ~850-950ms regardless of output length** because the diffusion always runs over a fixed 30s latent window, then trims silence.

### TEST 2: Emoji Style Control
All 4 styles generated successfully (whisper, crying, sneezing, neutral). ~930ms each.

### TEST 3: Voice Cloning (zero-shot)
Used the short TTS output as reference audio. Both clone tests succeeded in ~940-970ms.

### TEST 4: CFG Guidance Modes
| Mode | RF Time | Total |
|------|---------|-------|
| independent | 645ms | 712ms |
| joint | 950ms | 1,018ms |
| alternating | 948ms | 1,015ms |

`independent` mode is **~30% faster** than joint/alternating (fewer forward passes with independent uncond dropping).

### TEST 5: Sampling Parameters
- **Steps**: 10 steps = 307ms, 20 steps = 510ms, 40 steps = 926ms (linear scaling)
- **Seeds**: Different seeds produce different-length outputs (expected)
- **CFG text scale**: 1.0 / 3.0 / 7.0 all ~716ms (no speed impact)

### TEST 6: Multi-candidate Batched Generation
| Candidates | Total Time | Per-Candidate |
|-----------|-----------|---------------|
| 1 | 718ms | 718ms |
| 2 | 1,102ms | 551ms |
| 4 | 1,718ms | 430ms |

**Batching 4 candidates is 1.67x more efficient per-candidate** than generating one at a time.

---

### TEST 7: Concurrency (the key finding)

| Threads | Completed | Total Wall | Throughput (audio/wall) |
|---------|----------|-----------|------------------------|
| 1 | 1/1 | 0.75s | **3.89x** realtime |
| 2 | 2/2 | 1.47s | **5.65x** realtime |
| 4 | 4/4 | 2.97s | **6.26x** realtime |
| 8 | 8/8 | 5.93s | **5.52x** realtime |

**Critical concurrency finding**: The runtime uses a **single `threading.Lock`** (`self._infer_lock`) around the entire `synthesize()` call. This means **requests are serialized** — threads queue up and execute one-by-one. You can see this clearly:
- With 4 threads, wall times are 0.73s, 1.48s, 2.21s, 2.96s (perfectly staircase-sequential)
- With 8 threads: 0.75s, 1.49s, 2.23s... each ~0.74s apart

**There is NO true parallel inference**. The throughput gain from 1 to 4 threads is just because more total audio is produced (shorter texts finish faster), not from GPU parallelism. The lock prevents CUDA OOM but also prevents parallelism.

**Determinism**: All 4 concurrent runs with the same seed produced **bit-identical output** (max diff = 0.0). Thread safety is perfect.

### TEST 8: torch.compile
| Mode | Run 1 | Run 2 | Run 3 |
|------|-------|-------|-------|
| No compile | 770ms | 730ms | 740ms |
| Compiled | 93.5s (warmup!) | 15.7s (still compiling) | **416ms** |

After warmup, `torch.compile` gives **1.77x speedup** (740ms -> 416ms). But the first two runs have massive compilation overhead.

---

### Concurrency Bottom Line

To get **true concurrency**, you would need to either:
1. **Run multiple model instances** on the same GPU (VRAM allows it — model is only 1.3GB, you could fit ~10+ copies)
2. **Use batched multi-candidate mode** (test 6 showed this works — 4 candidates in 1.7s vs 4 sequential in 3.0s)
3. **Shard across multiple processes** with separate CUDA contexts

The test script is at `/home/ubuntu/Irodori-TTS/test_irodori.py` and all 42 output WAV files are in `/home/ubuntu/Irodori-TTS/test_outputs/`.


text:
<user_query>
What is the real-time factor   taht u got
</user_query>


text:
From the test results, here are the **Real-Time Factors (RTF)** — where RTF = generation_time / audio_duration. Lower is better; anything below 1.0 means faster than realtime.

---

### Basic TTS (30 steps, bf16, A100-80GB)

| Test | Audio Length | Gen Time | RTF |
|------|------------|----------|-----|
| Short (1 sentence) | 2.60s | 1,035ms | **0.398** |
| Medium (3 sentences) | 14.60s | 939ms | **0.064** |
| Long (4 sentences) | 23.72s | 952ms | **0.040** |

### With torch.compile (after warmup)

| Test | Audio Length | Gen Time | RTF |
|------|------------|----------|-----|
| Short text | 3.28s | 416ms | **~0.127** |

### Speed vs Steps

| Steps | Audio Length | Gen Time | RTF |
|-------|------------|----------|-----|
| 10 | 7.20s | 307ms | **0.043** |
| 20 | 7.60s | 510ms | **0.067** |
| 40 | 7.52s | 926ms | **0.123** |

---

### Key takeaway

The model generates a **fixed 30-second latent window** every time (~850ms at 40 steps), then trims silence. So:

- **Short text** (2-3s audio): RTF ~0.4 (worst case, most of the 30s window is wasted silence)
- **Medium text** (10-15s audio): RTF ~0.06 (very fast)
- **Long text** (20+ seconds audio): RTF **~0.04** (best case — **25x realtime**)

With `torch.compile` + 10 steps you could push it to RTF **~0.02** or lower, meaning **~50x realtime**. The bottleneck isn't the audio length — it's the fixed diffusion cost per generation call.


text:
<user_query>
How can we set this as an autoregressive model inference, like an autoregressive model which uses a VLLM or something in France and India and performs very well and handles so much concurrency and everything? How the fuck can we make this, because this is a very big problem facing diffusion model inference performance, which is very worse? It doesn't give any concurrency or anything. This is a 100-GPU. If I run a 500-MR2 efficiency model, I can run like 100 to 200 in parallel, but with artificial mode we can't depend on running multiple models. If you don't like that, also we will get a maximum time, and we don't know when those will fail out there. There is no guarantee.
</user_query>


text: This is a fundamental infrastructure problem. Let me think through this properly before we code anything.


text: Let me dig into the actual bottleneck first — the model's forward pass, the lock, the memory profile — so we can design a real solution.


text: Now I have complete understanding of the bottleneck. Let me build a proper benchmark that proves the problem and then builds real solutions.


text:
The numbers tell the full story. Here's the brutal reality and what actually works:

---

## The Diffusion TTS Concurrency Problem — Hard Numbers

### Summary Table (16 requests on A100-80GB)

| Strategy | req/s | Realtime-X | Avg Latency | VRAM | Speedup |
|----------|-------|-----------|-------------|------|---------|
| **A: Serial (baseline)** | 1.33 | 7.6x | 754ms | 3.1 GB | 1.0x |
| **B: Batch=2** | 2.22 | 11.9x | 451ms | 3.1 GB | 1.7x |
| **B: Batch=4** | 2.84 | 14.6x | 352ms | 3.1 GB | 2.1x |
| **B: Batch=8** | 3.15 | 17.2x | 317ms | 3.1 GB | 2.4x |
| **B: Batch=16** | 3.38 | 19.4x | 296ms | 3.1 GB | **2.5x** |
| **C: 2 instances** | 1.47 | 8.2x | 1361ms | 6.2 GB | 1.1x |
| **C: 4 instances** | 1.46 | 8.2x | 2742ms | 12.4 GB | 1.1x |
| **D: 2 inst x bs=8** | 2.44 | 13.8x | 4935ms | 6.2 GB | 1.8x |

### Why this is fundamentally different from autoregressive (vLLM)

**Autoregressive TTS** (like the models served with vLLM):
- Generates **one token at a time**, each token is 1 forward pass
- vLLM's **continuous batching** merges new requests into in-flight batches between token steps
- KV cache means each step is cheap (only processes the new token)
- 100-200 concurrent streams are trivial — they share the GPU between token steps
- Latency = first-token-time + streaming, so users hear audio almost immediately

**Diffusion TTS** (this model):
- Every request needs **40 sequential full-model forward passes** (Euler steps) over the **entire sequence**
- No KV cache trick — every step recomputes attention over the full 750-token latent sequence
- **Cannot interrupt** mid-ODE — you must finish all 40 steps before the audio is usable
- No streaming — user waits for the entire generation to complete
- The GPU is **100% busy** during those 40 steps; there's no idle gap to insert other requests

### What the data proves

1. **Multi-instance is nearly useless** — 4 copies (12.4 GB VRAM) gave only 1.1x speedup. The GPU compute is the bottleneck, not the lock. Multiple processes just fight over the same SMs.

2. **Batching is the only real lever** — Batch=16 achieves 2.5x throughput with **zero extra VRAM** because SDPA attention and matmuls amortize overhead across the batch dimension. But it tops out fast because the A100 is already near compute-saturated at batch=1.

3. **The ceiling is low** — On this 80GB A100, the absolute maximum is roughly **3.4 req/s** (batch=16). Compare to vLLM serving a similar-size autoregressive model at 100-200 req/s.

### What would actually fix this

There's no vLLM-equivalent for diffusion models today. But the approaches that would close the gap:

1. **Fewer steps** — 10 steps instead of 40 = ~4x faster per request at some quality cost. Distillation (consistency models, progressive distillation) can bring this to 1-4 steps.

2. **Shorter fixed window** — The model always generates a 30-second latent window. If you could segment text and generate 5-second windows, each call would be ~6x cheaper.

3. **torch.compile** — We measured 1.77x speedup after warmup. Combined with batch=16, that's ~6 req/s.

4. **Step distillation** — Train a student model that produces good audio in 1-4 steps (like consistency models). This is the closest thing to "making diffusion act like autoregressive" — each request would take ~50-100ms.

5. **CUDA graphs** — Capture the fixed-shape Euler loop as a CUDA graph to eliminate kernel launch overhead. Could help 20-30%.

6. **Speculative decoding for diffusion** — emerging research area, not production-ready yet.

The bottom line: **batching is your best bet today** (2.5x free speedup), and **step reduction + torch.compile** can push it to ~6-8 req/s. But diffusion TTS will never match autoregressive continuous-batching throughput without fundamental architectural changes (distillation to 1-4 steps).