Research the GitHub repository https://github.com/Scicom-AI-Enterprise-Organization/Multilingual-TTS (main branch). I need to answer these questions:

1. Is the model autoregressive or non-autoregressive?
2. What is the RTF (Real-Time Factor) mentioned?
3. What GPU concurrency info is provided?
4. What languages does it support?
5. Is it finetuneable on new languages?

Use WebFetch to read the README and any key source files. Look at the model architecture code if available. Provide detailed findings with evidence from the repo.


I'll research this repository by fetching the README and exploring key source files.


ToolSearch

Question: select:WebFetch


Let me fetch the README and explore key parts of the repository.


WebFetch

Prompt: Extract the full README content. I need: 1) Whether the model is autoregressive or non-autoregressive, 2) Any RTF (Real-Time Factor) numbers, 3) GPU concurrency info, 4) Supported languages, 5) Whether it's finetuneable on new languages. Also list the directory structure and any key source files mentioned.


Prompt: List the full directory structure and all files in the repository. I need to understand the project layout to find model architecture code, configuration files, and documentation.


There's an issue with the selected model (claude-haiku-4-5@20251001). It may not exist or you may not have access to it. Run /model to pick a different model.


Let me try using the `gh` CLI instead, as recommended for GitHub URLs.


Bash

gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/readme --jq '.content' | base64 -d 2>/dev/null || gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/readme -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch README from the GitHub repo


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/git/trees/main?recursive=1 --jq '.tree[].path' 2>/dev/null

List all files in the repo


# Multilingual-TTS

Open-source multilingual TTS with Voice Cloning support for 150+ languages, built on [Neucodec](https://github.com/neuphonic/neucodec) as the speech tokenizer at 50 TPS.

## Models

| Model | Link | Purpose |
|---|---|---|
| Multilingual-TTS-0.6B-Base | [🤗](https://huggingface.co/Scicom-intl/Multilingual-TTS-0.6B-Base) | Base |
| Multilingual-TTS-1.7B-Base | [🤗](https://huggingface.co/Scicom-intl/Multilingual-TTS-1.7B-Base) | Base |
| Multilingual-Expressive-TTS-0.6B | [🤗](https://huggingface.co/Scicom-intl/Multilingual-Expressive-TTS-0.6B) | Post-training TTS |
| Multilingual-Expressive-TTS-1.7B | [🤗](https://huggingface.co/Scicom-intl/Multilingual-Expressive-TTS-1.7B) | Post-training TTS |

## Evaluation

### [TTS Evaluation](tts-evaluation/README.md)

CER and MOS across 76 languages, compared against: Dia TTS, Orpheus, Chatterbox (23 languages), Fish Audio S2 Pro, Qwen3 TTS (11 languages).

### [VC Evaluation](vc-evaluation/README.md)

Speaker similarity and CER across 76 languages, compared against: Dia TTS, Orpheus, Chatterbox (23 languages), Fish Audio S2 Pro.

## Dataset

### Base

**Sources**

1. https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS
2. https://huggingface.co/datasets/Scicom-intl/Emilia-YODAS-Voice-Conversion
3. https://huggingface.co/datasets/Scicom-intl/Malaysian-Emilia

**Size**

1. Multi-speaker multilingual Voice Cloning — **up to 35.88B tokens**
2. Multi-speaker multilingual TTS, 150+ languages — **up to 25.35B tokens**

**Preparation:** [preparation](preparation)

### Expressive TTS

Natural-language descriptions of speech audio generated by computing acoustic statistics and categories, then prompting an LLM to summarise them.

**Sources**

1. https://huggingface.co/datasets/Scicom-intl/ExpressiveSpeech

**Size**

1. Multi-speaker multilingual Expressive TTS — **up to 1.15B tokens**

**Pipeline**

```
audio + transcript
      │
      ▼
speech_categories.py   ← categories (emotion, gender, fluency, accent, quality)
      │                   speech_stats_func  ← pitch, SNR, speaking rate, SQUIM
      ▼
calculate_bins.py      ← bin continuous stats, merge into a single parquet
      │
      ▼
synthetic.py           ← LLM generates a natural-language description per row
```

Uses [Qwen/Qwen2.5-72B-Instruct](https://deepinfra.com/Qwen/Qwen2.5-72B-Instruct) via DeepInfra by default. Any OpenAI-compatible endpoint works — set `API_KEY`, `BASE_URL`, and `MODEL_NAME`.

**Preparation:** [synthetic-description](synthetic-description)

## Ablation

### One Epoch

1. Global token size: 10240 × 256 × 8 GPUs ≈ 20,971,520 tokens
2. Warmup: 100 steps
3. FP32-BF16 mixed precision
4. Compared AdamW with WSD LR vs Muon + AdamW with WSD LR (decay = 10% of dataset)
5. Run on Qwen3 1.7B Base only
6. **AdamW performed better**

### Hyperparameter Search

One-epoch results used conservative learning rates, so we ran a focused search:

1. Global token size: 10240 × 256 × 8 GPUs ≈ 20,971,520 tokens
2. 100 steps, warmup 50 steps
3. FP32-BF16 mixed precision
4. Run on Qwen3 1.7B Base only
5. Grid search over AdamW LR, Muon LR, and decay rate — [hyperparameter_search.py](hyperparameter_search.py)
6. Aggressive LR sweep — [hyperparameter_search_extra.py](hyperparameter_search_extra.py) — turned out best
7. AdamW-only with the same aggressive LR — [1.7B-adamw-aggresive.sh](1.7B-adamw-aggresive.sh)
8. **Adding Muon performed better**

## Training

### Base

```bash
# 0.6B
bash 0.6B.sh

# 1.7B
bash 1.7B.sh
```

### Expressive TTS

```bash
# 0.6B
bash 0.6B-expressive.sh

# 1.7B
bash 1.7B-expressive.sh
```

## WandB

- Base experiments: https://wandb.ai/aies-scicom-scicom-ai/Multilingual-TTS
- Post TTS experiments: https://wandb.ai/aies-scicom-scicom-ai/Multilingual-TTS-Expressive

## Optimize Inference

For better concurrency, you can use https://github.com/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec

## Acknowledgements

1. [Malaysia-AI](https://huggingface.co/malaysia-ai) for the large-scale TTS dataset: https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS
2. [Scitix](https://www.scitix.ai/) for H100 node access


.gitignore
0.6B-expressive.sh
0.6B-vc.sh
0.6B.sh
1.7B-adamw-aggresive.sh
1.7B-adamw-old.sh
1.7B-adamw-wsdlr.sh
1.7B-expressive.sh
1.7B-muonadamw-wsdlr.sh
1.7B-vc.sh
1.7B.sh
4B.sh
README.md
hyperparameter-search.png
hyperparameter_search.py
hyperparameter_search_extra.py
one-epoch.png
preparation
preparation/combine-multipacking-expressive.ipynb
preparation/combine-multipacking-tts.ipynb
preparation/multipacking-emilia-yodas.ipynb
preparation/multipacking-expressivetts.ipynb
preparation/multipacking-malaysian-chinese-emilia.ipynb
preparation/multipacking-malaysian-emilia-dialects.ipynb
preparation/multipacking-malaysian-emilia.ipynb
preparation/multipacking-malaysian-tamil-emilia.ipynb
preparation/multipacking-tts.ipynb
qwen3_adamw.py
qwen3_muonadamw.py
qwen3_muonadamw_post.py
qwen3_muonadamw_search.py
synthetic-description
synthetic-description/README.md
synthetic-description/calculate_bins.py
synthetic-description/download.sh
synthetic-description/process.sh
synthetic-description/sampling.py
synthetic-description/sampling_folder.py
synthetic-description/speech_categories.py
synthetic-description/speech_categories_func.py
synthetic-description/speech_stats_func.py
synthetic-description/synthetic.py
synthetic-description/trim_audio.py
synthetic-description/upload-synthetic.ipynb
tts-evaluation
tts-evaluation/README.md
tts-evaluation/benchmark_results.png
tts-evaluation/calculate_cer.py
tts-evaluation/calculate_mos.py
tts-evaluation/chatterbox.py
tts-evaluation/common-voice-whisper-mapping.json
tts-evaluation/dia_tts.py
tts-evaluation/fishspeech2.py
tts-evaluation/multilingual_tts.py
tts-evaluation/orpheus.py
tts-evaluation/plot_results.py
tts-evaluation/qwen3_tts.py
tts-evaluation/requirements.txt
tts-evaluation/scatter_results.png
tts-evaluation/src
tts-evaluation/src/chatterbox
tts-evaluation/src/chatterbox/__init__.py
tts-evaluation/src/chatterbox/models
tts-evaluation/src/chatterbox/models/__init__.py
tts-evaluation/src/chatterbox/models/s3gen
tts-evaluation/src/chatterbox/models/s3gen/__init__.py
tts-evaluation/src/chatterbox/models/s3gen/configs.py
tts-evaluation/src/chatterbox/models/s3gen/const.py
tts-evaluation/src/chatterbox/models/s3gen/decoder.py
tts-evaluation/src/chatterbox/models/s3gen/f0_predictor.py
tts-evaluation/src/chatterbox/models/s3gen/flow.py
tts-evaluation/src/chatterbox/models/s3gen/flow_matching.py
tts-evaluation/src/chatterbox/models/s3gen/hifigan.py
tts-evaluation/src/chatterbox/models/s3gen/matcha
tts-evaluation/src/chatterbox/models/s3gen/matcha/decoder.py
tts-evaluation/src/chatterbox/models/s3gen/matcha/flow_matching.py
tts-evaluation/src/chatterbox/models/s3gen/matcha/text_encoder.py
tts-evaluation/src/chatterbox/models/s3gen/matcha/transformer.py
tts-evaluation/src/chatterbox/models/s3gen/s3gen.py
tts-evaluation/src/chatterbox/models/s3gen/transformer
tts-evaluation/src/chatterbox/models/s3gen/transformer/__init__.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/activation.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/attention.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/convolution.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/embedding.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/encoder_layer.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/positionwise_feed_forward.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/subsampling.py
tts-evaluation/src/chatterbox/models/s3gen/transformer/upsample_encoder.py
tts-evaluation/src/chatterbox/models/s3gen/utils
tts-evaluation/src/chatterbox/models/s3gen/utils/class_utils.py
tts-evaluation/src/chatterbox/models/s3gen/utils/mask.py
tts-evaluation/src/chatterbox/models/s3gen/utils/mel.py
tts-evaluation/src/chatterbox/models/s3gen/xvector.py
tts-evaluation/src/chatterbox/models/s3tokenizer
tts-evaluation/src/chatterbox/models/s3tokenizer/__init__.py
tts-evaluation/src/chatterbox/models/s3tokenizer/s3tokenizer.py
tts-evaluation/src/chatterbox/models/t3
tts-evaluation/src/chatterbox/models/t3/__init__.py
tts-evaluation/src/chatterbox/models/t3/inference
tts-evaluation/src/chatterbox/models/t3/inference/alignment_stream_analyzer.py
tts-evaluation/src/chatterbox/models/t3/inference/t3_hf_backend.py
tts-evaluation/src/chatterbox/models/t3/llama_configs.py
tts-evaluation/src/chatterbox/models/t3/modules
tts-evaluation/src/chatterbox/models/t3/modules/cond_enc.py
tts-evaluation/src/chatterbox/models/t3/modules/learned_pos_emb.py
tts-evaluation/src/chatterbox/models/t3/modules/perceiver.py
tts-evaluation/src/chatterbox/models/t3/modules/t3_config.py
tts-evaluation/src/chatterbox/models/t3/t3.py
tts-evaluation/src/chatterbox/models/tokenizers
tts-evaluation/src/chatterbox/models/tokenizers/__init__.py
tts-evaluation/src/chatterbox/models/tokenizers/tokenizer.py
tts-evaluation/src/chatterbox/models/utils.py
tts-evaluation/src/chatterbox/models/voice_encoder
tts-evaluation/src/chatterbox/models/voice_encoder/__init__.py
tts-evaluation/src/chatterbox/models/voice_encoder/config.py
tts-evaluation/src/chatterbox/models/voice_encoder/melspec.py
tts-evaluation/src/chatterbox/models/voice_encoder/voice_encoder.py
tts-evaluation/src/chatterbox/mtl_tts.py
tts-evaluation/src/chatterbox/tts.py
tts-evaluation/src/chatterbox/vc.py
vc-evaluation
vc-evaluation/README.md
vc-evaluation/benchmark_results.png
vc-evaluation/calculate_cer.py
vc-evaluation/calculate_similarity.py
vc-evaluation/chatterbox.py
vc-evaluation/common-voice-whisper-mapping.json
vc-evaluation/dia_tts.py
vc-evaluation/fishspeech2.py
vc-evaluation/gather-dataset.ipynb
vc-evaluation/install-neucodec.sh
vc-evaluation/multilingual_tts.py
vc-evaluation/orpheus.py
vc-evaluation/plot_results.py
vc-evaluation/qwen_tts.py
vc-evaluation/requirements.txt
vc-evaluation/scatter_results.png
vc-evaluation/src
vc-evaluation/src/chatterbox
vc-evaluation/src/chatterbox/__init__.py
vc-evaluation/src/chatterbox/models
vc-evaluation/src/chatterbox/models/__init__.py
vc-evaluation/src/chatterbox/models/s3gen
vc-evaluation/src/chatterbox/models/s3gen/__init__.py
vc-evaluation/src/chatterbox/models/s3gen/configs.py
vc-evaluation/src/chatterbox/models/s3gen/const.py
vc-evaluation/src/chatterbox/models/s3gen/decoder.py
vc-evaluation/src/chatterbox/models/s3gen/f0_predictor.py
vc-evaluation/src/chatterbox/models/s3gen/flow.py
vc-evaluation/src/chatterbox/models/s3gen/flow_matching.py
vc-evaluation/src/chatterbox/models/s3gen/hifigan.py
vc-evaluation/src/chatterbox/models/s3gen/matcha
vc-evaluation/src/chatterbox/models/s3gen/matcha/decoder.py
vc-evaluation/src/chatterbox/models/s3gen/matcha/flow_matching.py
vc-evaluation/src/chatterbox/models/s3gen/matcha/text_encoder.py
vc-evaluation/src/chatterbox/models/s3gen/matcha/transformer.py
vc-evaluation/src/chatterbox/models/s3gen/s3gen.py
vc-evaluation/src/chatterbox/models/s3gen/transformer
vc-evaluation/src/chatterbox/models/s3gen/transformer/__init__.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/activation.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/attention.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/convolution.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/embedding.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/encoder_layer.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/positionwise_feed_forward.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/subsampling.py
vc-evaluation/src/chatterbox/models/s3gen/transformer/upsample_encoder.py
vc-evaluation/src/chatterbox/models/s3gen/utils
vc-evaluation/src/chatterbox/models/s3gen/utils/class_utils.py
vc-evaluation/src/chatterbox/models/s3gen/utils/mask.py
vc-evaluation/src/chatterbox/models/s3gen/utils/mel.py
vc-evaluation/src/chatterbox/models/s3gen/xvector.py
vc-evaluation/src/chatterbox/models/s3tokenizer
vc-evaluation/src/chatterbox/models/s3tokenizer/__init__.py
vc-evaluation/src/chatterbox/models/s3tokenizer/s3tokenizer.py
vc-evaluation/src/chatterbox/models/t3
vc-evaluation/src/chatterbox/models/t3/__init__.py
vc-evaluation/src/chatterbox/models/t3/inference
vc-evaluation/src/chatterbox/models/t3/inference/alignment_stream_analyzer.py
vc-evaluation/src/chatterbox/models/t3/inference/t3_hf_backend.py
vc-evaluation/src/chatterbox/models/t3/llama_configs.py
vc-evaluation/src/chatterbox/models/t3/modules
vc-evaluation/src/chatterbox/models/t3/modules/cond_enc.py
vc-evaluation/src/chatterbox/models/t3/modules/learned_pos_emb.py
vc-evaluation/src/chatterbox/models/t3/modules/perceiver.py
vc-evaluation/src/chatterbox/models/t3/modules/t3_config.py
vc-evaluation/src/chatterbox/models/t3/t3.py
vc-evaluation/src/chatterbox/models/tokenizers
vc-evaluation/src/chatterbox/models/tokenizers/__init__.py
vc-evaluation/src/chatterbox/models/tokenizers/tokenizer.py
vc-evaluation/src/chatterbox/models/utils.py
vc-evaluation/src/chatterbox/models/voice_encoder
vc-evaluation/src/chatterbox/models/voice_encoder/__init__.py
vc-evaluation/src/chatterbox/models/voice_encoder/config.py
vc-evaluation/src/chatterbox/models/voice_encoder/melspec.py
vc-evaluation/src/chatterbox/models/voice_encoder/voice_encoder.py
vc-evaluation/src/chatterbox/mtl_tts.py
vc-evaluation/src/chatterbox/tts.py
vc-evaluation/src/chatterbox/vc.py
vc-rl
vc-rl/grpo_async_vllm.py
vc-tts
vc-tts/README.md
vc-tts/permutate-tts-speaker.ipynb


Now let me look at the model architecture code and the TTS-API-Neucodec repo for RTF/concurrency info.


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/contents/qwen3_muonadamw.py -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch the main training script


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/contents/tts-evaluation/multilingual_tts.py -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch the TTS inference/evaluation script


gh api repos/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec/readme -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch the TTS-API-Neucodec README for RTF and concurrency info


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/contents/tts-evaluation/README.md -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch TTS evaluation README


#!/usr/bin/env python
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.

Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=text-generation
"""
# You can also adapt this script on your own causal language modeling
# task. Pointers for this are left as comments.

import torch
from torch import nn
import torch.nn.functional as F
import torch.nn.init as init

import logging
import math
import os
import sys
import warnings
from dataclasses import dataclass, field
from itertools import chain
from typing import Optional

import transformers
import random
from transformers import (
    CONFIG_MAPPING,
    MODEL_FOR_CAUSAL_LM_MAPPING,
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
    AddedToken,
    HfArgumentParser,
    Trainer,
    TrainingArguments,
    default_data_collator,
    DataCollatorWithPadding,
    DataCollatorForLanguageModeling,
    set_seed,
    get_wsd_schedule,
)
from transformers.testing_utils import CaptureLogger
from transformers.trainer_utils import get_last_checkpoint
from transformers import Qwen3ForCausalLM
import json
import numpy as np
from streaming import LocalDataset
from streaming.base.format.mds.encodings import Encoding, _encodings
from cut_cross_entropy import linear_cross_entropy
from liger_kernel.transformers import apply_liger_kernel_to_qwen3, LigerFusedLinearCrossEntropyLoss

torch.serialization.add_safe_globals([np.core.multiarray._reconstruct])

apply_liger_kernel_to_qwen3(
    rope=True,
    swiglu=True,
    rms_norm=True,
    cross_entropy=False,
    fused_linear_cross_entropy=False,
)

logger = logging.getLogger(__name__)

MODEL_CONFIG_CLASSES = list(MODEL_FOR_CAUSAL_LM_MAPPING.keys())
MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)

@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune, or train from scratch.
    """

    model_name_or_path: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "The model checkpoint for weights initialization.Don't set if you want to train a model from scratch."
            )
        },
    )
    model_type: Optional[str] = field(
        default=None,
        metadata={
            "help": "If training from scratch, pass a model type from the list: " +
            ", ".join(MODEL_TYPES)},
    )
    config_overrides: Optional[str] = field(
        default=None, metadata={
            "help": (
                "Override some existing default config settings when a model is trained from scratch. Example: "
                "n_embd=10,resid_pdrop=0.2,scale_attn_weights=false,summary_type=cls_index")}, )
    config_name: Optional[str] = field(
        default=None, metadata={
            "help": "Pretrained config name or path if not the same as model_name"})
    tokenizer_name: Optional[str] = field(
        default=None, metadata={
            "help": "Pretrained tokenizer name or path if not the same as model_name"})
    cache_dir: Optional[str] = field(
        default=None, metadata={
            "help": "Where do you want to store the pretrained models downloaded from huggingface.co"}, )
    use_fast_tokenizer: bool = field(
        default=True, metadata={
            "help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."}, )
    model_revision: str = field(
        default="main", metadata={
            "help": "The specific model version to use (can be a branch name, tag name or commit id)."}, )
    token: str = field(
        default=None,
        metadata={
            "help": (
                "The token to use as HTTP bearer authorization for remote files. If not specified, will use the token "
                "generated when running `huggingface-cli login` (stored in `~/.huggingface`)."
            )
        },
    )
    use_auth_token: bool = field(
        default=None,
        metadata={
            "help": "The `use_auth_token` argument is deprecated and will be removed in v4.34. Please use `token`."
        },
    )
    trust_remote_code: bool = field(
        default=False, metadata={
            "help": (
                "Whether or not to allow for custom models defined on the Hub in their own modeling files. This option"
                "should only be set to `True` for repositories you trust and in which you have read the code, as it will"
                "execute code present on the Hub on your local machine.")}, )
    torch_dtype: Optional[str] = field(
        default=None,
        metadata={
            "help": (
                "Override the default `torch.dtype` and load the model under this dtype. If `auto` is passed, the "
                "dtype will be automatically derived from the model's weights."),
            "choices": [
                "auto",
                "bfloat16",
                "float16",
                "float32"],
        },
    )
    low_cpu_mem_usage: bool = field(
        default=False,
        metadata={
            "help": (
                "It is an option to create the model as an empty shell, then only materialize its parameters when the pretrained weights are loaded."
                "set True will benefit LLM loading time and RAM consumption."
            )
        },
    )

    def __post_init__(self):
        if self.config_overrides is not None and (
                self.config_name is not None or self.model_name_or_path is not None):
            raise ValueError(
                "--config_overrides can't be used in combination with --config_name or --model_name_or_path"
            )

@dataclass
class DataTrainingArguments:
    """
    Arguments pertaining to what data we are going to input our model for training and eval.
    """

    train_file: Optional[str] = field(
        default=None, metadata={
            "help": "The input training data file (a text file)."})
    block_size: Optional[int] = field(
        default=None,
        metadata={
            "help": (
                "Optional input sequence length after tokenization. "
                "The training dataset will be truncated in block of this size for training. "
                "Default to the model max input length for single sentence inputs (take into account special tokens)."
            )
        },
    )

class Model(Qwen3ForCausalLM):
    def __init__(self, config):
        super().__init__(config)
        self.loss = LigerFusedLinearCrossEntropyLoss(reduction="sum")

    def forward(self, input_ids, attention_mask=None, position_ids=None, labels=None, num_items_in_batch=None, **kwargs):
        super_out = self.model.forward(
            input_ids = input_ids,
            position_ids = position_ids,
            attention_mask = attention_mask,
            output_hidden_states = True,
            **kwargs,
        )
        if labels is not None:
            embeddings = super_out.last_hidden_state
            embeddings = embeddings[:,:-1].reshape(-1, embeddings.shape[-1])
            labels = labels[..., 1:].contiguous()
            labels = labels.reshape(-1)
            loss = self.loss(self.lm_head.weight, embeddings, labels)
            num_items_in_batch = num_items_in_batch.to(loss.device)
            loss = loss / num_items_in_batch
            return {'loss': loss}
        return super_out

class MuonPlusAdamW(torch.optim.Optimizer):
    """
    Hybrid optimizer: Muon for hidden layer 2D weights, AdamW for everything else.
    Based on Moonshot AI's "Muon is Scalable for LLM Training" (arXiv:2502.16982)
    """

    def __init__(
        self,
        params,
        lr: float = 1e-4,
        # Muon hyperparameters
        muon_lr: Optional[float] = 1e-4,
        muon_momentum: float = 0.95,
        muon_weight_decay: float = 0.1,
        muon_nesterov: bool = True,
        muon_ns_steps: int = 5,
        # AdamW hyperparameters
        # make sure consistent with https://huggingface.co/docs/transformers/v4.56.2/en/main_classes/trainer#transformers.TrainingArguments
        adamw_betas: tuple = (0.9, 0.999),
        adamw_weight_decay: float = 0.1,
        adamw_eps: float = 1e-8,
        # Parameter filtering
        embed_patterns: tuple = ('embed', 'wte', 'wpe'),
        head_patterns: tuple = ('lm_head', 'head', 'output'),
    ):
        """
        Args:
            params: Model parameters
            lr: Base learning rate (used for AdamW, Muon uses muon_lr)
            muon_lr: Learning rate for Muon (default 1e-4)
            muon_momentum: Momentum for Muon
            muon_weight_decay: Weight decay for Muon (critical for scaling!)
            muon_nesterov: Use Nesterov momentum
            muon_ns_steps: Newton-Schulz iteration steps
            adamw_betas: Adam betas
            adamw_weight_decay: Weight decay for AdamW
            adamw_eps: Adam epsilon
            embed_patterns: Name patterns for embedding layers (use AdamW)
            head_patterns: Name patterns for output head layers (use AdamW)
        """
        if lr <= 0:
            raise ValueError("lr must be positive")

        # Convert to list of (name, param) if needed
        params = list(params)

        if params and isinstance(params[0], tuple):
            named_params = params
        else:
            named_params = [('', p) for p in params]

        muon_params, muon_params_name = [], []
        adamw_params, adamw_params_name = [], []

        embed_patterns: tuple = ('embed', 'wte', 'wpe')
        head_patterns: tuple = ('lm_head', 'head', 'output')

        for name, p in named_params:
            if not p.requires_grad:
                continue

            name_lower = name.lower()

            # Check if this is an embedding or head layer
            is_embed = any(pattern in name_lower for pattern in embed_patterns)
            is_head = any(pattern in name_lower for pattern in head_patterns)

            # Muon: only for 2D hidden layer weights (not embed/head)
            if p.ndim == 2 and not is_embed and not is_head:
                muon_params.append(p)
                muon_params_name.append(name)
            else:
                adamw_params.append(p)
                adamw_params_name.append(name)

        print('muon_params_name', muon_params_name)
        print('adamw_params_name', adamw_params_name)

        # Default Muon LR is 0.02 (paper recommendation)
        effective_muon_lr = muon_lr if muon_lr is not None else 0.02

        param_groups = [
            {"params": muon_params, "type": "muon", "lr": effective_muon_lr},
            {"params": adamw_params, "type": "adamw", "lr": lr},
        ]

        defaults = {"lr": lr}
        self._muon = None
        self._adamw = None

        super().__init__(param_groups, defaults)

        # Store param counts for logging
        self.muon_param_count = sum(p.numel() for p in muon_params)
        self.adamw_param_count = sum(p.numel() for p in adamw_params)

        # Now create the actual sub-optimizers
        if muon_params:
            self._muon = torch.optim.Muon(
                muon_params,
                lr=effective_muon_lr,
                momentum=muon_momentum,
                weight_decay=muon_weight_decay,
                nesterov=muon_nesterov,
                ns_steps=muon_ns_steps,
            )

        if adamw_params:
            self._adamw = torch.optim.AdamW(
                adamw_params,
                lr=lr,
                betas=adamw_betas,
                weight_decay=adamw_weight_decay,
                eps=adamw_eps,
            )

    def __repr__(self):
        return (
            f"MuonPlusAdamW(\n"
            f"  muon_params: {self.muon_param_count:,}\n"
            f"  adamw_params: {self.adamw_param_count:,}\n"
            f")"
        )

    @torch.no_grad()
    def step(self, closure = None):
        loss = None
        if closure is not None:
            with torch.enable_grad():
                loss = closure()

        if self._adamw is not None:
            self._adamw.step()
        if self._muon is not None:
            self._muon.step()

        return loss

    def zero_grad(self, set_to_none: bool = True):
        if self._adamw is not None:
            self._adamw.zero_grad(set_to_none)
        if self._muon is not None:
            self._muon.zero_grad(set_to_none)

    def state_dict(self):
        """Return combined state dict."""
        return {
            'muon': self._muon.state_dict() if self._muon else None,
            'adamw': self._adamw.state_dict() if self._adamw else None,
        }

    def load_state_dict(self, state_dict):
        """Load combined state dict."""
        if self._muon is not None and state_dict.get('muon'):
            self._muon.load_state_dict(state_dict['muon'])
        if self._adamw is not None and state_dict.get('adamw'):
            self._adamw.load_state_dict(state_dict['adamw'])

    @property
    def param_groups(self):
        """Return combined param groups from both optimizers."""
        # During __init__, before sub-optimizers are created, fall back to stored groups
        if not hasattr(self, '_muon'):
            # Access the underlying attribute directly during initialization
            return self.__dict__.get('param_groups', [])

        if self._muon is None and self._adamw is None:
            return self.__dict__.get('param_groups', [])

        groups = []
        if self._muon is not None:
            groups.extend(self._muon.param_groups)
        if self._adamw is not None:
            groups.extend(self._adamw.param_groups)
        return groups

    @param_groups.setter
    def param_groups(self, value):
        # This is needed for compatibility but we manage param_groups internally
        pass

def main():

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        model_args, data_args, training_args = parser.parse_json_file(
            json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

    if training_args.should_log:
        transformers.utils.logging.set_verbosity_info()

    log_level = training_args.get_process_log_level()
    logger.setLevel(log_level)
    transformers.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()

    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}" +
        f"distributed training: {training_args.parallel_mode.value == 'distributed'}, 16-bits training: {training_args.fp16}")
    logger.info(f"Training/evaluation parameters {training_args}")

    last_checkpoint = None
    if os.path.isdir(
            training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)

    set_seed(training_args.seed)

    tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path)
    extra = [AddedToken('<|speech_start|>')]
    for i in range(65536):
        extra.append(AddedToken(f'<|s_{i}|>'))
    tokenizer.add_tokens(extra)

    torch_dtype = (
        model_args.torch_dtype
        if model_args.torch_dtype in ["auto", None]
        else getattr(torch, model_args.torch_dtype)
    )
    min_dtype = torch.finfo(torch_dtype).min
    sequence_length = data_args.block_size

    class UInt32(Encoding):
        def encode(self, obj) -> bytes:
            return obj.tobytes()

        def decode(self, data: bytes):
            return np.frombuffer(data, np.uint32)

    _encodings['uint32'] = UInt32

    class DatasetFixed(torch.utils.data.Dataset):
        def __init__(self, local):
            self.dataset = LocalDataset(local=local)

        def __getitem__(self, idx):
            data = self.dataset[idx]
            data.pop('audio', None)
            data.pop('text', None)
            data.pop('token_type_ids', None)

            if data['attention_mask'].max() > sequence_length:
                print(data)
                return

            for k in data.keys():
                data[k] = data[k].astype(np.int64)

            return data

        def __len__(self):
            return len(self.dataset)

    model = Model.from_pretrained(
        model_args.model_name_or_path,
        attn_implementation = 'kernels-community/vllm-flash-attn3',
        torch_dtype = model_args.torch_dtype,
    )
    model.resize_token_embeddings(len(tokenizer), mean_resizing=False, pad_to_multiple_of=8)
    print(model)

    dataset = DatasetFixed(data_args.train_file)
    print('dataset', len(dataset), dataset[0]['attention_mask'].shape)

    def collator(batch):
        batch = [b for b in batch if b is not None]
        input_ids = [b['input_ids'] for b in batch]
        position_ids = [b['position_ids'] for b in batch]
        labels = [b['input_ids'].copy() for b in batch]
        attention_mask = [b['attention_mask'] for b in batch]
        input_ids = np.concatenate(input_ids)
        position_ids = np.concatenate(position_ids)
        labels = np.concatenate(labels)
        query_lens = np.concatenate(attention_mask)
        cumsum = [0] + np.cumsum(query_lens).tolist()
        max_cumsum = int(np.max(cumsum))
        cu_seq_lens_q = torch.tensor(cumsum, dtype=torch.int32)
        cu_seq_lens_k = torch.tensor(cumsum, dtype=torch.int32)
        max_seqlen_q = np.max(query_lens)
        return {
            'input_ids': torch.tensor(input_ids)[None],
            'position_ids': torch.tensor(position_ids)[None],
            'labels': torch.tensor(labels)[None],
            'cu_seq_lens_q': cu_seq_lens_q,
            'cu_seq_lens_k': cu_seq_lens_k,
            'max_length_q': max_seqlen_q,
            'max_length_k': max_seqlen_q
        }

    optimizer = MuonPlusAdamW(
        model.named_parameters(),
        lr=training_args.lr_scheduler_kwargs['lr'],
        muon_lr=training_args.lr_scheduler_kwargs['lr_muon'],
        muon_momentum=0.95,
        muon_weight_decay=training_args.lr_scheduler_kwargs['decay'],
        adamw_weight_decay=training_args.lr_scheduler_kwargs['decay'],
        muon_nesterov=True,
        muon_ns_steps=5,
    )
    len_dataset = math.ceil(len(dataset) / torch.cuda.device_count())
    len_dataloader = math.ceil(len_dataset / training_args.per_device_train_batch_size)
    num_update_steps_per_epoch = max(
        len_dataloader // training_args.gradient_accumulation_steps
        + int(len_dataloader % training_args.gradient_accumulation_steps > 0),
        1,
    )
    max_steps = math.ceil(training_args.num_train_epochs * num_update_steps_per_epoch)
    print('max_steps', max_steps)
    lr_scheduler = get_wsd_schedule(
        optimizer,
        num_warmup_steps=training_args.warmup_steps,
        num_decay_steps=training_args.lr_scheduler_kwargs['num_decay_steps'],
        num_training_steps=max_steps,
        min_lr_ratio=training_args.lr_scheduler_kwargs['min_lr_ratio'],
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=dataset,
        eval_dataset=None,
        tokenizer=tokenizer,
        data_collator=collator,
        compute_metrics=None,
        preprocess_logits_for_metrics=None,
        optimizers=(optimizer, lr_scheduler),
    )

    if training_args.do_train:
        checkpoint = None
        if training_args.resume_from_checkpoint is not None:
            checkpoint = training_args.resume_from_checkpoint
        elif last_checkpoint is not None:
            checkpoint = last_checkpoint
        trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()
        trainer.save_state()

def _mp_fn(index):
    # For xla_spawn (TPUs)
    main()

if __name__ == "__main__":
    main()


import os
import soundfile as sf
import copy
import itertools
import librosa
import re
from functools import partial
from multiprocess import Pool
from datasets import load_dataset
from tqdm import tqdm
import click

MODEL_NAME = os.environ.get('MODEL_NAME', 'Scicom-intl/Multilingual-Expressive-TTS-1.7B')
REVISION = os.environ.get('REVISION', 'main')

def old_chunks(l, n):
    for i in range(0, len(l), n):
        yield (l[i: i + n], i // n)

def chunks(l, devices):
    chunk_size = len(l) // len(devices)
    remainder = len(l) % len(devices)
    start = 0
    for i in range(len(devices)):
        extra = 1 if i < remainder else 0
        end = start + chunk_size + extra
        yield (l[start:end], devices[i])
        start = end

def multiprocessing(strings, function, cores=6, returned=True):
    df_split = old_chunks(strings, len(strings) // cores)
    pool = Pool(cores)
    pooled = pool.map(function, df_split)
    pool.close()
    pool.join()

    if returned:
        return list(itertools.chain(*pooled))

def check(indices_device_pair):
    rows, device = indices_device_pair
    filtered = []
    for r in tqdm(rows):
        filename = os.path.join(r['output'], f"{r['index']}-{r['retry']}.mp3")
        try:
            sf.read(filename)
            continue
        except:
            pass
        filtered.append(r)

    return filtered

def loop(indices_device_pair):
    rows, device = indices_device_pair
    os.environ['CUDA_VISIBLE_DEVICES'] = str(device)

    import torch
    torch.set_float32_matmul_precision('high')

    from transformers import AutoTokenizer, AutoModelForCausalLM
    from neucodec import NeuCodec

    codec = NeuCodec.from_pretrained("neuphonic/neucodec")
    _ = codec.eval().to('cuda')

    model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to('cuda')
    # model.generation_config.cache_implementation = "static"
    # model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
    tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

    for r in tqdm(rows):
        speaker = r['speaker']
        filename = os.path.join(r['output'], f"{r['index']}-{r['retry']}.mp3")

        try:
            prompt = f"<|im_start|>{speaker}: {r['target_text']}<|speech_start|>"
            inputs = tokenizer(prompt,return_tensors="pt", add_special_tokens=True).to(model.device)

            with torch.no_grad():
                outputs = model.generate(
                    **inputs,
                    max_new_tokens=1024,
                    do_sample=True,
                    temperature=0.7,
                    repetition_penalty=1.15,
                )
            generated_text = tokenizer.decode(outputs[0], skip_special_tokens=False)
            audio_tokens = re.findall(r'<\|s_(\d+)\|>', generated_text.split('<|speech_start|>')[-1])
            audio_tokens = [int(token) for token in audio_tokens]
            audio_codes = torch.tensor(audio_tokens)[None, None]

            with torch.no_grad():
                audio_waveform = codec.decode_code(audio_codes.cuda())

            os.makedirs(os.path.split(filename)[0], exist_ok = True)
            sf.write(filename, audio_waveform[0, 0].cpu().numpy(), 24000)
        except Exception as e:
            print(e)

@click.command()
@click.option('--output')
@click.option('--speaker')
@click.option('--replication', default = 1)
@click.option('--retry', default = 2)
def main(output, speaker, replication, retry):
    devices = os.environ.get('CUDA_VISIBLE_DEVICES')
    if devices is None:

        import torch
        devices = list(range(torch.cuda.device_count()))
    else:
        devices = [d.strip() for d in devices.split(',')]

    devices = replication * devices

    ds = load_dataset("Scicom-intl/Evaluation-Multilingual-VC", 'combine_filtered_whisper_large_v3')
    df = ds['train'].to_pandas()
    df['index'] = df.index
    rows = df.to_dict(orient='records')
    actual_rows = []
    for r in rows:
        if len(r['target_text']) < 5:
            continue
        for k in range(retry):
            r = copy.copy(r)
            r['speaker'] = speaker
            r['retry'] = k
            r['output'] = output
            actual_rows.append(r)

    filtered = multiprocessing(actual_rows, check, cores=20)
    print(len(filtered))
    if len(filtered):
        df_split = list(chunks(filtered, devices))

        loop_partial = partial(loop)

        with Pool(len(devices)) as pool:
            pooled = pool.map(loop_partial, df_split)

if __name__ == '__main__':
    main()


# Streamable TTS API

Streaming Text-to-Speech and Voice Conversion API with dynamic batching, CUDA Graphs, and torch.compile support.

## Setup

### 1. Start vLLM backend

```bash
STT_MODEL=Scicom-intl/Multilingual-TTS-1.7B-Base GPU_MEM_UTIL=0.7 \
docker compose -f vllm.yaml up --detach
```

Set `TTS_API=http://tts-engine:9093` in your [.env](.env) to point to the vLLM backend.

### 2. Configure environment

Copy [.env_example](.env_example) to `.env` and adjust as needed. See [app/env.py](app/env.py) for all available variables:

| Variable | Default | Description |
|---|---|---|
| `TTS_API` | `http://tts-engine:9093` | vLLM backend URL |
| `MODEL_NAME` | `TTS-model` | Model identifier |
| `DEFAULT_SPEAKER` | `husein` | Default voice |
| `SPEAKERS` | `husein,idayu,jenny` | Available voices |
| `DEFAULT_TEMPERATURE` | `0.6` | Sampling temperature |
| `DEFAULT_REPETITION_PENALTY` | `1.15` | Repetition penalty |
| `DEFAULT_MAX_TOKENS` | `3072` | Max output tokens |
| `DEFAULT_PLAYBACK_SPEED` | `1.5` | Playback speed multiplier |
| `DEFAULT_PLAYBACK_OVERLAP_SPEED` | `0.2` | Overlap speed for crossfading |
| `DYNAMIC_BATCHING` | `false` | Enable dynamic batching |
| `MICROSLEEP` | `1e-4` | Batch collection interval (seconds) |
| `MAX_BATCH_SIZE` | `16` | Max requests per batch |
| `CUDA_GRAPH_BATCH` | `[]` | CUDA graph bucket sizes, e.g. `[0.1, 0.2, 0.5, 1.0, 1.5, 2.0, 3.0, 10.0]` |
| `TORCH_COMPILE` | `false` | Use torch.compile instead of CUDA Graphs |
| `DEBUG_AUDIO` | `false` | Save intermediate audio chunks to disk |
| `SENTRY_DSN` | ` ` | Sentry DSN for error tracking |

### 3. Run the API

**GPU:**

```bash
docker compose up --build
```

**CPU:**

```bash
docker compose -f docker-compose-cpu.yaml up --build
```

## API Endpoints

### `GET /v1/audio/speaker`

Returns the list of available speaker voices.

### `POST /v1/audio/speech` — Text-to-Speech

Accepts JSON body.

**Parameters:**

| Field | Type | Default | Description |
|---|---|---|---|
| `input` | string | required | Text to synthesize |
| `voice` | string | `husein` | Speaker voice |
| `model` | string | `TTS-model` | Model name |
| `response_format` | `pcm` \| `wav` | `pcm` | Output audio format |
| `temperature` | float | `0.6` | Sampling temperature |
| `repetition_penalty` | float | `1.15` | Repetition penalty |
| `max_tokens` | int | `3072` | Max output tokens |
| `stream` | bool | `true` | Stream audio response |
| `playback_speed` | float | `1.5` | Playback speed |
| `playback_overlap_speed` | float | `0.2` | Overlap for crossfading |
| `normalize_malaysian` | bool | `true` | Apply Malaysian text normalization |

**Example:**

```bash
curl -X POST 'http://localhost:9091/v1/audio/speech' \
  -H 'Content-Type: application/json' \
  -d '{
    "input": "Hello! How can I help you? can I get your passport number sir.",
    "voice": "husein",
    "model": "TTS-model",
    "response_format": "wav",
    "temperature": 0.7,
    "stream": true,
    "playback_speed": 0.5,
    "playback_overlap_speed": 0.1,
    "normalize_malaysian": true
  }' \
  --output output.wav
```

### `POST /v1/audio/vc` — Voice Conversion

Accepts multipart form data. Clones the reference voice and generates speech for new text.

**Parameters:**

| Field | Type | Default | Description |
|---|---|---|---|
| `reference_audio` | file | required | Reference audio file |
| `reference_text` | string | required | Transcript of the reference audio |
| `generate_text` | string | required | Text to generate in the cloned voice |
| `model` | string | `TTS-model` | Model name |
| `response_format` | `pcm` \| `wav` | `pcm` | Output audio format |
| `temperature` | float | `0.6` | Sampling temperature |
| `repetition_penalty` | float | `1.15` | Repetition penalty |
| `max_tokens` | int | `3072` | Max output tokens |
| `stream` | bool | `true` | Stream audio response |
| `playback_speed` | float | `1.5` | Playback speed |
| `playback_overlap_speed` | float | `0.2` | Overlap for crossfading |

**Example:**

```bash
curl -X POST 'http://localhost:9091/v1/audio/vc' \
  -H 'Content-Type: multipart/form-data' \
  --form 'reference_audio=@jenny.wav' \
  --form 'reference_text=I wonder if I shall ever be happy enough to have real lace on my clothes and bows on my caps.' \
  --form 'generate_text=Ye encik, apa yang saya boleh tolong?' \
  --form 'model=TTS-model' \
  --form 'response_format=wav' \
  --form 'temperature=0.7' \
  --form 'repetition_penalty=1.15' \
  --form 'max_tokens=3072' \
  --form 'stream=true' \
  --form 'playback_speed=1.5' \
  --form 'playback_overlap_speed=0.2' \
  --output vc.wav
```

## Stress Test

Run the stress test against the current worker configuration defined in [docker-compose.yaml](docker-compose.yaml):

```bash
docker compose -f stress-test.yaml up --build
```

Uses default parameters from [app/env.py](app/env.py).

### H100 NVL

This one based on H100 NVL in Azure Standard_NC40ads_H100_v5

#### dynamic batching v1, older private commit

```
stress-test  | === SLO LATENCY REPORT ===
stress-test  | Total Requests: 50
stress-test  | Min Time: 1.606s
stress-test  | Max Time: 3.539s
stress-test  | Avg Time: 2.676s
stress-test  | P50 (Median): 2.568s
stress-test  | P90: 3.431s
stress-test  | P95: 3.474s
stress-test  | P99: 3.510s
stress-test  |
stress-test  | === AUDIO & REAL-TIME FACTOR (RTF) REPORT ===
stress-test  | Avg Audio Duration: 5.702s
stress-test  |
stress-test  | --- RTF Percentiles ---
stress-test  | Min RTF: 0.294s
stress-test  | Max RTF: 0.591s
stress-test  | Avg RTF: 0.470
stress-test  | P50 RTF: 0.434
stress-test  | P90 RTF: 0.581
stress-test  | P95 RTF: 0.587
stress-test  | P99 RTF: 0.589
stress-test  | ==========================
```

#### dynamic batching v2

```
stress-test  | === SLO LATENCY REPORT ===
stress-test  | Total Requests: 50
stress-test  | Min Time: 1.840s
stress-test  | Max Time: 2.296s
stress-test  | Avg Time: 2.164s
stress-test  | P50 (Median): 2.198s
stress-test  | P90: 2.286s
stress-test  | P95: 2.290s
stress-test  | P99: 2.295s
stress-test  |
stress-test  | === AUDIO & REAL-TIME FACTOR (RTF) REPORT ===
stress-test  | Avg Audio Duration: 5.573s
stress-test  |
stress-test  | --- RTF Percentiles ---
stress-test  | Min RTF: 0.329s
stress-test  | Max RTF: 0.554s
stress-test  | Avg RTF: 0.392
stress-test  | P50 RTF: 0.379
stress-test  | P90 RTF: 0.427
stress-test  | P95 RTF: 0.503
stress-test  | P99 RTF: 0.529
stress-test  | ==========================
```

### GPU RTX 3090 Ti

#### dynamic batching

Use [.env_dynamicbatching](.env_dynamicbatching).

```
stress-test  | === SLO LATENCY REPORT ===
stress-test  | Total Requests: 50
stress-test  | Min Time: 6.229s
stress-test  | Max Time: 6.545s
stress-test  | Avg Time: 6.429s
stress-test  | P50 (Median): 6.424s
stress-test  | P90: 6.530s
stress-test  | P95: 6.543s
stress-test  | P99: 6.544s
stress-test  |
stress-test  | === AUDIO & REAL-TIME FACTOR (RTF) REPORT ===
stress-test  | Avg Audio Duration: 5.759s
stress-test  |
stress-test  | --- RTF Percentiles ---
stress-test  | Min RTF: 0.966s
stress-test  | Max RTF: 1.702s
stress-test  | Avg RTF: 1.124
stress-test  | P50 RTF: 1.099
stress-test  | P90 RTF: 1.194
stress-test  | P95 RTF: 1.203
stress-test  | P99 RTF: 1.405
stress-test  | ==========================
```

#### dynamic batching with CUDA Graphs

Use [.env_dynamicbatching_cudagraph](.env_dynamicbatching_cudagraph).

```
stress-test  | === SLO LATENCY REPORT ===
stress-test  | Total Requests: 50
stress-test  | Min Time: 6.125s
stress-test  | Max Time: 6.323s
stress-test  | Avg Time: 6.262s
stress-test  | P50 (Median): 6.273s
stress-test  | P90: 6.313s
stress-test  | P95: 6.322s
stress-test  | P99: 6.322s
stress-test  |
stress-test  | === AUDIO & REAL-TIME FACTOR (RTF) REPORT ===
stress-test  | Avg Audio Duration: 5.731s
stress-test  |
stress-test  | --- RTF Percentiles ---
stress-test  | Min RTF: 0.983s
stress-test  | Max RTF: 1.701s
stress-test  | Avg RTF: 1.105
stress-test  | P50 RTF: 1.067
stress-test  | P90 RTF: 1.160
stress-test  | P95 RTF: 1.278
stress-test  | P99 RTF: 1.540
stress-test  | ==========================
```

#### Without dynamic batching

Use [.env_nodynamicbatching](.env_nodynamicbatching).

```
stress-test  | === SLO LATENCY REPORT ===
stress-test  | Total Requests: 50
stress-test  | Min Time: 4.847s
stress-test  | Max Time: 7.734s
stress-test  | Avg Time: 7.136s
stress-test  | P50 (Median): 7.291s
stress-test  | P90: 7.637s
stress-test  | P95: 7.667s
stress-test  | P99: 7.710s
stress-test  |
stress-test  | === AUDIO & REAL-TIME FACTOR (RTF) REPORT ===
stress-test  | Avg Audio Duration: 5.630s
stress-test  |
stress-test  | --- RTF Percentiles ---
stress-test  | Min RTF: 1.228s
stress-test  | Max RTF: 1.282s
stress-test  | Avg RTF: 1.267
stress-test  | P50 RTF: 1.269
stress-test  | P90 RTF: 1.278
stress-test  | P95 RTF: 1.281
stress-test  | P99 RTF: 1.281
stress-test  | ==========================
```


# TTS Evaluation

Benchmarking multilingual TTS models across 76 languages using Character Error Rate (CER) and MOS based on UTMOSv2.

![Benchmark scatter](scatter_results.png)

## Models

| Model | Description |
|-------|-------------|
| **Dia TTS** | [Nari Labs Dia TTS](https://github.com/nari-labs/dia) |
| **Multilingual TTS 0.6B** | [Scicom-intl/Multilingual-Expressive-TTS-0.6B](https://huggingface.co/Scicom-intl/Multilingual-Expressive-TTS-0.6B) |
| **Multilingual TTS 1.7B** | [Scicom-intl/Multilingual-Expressive-TTS-1.7B](https://huggingface.co/Scicom-intl/Multilingual-Expressive-TTS-1.7B) |
| **Orpheus** | [Orpheus TTS](https://github.com/canopyai/Orpheus-TTS) |
| **Chatterbox** | [Chatterbox TTS](https://github.com/resemble-ai/chatterbox) — 23 languages only |
| **Fish Audio S2 Pro** | [Fish Audio S2 Pro](https://github.com/fishaudio/fish-speech) |
| **Qwen3 TTS** | [Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice) — 11 languages only |

## Setup

```bash
pip install -r requirements.txt
```

## Run Generations

Each prompt is generated **twice** and the scores are averaged to reduce variance. We also upload all the generations done by us at [Scicom-intl/Evaluation-Multilingual-VC](https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC),

- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/dia-tts-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/resolve/main/multilingual-tts-0.6b-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/multilingual-tts-1.7b-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/orpheus-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/chatterbox-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/fishspeech2-tts.zip
- https://huggingface.co/datasets/Scicom-intl/Evaluation-Multilingual-VC/blob/main/qwen3_tts-tts.zip

### Dia TTS

```bash
python3 dia_tts.py --output 'dia-tts'
```

### Scicom Multilingual TTS

```bash
# 0.6B
MODEL_NAME="Scicom-intl/Multilingual-Expressive-TTS-0.6B" python3 multilingual_tts.py \
  --speaker 'multilingual-tts_audio_Grace' --output 'multilingual-tts-0.6b'

# 1.7B
MODEL_NAME="Scicom-intl/Multilingual-Expressive-TTS-1.7B" python3 multilingual_tts.py \
  --speaker 'multilingual-tts_audio_Grace' --output 'multilingual-tts-1.7b'
```

### Orpheus

```bash
python3 orpheus.py   --output 'orpheus'
```

### Chatterbox

```bash
python3 chatterbox.py --output 'chatterbox'
```

### Fish Audio S2 Pro

```bash
python3 fishspeech2.py --output 'fishspeech2'
```

### Qwen3 TTS

```bash
python3 qwen3_tts.py --output 'qwen3_tts'
```

## Evaluate

### CER

```bash
python3 calculate_cer.py --output_folder "dia-tts"               --output "dia-tts-cer"
python3 calculate_cer.py --output_folder "multilingual-tts-0.6b" --output "multilingual-tts-0.6b-cer"
python3 calculate_cer.py --output_folder "multilingual-tts-1.7b" --output "multilingual-tts-1.7b-cer"
python3 calculate_cer.py --output_folder "orpheus"               --output "orpheus-cer"
python3 calculate_cer.py --output_folder "chatterbox"            --output "chatterbox-cer"
python3 calculate_cer.py --output_folder "fishspeech2"           --output "fishspeech2-cer"
python3 calculate_cer.py --output_folder "qwen3_tts"             --output "qwen3_tts-cer"
```

### MOS

We evaluate using [Scicom-AI-Enterprise-Organization/faster-UTMOSv2](https://github.com/Scicom-AI-Enterprise-Organization/faster-UTMOSv2) with 5 repetitions.

```bash
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "dia-tts"               --output "dia-tts-mos"
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "multilingual-tts-0.6b" --output "multilingual-tts-0.6b-mos" --replication 5
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "multilingual-tts-1.7b" --output "multilingual-tts-1.7b-mos" --replication 5
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "orpheus"               --output "orpheus-mos" --replication 5
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "chatterbox"            --output "chatterbox-mos" --replication 5
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "fishspeech2"           --output "fishspeech2-mos" --replication 5
OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1 \
python3 calculate_mos.py --output_folder "qwen3_tts"             --output "qwen3_tts-mos" --replication 5
```

## Results

Summary across all evaluated languages. Full per-language heatmap below.

| Model | Languages | CER ↓ | MOS ↑ |
|-------|:---------:|:-----:|:-----:|
| Dia TTS | 76 | 0.8131 | 1.8575 |
| Multilingual TTS 0.6B | 76 | 0.2384 | 3.2273 |
| Multilingual TTS 1.7B | 76 | **0.2362** | **3.2330** |
| Orpheus | 76 | 0.6075 | 2.7267 |
| Chatterbox | 23 | 0.1698 | 2.8405 |
| Fish Audio S2 Pro | 76 | 0.2370 | 2.9698 |
| Qwen3 TTS | 11 | **0.1064** | 2.6073 |

> Chatterbox covers 23 languages only; Qwen3 TTS covers 11 languages only. Their averages are not directly comparable to 76-language models.

### Full Breakdown (76 languages)

![Benchmark heatmap](benchmark_results.png)


Now let me check if there's any info about finetuning on new languages, and look at the training shell scripts for more details.


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/contents/1.7B.sh -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch 1.7B training script


gh api repos/Scicom-AI-Enterprise-Organization/Multilingual-TTS/contents/vc-tts/README.md -H "Accept: application/vnd.github.raw+json" 2>/dev/null

Fetch VC-TTS README


export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
WANDB_PROJECT="Multilingual-TTS" \
WANDB_NAME="Qwen3-1.7B-float32-1.5" \
TORCH_DISTRIBUTED_DEBUG="info" \
torchrun --nproc_per_node 8 \
-m qwen3_muonadamw \
--model_name_or_path "Qwen/Qwen3-1.7B-Base" \
--per_device_train_batch_size 8 \
--gradient_accumulation_steps 32 \
--output_dir gfs/01be5b33/Multilingual-TTS-Qwen3-1.7B-float32-1.5 \
--bf16 --do_train --do_eval false --num_train_epochs 3 \
--train_file "gfs/01be5b33/multipacking-tts" \
--logging_steps 1 \
--learning_rate 1e-3 \
--warmup_steps 50 \
--block_size 10240 \
--save_steps 50 \
--save_total_limit 10 \
--gradient_checkpointing true \
--torch_dtype float32 \
--ddp_find_unused_parameters false \
--dataloader_num_workers 5 \
--dataloader_prefetch_factor 20 \
--remove_unused_columns false \
--lr_scheduler_type "warmup_stable_decay" \
--lr_scheduler_kwargs '{"num_decay_steps": 243, "min_lr_ratio": 1e-1, "lr": 1e-3, "lr_muon": 1e-2, "decay": 0.01}'


# VC TTS

Building Voice Conversion dataset using TTS dataset,

1. Group the dataset based on speakers.
2. Calculate similarity using Titanet Large.
3. Filter high similarity only.

Dataset pushed into https://huggingface.co/datasets/Scicom-intl/Multilingual-TTS-Voice-Conversion


Now let me check the TTS-API-Neucodec repo for more concurrency details.


gh api repos/Scicom-AI-Enterprise-Organization/TTS-API-Neucodec/git/trees/main?recursive=1 --jq '.tree[].path' 2>/dev/null | head -30

List files in TTS-API-Neucodec repo


.dockerignore
.env_dynamicbatching
.env_dynamicbatching_cudagraph
.env_dynamicbatching_torchcompile
.env_example
.env_nodynamicbatching
.gitignore
Dockerfile
README.md
app
app/env.py
app/main.py
app/neucodec
app/neucodec/__init__.py
app/neucodec/activations.py
app/neucodec/alias_free_torch
app/neucodec/alias_free_torch/__init__.py
app/neucodec/alias_free_torch/act.py
app/neucodec/alias_free_torch/filter.py
app/neucodec/alias_free_torch/resample.py
app/neucodec/bs_roformer5.py
app/neucodec/codec_decoder_vocos.py
app/neucodec/codec_encoder.py
app/neucodec/codec_encoder_distill.py
app/neucodec/distill_layers.py
app/neucodec/model.py
app/neucodec/module.py
app/neucodec/tconv
app/neucodec/tconv/__init__.py
app/neucodec/tconv/base.py


I now have all the information needed. Here are my detailed findings:

---

## 1. Autoregressive or Non-Autoregressive?

**The model is autoregressive.** The evidence is conclusive:

- The model is built on top of **Qwen3 (a causal LM)** -- specifically `Qwen3ForCausalLM`. The training script (`qwen3_muonadamw.py`) subclasses `Qwen3ForCausalLM` and trains it with a standard causal language modeling loss (cross-entropy on next-token prediction, with `labels[..., 1:]`).
- At inference time (`tts-evaluation/multilingual_tts.py`), it uses `model.generate()` with `do_sample=True, temperature=0.7` -- standard autoregressive decoding.
- The model generates discrete speech tokens sequentially (extracted via regex `<|s_(\d+)\|>` from the generated text), which are then decoded by the **Neucodec** codec at 50 tokens per second.
- The base models used are **Qwen3-0.6B-Base** and **Qwen3-1.7B-Base**, which are decoder-only causal transformers.

The tokenizer adds 65,536 speech tokens (`<|s_0|>` through `<|s_65535|>`) plus a `<|speech_start|>` delimiter, turning a text LLM into a text-to-speech-token autoregressive model.

---

## 2. RTF (Real-Time Factor)

RTF benchmarks are documented in the **TTS-API-Neucodec** companion repo. All tests used 50 concurrent requests with ~5.6-5.7s average audio duration:

**H100 NVL (Azure Standard_NC40ads_H100_v5):**
- Dynamic batching v2: **Avg RTF 0.392**, P50 0.379, P90 0.427, P95 0.503
- Dynamic batching v1: Avg RTF 0.470, P50 0.434, P90 0.581

**RTX 3090 Ti:**
- Dynamic batching: **Avg RTF 1.124**, P50 1.099, P90 1.194
- Dynamic batching + CUDA Graphs: **Avg RTF 1.105**, P50 1.067, P90 1.160
- Without dynamic batching: Avg RTF 1.267, P50 1.269

An RTF below 1.0 means faster than real-time. So on H100 it runs at roughly **2.5x real-time**, while on RTX 3090 Ti it is roughly at real-time (RTF ~1.1).

---

## 3. GPU Concurrency Info

The companion repo **TTS-API-Neucodec** provides a full production-ready inference API with these concurrency features:

- **Dynamic batching**: configurable via `DYNAMIC_BATCHING` env var, with `MICROSLEEP` (batch collection interval, default 1e-4 seconds) and `MAX_BATCH_SIZE` (default 16).
- **CUDA Graphs**: configurable via `CUDA_GRAPH_BATCH` with bucket sizes (e.g., `[0.1, 0.2, 0.5, 1.0, 1.5, 2.0, 3.0, 10.0]`).
- **torch.compile**: alternative to CUDA Graphs, enabled via `TORCH_COMPILE`.
- **vLLM backend**: the LLM inference is served via vLLM (`docker compose -f vllm.yaml`), which itself handles batching/scheduling.
- Stress tests were run with **50 concurrent requests** on both H100 and RTX 3090 Ti.

---

## 4. Supported Languages

The model supports **150+ languages** for TTS, plus voice cloning support. From the README:

- Multi-speaker multilingual TTS covering 150+ languages (up to 25.35B tokens of training data)
- Multi-speaker multilingual Voice Cloning (up to 35.88B tokens)
- Evaluation was performed across **76 languages** (CER and MOS benchmarks)
- The evaluation dataset is at `Scicom-intl/Evaluation-Multilingual-VC`

---

## 5. Finetuneability on New Languages

**Yes, it is finetuneable.** The entire training pipeline is open-sourced:

- The training script (`qwen3_muonadamw.py`) is a standard HuggingFace `Trainer`-based pipeline that fine-tunes Qwen3 base models on speech token data.
- The base models start from **Qwen3-0.6B-Base** or **Qwen3-1.7B-Base** and add 65,536 speech tokens to the vocabulary.
- Data preparation notebooks are provided in the `preparation/` directory showing how to create multipacked training datasets from audio.
- The training scripts (`0.6B.sh`, `1.7B.sh`, etc.) show exactly how to launch training on 8 GPUs.
- The model uses the **Neucodec** tokenizer (50 tokens/sec) to convert audio to discrete tokens -- any language's audio can be tokenized this way.
- There is even an expressive TTS post-training pipeline (`synthetic-description/`) showing how to add style/emotion control.

To fine-tune on a new language, you would: (1) tokenize your audio with Neucodec, (2) prepare multipacked datasets following the `preparation/` notebooks, and (3) run the training scripts. The architecture itself is language-agnostic since Qwen3's text tokenizer already has broad multilingual coverage, and the speech tokens are language-independent codec codes.