Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 200
Output:

# Strategic Plan for Building a Human-Like TTS LLM from YouTube Data

## 1. Transcription Strategy: Overcoming the Quality Bottleneck at Scale

The foundational challenge in developing a high-fidelity Text-to-Speech (TTS) model, particularly one based on a Large Language Model (LLM) architecture, is the acquisition of a massive, high-quality, and accurately transcribed audio-text dataset. The project's initial strategy relied on sourcing this data from YouTube, a vast and diverse repository of speech. However, the subsequent attempt to transcribe this "in-the-wild" audio using a variety of state-of-the-art Automatic Speech Recognition (ASR) models and Large Language Models (LLMs) has proven to be a significant bottleneck. The core issue lies in the inability of these models to consistently achieve the required transcription accuracy, specifically a **Word Error Rate (WER) below 10%**, which is a critical threshold for producing training data suitable for a human-like TTS system. This section details the systematic failures of conventional transcription approaches and proposes a novel, multi-pronged strategy to overcome this obstacle. The proposed solution pivots from a direct transcription paradigm to a more robust, quality-centric approach that leverages Generative Speech Enhancement (GSE) models for both audio cleaning and, crucially, for generating confidence scores that can be used to filter the dataset. This method allows for the curation of a high-quality subset of audio data even in the absence of perfect ground-truth transcripts. Furthermore, this section explores alternative methods for acquiring transcriptions, such as leveraging pre-existing YouTube Closed Captions (CCs), and outlines a comprehensive strategy for transcript normalization, including the handling of code-mixed languages, multiple scripts, and the integration of semantic audio events.

### 1.1. Core Challenge: Inadequate Accuracy from Standard ASR Models

The initial phase of the project involved a systematic evaluation of various ASR and LLM-based transcription models to process the vast corpus of audio data extracted from YouTube. The goal was to generate accurate text-audio pairs for the subsequent TTS model training. Despite the sophistication of the models tested, none were able to meet the stringent accuracy requirements necessary for high-quality TTS synthesis. The failure was consistent across different model families, including open-source ASR systems, proprietary large language models, and models specifically fine-tuned for Indic languages. This widespread underperformance highlights the inherent difficulty of transcribing "in-the-wild" audio, which is characterized by a wide range of acoustic conditions, background noise, speaker accents, and spontaneous speech patterns that are not well-represented in the training data of most standard ASR systems. The inability to secure reliable transcriptions at scale has necessitated a fundamental rethinking of the data curation strategy, moving away from a reliance on perfect transcripts and towards a methodology that can identify and select high-quality audio based on intrinsic, model-derived quality metrics. This pivot is critical for unlocking the value of the massive YouTube dataset and ensuring that the TTS model is trained on data that is not only large in volume but also high in fidelity.

#### 1.1.1. Failure of Open-Source Models (Whisper, Indic-Whisper, IndicConformer)

The first line of attack in the transcription effort involved a suite of powerful open-source ASR models. The team evaluated **Whisper-large-v3**, a state-of-the-art multilingual model known for its robustness. However, when applied to the diverse and often noisy YouTube audio, its performance fell short of the required **<10% WER benchmark**. Recognizing the specific linguistic challenges of the target dataset, the team then turned to specialized models. **Indic-Whisper**, a variant of Whisper specifically fine-tuned on a variety of Indic languages, was tested with the expectation that its specialized training would yield better results. Similarly, **IndicConformer**, another model designed for the Indian linguistic landscape, was also put through its paces. Despite their specialized nature, both models exhibited the same fundamental limitation: they were unable to consistently and accurately transcribe the long-tail of acoustic phenomena present in the YouTube data. The performance degradation was particularly noticeable in segments with background music, overlapping speech, or non-native accents, which are common in podcast-style content. The consistent failure of these models, even those tailored for the target languages, underscores the gap between the controlled conditions of academic benchmarks and the chaotic reality of "in-the-wild" internet audio. This experience demonstrated that simply selecting a better open-source model was not a viable path forward and that a more fundamental change in approach was required.

#### 1.1.2. Failure of Large Language Models (Gemini, Gemma, Voxtral)

With the limitations of traditional ASR models becoming apparent, the project explored the potential of Large Language Models (LLMs) for the transcription task. The hypothesis was that the vast world knowledge and powerful reasoning capabilities of modern LLMs could be leveraged to "understand" the context of speech and produce more accurate transcriptions, especially in challenging acoustic scenarios. The team experimented with prompting strategies for models like **Gemini-2.5-flash**, **Gemini-3-flash**, and **Gemini-3-pro**, providing detailed instructions to guide the model towards verbatim transcription. However, even with temperature set to 0.0 to minimize creativity, the models consistently failed to produce the required level of accuracy. The LLMs exhibited a tendency to **"hallucinate"** or creatively interpret the audio, adding, removing, or altering words in a way that, while often semantically plausible, was not a faithful transcription of the source audio. This behavior is particularly problematic for TTS training, where the alignment between the audio and the exact text is paramount. The models also struggled with code-mixed language (e.g., Hinglish), often failing to recognize the language switch or transcribing it incorrectly. The failure of these powerful LLMs revealed a critical insight: their training objectives, which prioritize fluent and coherent text generation, are fundamentally misaligned with the task of verbatim transcription. This misalignment makes them unsuitable as a primary tool for generating the high-fidelity text-audio pairs needed for TTS model training, forcing the project to seek alternative methods that are more robust to the challenges of "in-the-wild" audio.

#### 1.1.3. Failure of Specialized Models (wav2vec 2.0, w2vindia)

The final category of models evaluated were those based on the **wav2vec 2.0** architecture, which has set new standards in self-supervised speech representation learning. The **`w2vindia`** model, a variant specifically pre-trained and fine-tuned on a large corpus of Indian language speech data, was a key candidate . The expectation was that its specialized training would make it the most suitable model for the task. However, even `w2vindia` failed to consistently achieve a WER below 10%. The model's performance, while better than some of the other candidates on certain language subsets, was still not reliable enough for the rigorous demands of TTS dataset creation. The high error rate suggests that the model's training data may not have fully captured the acoustic and linguistic variability present in the YouTube corpus, or that the fine-tuning process was not optimized for the specific characteristics of the audio being processed. Research on similar models, such as `IndicWav2Vec`, has shown that while they can achieve state-of-the-art results on specific benchmarks, their performance can vary significantly across different languages and domains . The failure of even this highly specialized model underscores the extreme difficulty of the transcription task and highlights the need for a fundamentally different approach. It became clear that relying on a single, off-the-shelf ASR model to transcribe the entire 50,000-hour dataset with the required accuracy was not a viable path forward.

### 1.2. Proposed Solution: Confidence-Based Filtering with Generative Speech Enhancement (GSE)

Faced with the inability of standard ASR and LLM models to provide accurate transcriptions, the project has identified a promising alternative strategy centered on **Generative Speech Enhancement (GSE)** . This approach represents a paradigm shift from the traditional "transcribe everything" methodology to a more nuanced **"curate for quality"** philosophy. The core idea is to leverage a GSE model not just as a tool for cleaning noisy audio, but as a sophisticated quality assessment engine. GSE models, which are trained to generate clean speech from noisy inputs, are known to be prone to "hallucination" errors—subtle but critical mistakes where the model omits phonemes, alters speaker characteristics, or introduces artifacts. A 2026 paper on confidence-based filtering for speech dataset curation proposes a method to detect these errors by using the model's own internal confidence scores . This technique allows for the non-intrusive filtering of a large "in-the-wild" dataset, effectively identifying and retaining only the audio segments that the GSE model processes with high confidence. By applying this method, the project can create a high-quality subset of the YouTube data that is free from the most egregious enhancement artifacts, providing a solid foundation for training the TTS model, even without perfect transcriptions for every segment. This strategy is particularly well-suited for the project's goal of training a neural codec, as the codec can be trained on this clean, high-confidence audio, and the subsequent TTS model can be trained to generate the discrete tokens produced by this codec.

#### 1.2.1. Concept: Leveraging GSE for Both Enhancement and Quality Scoring

The proposed solution is built on the dual-purpose utility of Generative Speech Enhancement (GSE) models. These models are typically used for their primary function: to take a noisy audio signal, such as a recording from a YouTube video, and generate a cleaner, higher-quality version of the speech. This is particularly valuable for the project's "in-the-wild" dataset, which contains a wide variety of background noises, reverberations, and other acoustic distortions. However, the key innovation lies in the second, more subtle application of these models: as a quality scoring mechanism. The 2026 paper "Confidence-based Filtering for Speech Dataset Curation with Generative Speech Enhancement Using Discrete Tokens" details a method that uses the internal state of a GSE model to assess the quality of its own output . The core insight is that when a GSE model is presented with a particularly challenging or noisy input, it may produce a clean-sounding output that is nonetheless flawed in terms of its content. These **"hallucination" errors**, such as phoneme omissions or speaker inconsistencies, are often missed by conventional non-intrusive quality metrics like DNSMOS or UTMOS, which focus more on the overall acoustic quality rather than the semantic fidelity of the speech . By analyzing the model's internal confidence, it becomes possible to detect these subtle but critical errors, providing a much more reliable signal for filtering the dataset. This dual approach allows the project to both clean the audio and simultaneously assess the trustworthiness of the cleaned output, creating a curated dataset that is not only acoustically clean but also semantically reliable.

#### 1.2.2. Implementation: Using Log-Probabilities of Discrete Tokens as Confidence Scores

The technical implementation of the confidence-based filtering strategy relies on the use of discrete token-based GSE models, such as the **Genhancer** model mentioned in the research . These models operate by first encoding the audio into a sequence of discrete tokens using a neural audio codec (like the Descript Audio Codec, DAC). The GSE model then processes these tokens to generate a new, enhanced sequence of tokens, which is finally decoded back into a clean audio waveform. The key to the quality scoring mechanism lies in the probabilities associated with the generation of these discrete tokens. The proposed method defines a token-level confidence score, `s_t`, for each time step `t` as the log-probability of the generated token from the first quantizer layer of the codec. This layer is chosen because it has the greatest perceptual impact on the final audio quality. The formula for this token-level confidence is given as:

`s_t = log p(x_t,1 = x̂_t,1 | c; θ)`

where `x_t,1` is the generated token, `c` is the conditioning information (the noisy audio), and `θ` represents the model's parameters .

To obtain a single quality score for an entire utterance, these token-level scores are averaged over the sequence length `T` to produce an utterance-level confidence score, `S_utt`:

`S_utt = (1/T) * Σ(s_t)`

This final score, `S_utt`, serves as a non-intrusive quality metric. A high score indicates that the model generated the enhanced speech with high confidence, suggesting a successful enhancement. Conversely, a low score implies that the model struggled with the input, flagging the output as a potential enhancement failure or hallucination. This score can then be used to filter the dataset, retaining only the utterances that exceed a certain confidence threshold, thereby curating a high-quality corpus for TTS training .

#### 1.2.3. Application: Filtering the "In-the-Wild" YouTube Dataset to Create a High-Quality Subset

The application of this confidence-based filtering method to the project's massive YouTube dataset offers a clear and actionable path forward. The process would involve a two-stage pipeline. In the first stage, a pre-trained GSE model, such as Genhancer, would be applied to every audio segment in the cleaned dataset. For each segment, the model would produce an enhanced audio file and its corresponding utterance-level confidence score, `S_utt`. In the second stage, a filtering threshold, `τ`, would be determined. This threshold could be set empirically, for example, by retaining the top N% of utterances based on the distribution of confidence scores across the entire dataset. All enhanced audio segments with a confidence score `S_utt` below this threshold `τ` would be discarded. The result would be a curated dataset, `D_curated`, containing only the high-confidence, high-quality audio:

`D_curated = {w_enhanced ∈ D_enhanced | S_utt ≥ τ}`

The research demonstrates the practical utility of this method. In their experiments, they curated an "in-the-wild" TTS dataset (TITW-hard, sourced from VoxCeleb) using this confidence-based filtering and showed that it improved the performance of a subsequently trained TTS model (Matcha-TTS) . The confidence score was shown to have a strong correlation with a suite of intrusive speech enhancement metrics, and it was particularly effective at identifying hallucination errors that were missed by other non-intrusive metrics like UTMOS. For instance, in one example, an enhanced speech sample with a high UTMOS score of 4.01 (top 39%) was found to have significant content corruption. In contrast, the proposed confidence score and the intrusive LPS metric both gave low scores, placing the sample in the top 93% and 81% respectively, correctly identifying it as a failure case . This demonstrates that the confidence-based filtering can effectively and non-intrusively identify and remove low-quality data, making it an ideal solution for curating the project's large-scale YouTube dataset.

### 1.3. Alternative Transcription Sources

While the primary strategy for data curation has shifted towards confidence-based filtering of audio, the project also recognizes the value of exploring alternative sources for acquiring accurate transcriptions. The ideal scenario for training a high-quality TTS model remains the availability of large volumes of perfectly aligned, high-fidelity audio-text pairs. Although the project's own ASR efforts have fallen short, there are other potential avenues for obtaining reliable text transcripts that should be investigated. These alternative methods could serve as a valuable supplement to the confidence-filtered audio, providing a smaller but potentially higher-quality set of transcribed data for specific use cases, such as supervised fine-tuning (SFT) or for training a more traditional ASR model to eventually bootstrap the transcription process. The two most promising avenues are the direct utilization of pre-existing YouTube Closed Captions (CCs) and the alignment of the audio with external text corpora, such as Wikipedia articles. These methods, while not without their own challenges, offer a different path to acquiring the necessary text data and could be particularly effective for a subset of the videos that meet specific criteria.

#### 1.3.1. Utilizing Pre-Existing YouTube Closed Captions (CCs)

One of the most direct methods for obtaining transcriptions is to leverage the Closed Captions (CCs) that are already available on many YouTube videos. These captions come in two main forms: **manually uploaded subtitles**, which are often of high quality, and **auto-generated captions**, which are created by YouTube's own ASR system. While the quality of auto-generated captions can be variable, they can still serve as a useful starting point, especially if they can be further processed or filtered. The project `YTTTS` (YouTube Text-To-Speech dataset) provides a clear precedent for this approach, demonstrating a pipeline that downloads both audio and captions from YouTube videos and then aligns them by parsing the `.srt` subtitle files . This method has the significant advantage of providing a direct, time-aligned text-audio pair, bypassing the need for a separate, error-prone transcription step.

However, this approach is not without its challenges. Firstly, not all videos have captions, and the availability of captions in the target Indic languages may be limited. Secondly, the accuracy of auto-generated captions, particularly for the long-tail of accents and dialects present in the dataset, is likely to be insufficient for direct use in TTS training. A 2021 article on preparing speech recognition datasets from YouTube notes that while using speech-to-text APIs can reduce the workload, manual editing is still required to produce a high-quality dataset . Despite these limitations, targeting videos with high-quality, manually created captions could provide a valuable, albeit smaller, subset of pristine data. Furthermore, even the imperfect auto-generated captions could be used as a weak supervision signal or as a starting point for a human-in-the-loop correction process. The project `yt-tts` also emphasizes the importance of choosing videos with subtitles, further validating this as a viable, if partial, solution . Therefore, a key part of the data strategy should be to identify and prioritize the download of videos that have reliable captions, using them to build a "gold standard" subset of the training data.

#### 1.3.2. Aligning Audio with External Text Corpora (e.g., Wikipedia)

A more advanced and potentially more fruitful strategy is to align the YouTube audio with external, high-quality text corpora. The central hypothesis is that a large number of YouTube videos, especially those in the "educational" or "informative" podcast category, are based on or directly read from existing written content. **Wikipedia**, with its vast repository of articles in multiple Indic languages, is a …4042 tokens truncated…ring the dataset is the isolation of **single-speaker segments**. TTS models, especially those designed for high-fidelity voice cloning, typically perform best when trained on clean, single-speaker audio. The raw YouTube videos, however, often contain multiple speakers, background noise, music, and other non-speech audio. To address this, the pipeline employs a **GPU-accelerated single-speaker segmentation** process. This process takes the raw audio from a `videoID.webm` file and uses advanced audio diarization techniques to identify and segment continuous stretches of audio belonging to a single speaker. This is a computationally demanding task that benefits significantly from GPU acceleration, allowing for efficient processing of the vast amount of audio data. The segmentation algorithm is designed to be robust, handling variations in speaker volume, speaking style, and background conditions. It intelligently identifies speaker change points and creates distinct segments, each containing audio from only one speaker. This step is crucial for ensuring that the subsequent TTS model learns to generate coherent and consistent speech for a single voice, rather than being confused by overlapping speakers or conversational turn-taking, which are better handled in a dedicated dialogue modeling stage. The output of this stage is a collection of single-speaker audio segments for each video, which are then passed to the next stage of the pipeline for further cleaning and quality assessment.

#### 2.2.2. Music and Noise Detection (PANN CNNs)

Even after single-speaker segmentation, the audio segments may contain significant amounts of background music, ambient noise, or other non-speech sounds that are detrimental to TTS model training. To filter these out, the pipeline incorporates a music and noise detection module based on **PANN (PANNs: Large-Scale Pretrained Audio Neural Networks) CNNs**. PANN models are state-of-the-art audio classification models that have been pre-trained on massive datasets like AudioSet, giving them a strong ability to recognize a wide variety of audio events and scenes. In this pipeline, a PANN-based classifier is applied to each single-speaker segment to detect the presence of music or other undesirable background noise. The model provides a confidence score or a set of statistics (e.g., `music_stats`) indicating the likelihood that a segment is contaminated. This information is stored in the segment's metadata, allowing for downstream filtering. For example, segments with a high probability of containing music can be automatically discarded or flagged for manual review. This automated detection is essential for maintaining the quality of the dataset at scale, as manually listening to and annotating hundreds of thousands of hours of audio would be infeasible. The use of a pre-trained model like PANN ensures high accuracy and robustness across the diverse range of acoustic conditions present in the YouTube data, from studio-quality podcasts to noisy street interviews. This step is a key component in the data cleaning process, ensuring that the final dataset is composed primarily of clean, intelligible speech.

#### 2.2.3. Resampling and Quality Bucketing (16/24/32/44 kHz)

The raw audio downloaded from YouTube can have a variety of different sample rates, depending on the original upload and the encoding settings used by YouTube. To ensure consistency and to optimize the training process, all audio segments are resampled to a set of standard sample rates. The strategy here is to create multiple **"buckets" of data**, each with a different sample rate, to cater to different stages of the training process. The primary buckets are **16 kHz, 24 kHz, 32 kHz, and 44.1 kHz**. The 16 kHz bucket is used for initial experiments and for training models that are less sensitive to high-frequency content. The higher sample rate buckets (24, 32, and 44.1 kHz) are used for training the final, high-fidelity TTS model, as they contain more detailed acoustic information. The decision of which bucket to place a segment in is based on its original sample rate and its quality. Segments with a native sample rate of 44.1 kHz are placed in the 44.1 kHz bucket. Segments with a lower native sample rate are placed in the highest possible bucket without upsampling, to avoid introducing artifacts. This bucketing strategy allows for a flexible and efficient training pipeline, where different models can be trained on data with different levels of fidelity.

#### 2.2.4. Packaging into WebDatasets (`.tar` and FLAC)

Once the audio segments have been cleaned, segmented, and quality-scored, they need to be packaged into a format that is efficient for large-scale, distributed training. The chosen format is **WebDataset**, which stores data in `.tar` archives. This format is highly scalable and allows for efficient streaming of data to training workers without the need for a centralized database or file system. Each `videoID.tar` file contains all the cleaned audio segments for a particular YouTube video, along with a `metadata.json` file that stores all the pre-computed information about those segments, such as their duration, quality scores, and results from the music/noise detection. The audio segments themselves are stored using the **FLAC (Free Lossless Audio Codec)** codec. FLAC is chosen because it is a lossless compression format, meaning it preserves the original audio quality without any degradation, which is crucial for training a high-fidelity TTS model. It also provides good compression, reducing storage costs and speeding up data transfer. The use of WebDatasets simplifies the data loading process during training, as the training script can simply read the `.tar` files sequentially, treating them as a stream of data samples. This architecture is well-suited for the planned training infrastructure, which will likely involve multiple GPUs processing data in parallel. The final packaged data, consisting of thousands of these `.tar` files, is then uploaded back to a designated Cloudflare R2 bucket, ready to be pulled by the training jobs.

### 2.3. Metadata Management and Orchestration

Effective metadata management and orchestration are crucial for handling a dataset of this magnitude and complexity. The system must be able to track the state of each video and audio segment through the entire pipeline, from raw download to final packaged shard, and manage the distributed workers that perform the processing. The architecture uses a combination of a centralized database (Supabase) for metadata storage and a distributed worker system with robust task management to ensure the pipeline runs smoothly and efficiently. This setup allows for parallel processing of videos, fault tolerance in case of worker failures, and the ability to monitor the progress of the entire operation. The metadata stored in Supabase is not just for bookkeeping; it is an integral part of the data curation process, containing the quality scores and other annotations that will be used to filter the dataset and create the final training splits.

#### 2.3.1. Supabase for Centralized Metadata Storage

**Supabase**, an open-source Firebase alternative, is used as the central repository for all metadata associated with the dataset. For each of the over **500,000 YouTube videos**, a corresponding entry in a Supabase table stores a wealth of information. This includes the video ID, language, source channel, and the initial classification as a podcast or non-podcast. As the video progresses through the pipeline, the metadata is continuously updated. After the single-speaker segmentation and cleaning, the metadata for each video will include a list of its constituent audio segments, along with their durations, file paths in Cloudflare R2, and the results of the quality analysis (e.g., confidence scores from the GSE model, music detection scores from PANN). This centralized metadata store is essential for several reasons. First, it provides a single source of truth for the state of the entire dataset, allowing for easy querying and analysis. For example, one could quickly determine the total hours of high-quality, single-speaker audio available for a specific language. Second, it enables the distributed worker system by providing a way to assign tasks and track their completion. Workers can query Supabase for the next available video to process and update the status once they are done. Finally, the metadata itself becomes a valuable asset for the final stages of data preparation, as it contains all the information needed to create filtered datasets and training/validation splits based on various quality criteria.

#### 2.3.2. Distributed Worker System with Row-Level Locks

To process the massive dataset in a reasonable amount of time, the data cleaning and packaging pipeline is designed to be executed by a distributed system of workers. These workers run on GPU-enabled machines and are responsible for picking up raw video files from Cloudflare R2, running the entire cleaning pipeline, and uploading the resulting `.tar` shards back to R2. To manage this distributed process and avoid race conditions where multiple workers might try to process the same video, the system uses **row-level locks in Supabase**. When a worker is ready to process a new video, it queries the Supabase database for a video with a "pending" status. It then attempts to acquire a lease on that video's row by updating its status to "processing" and adding a timestamp. This operation is atomic, ensuring that only one worker can successfully acquire the lease. The worker then has a set amount of time (the lease duration) to complete the processing. If the worker fails or crashes, the lease will expire, and another worker can pick up the task. This mechanism provides fault tolerance and ensures that all videos in the dataset are eventually processed. The system also uses **deterministic shard IDs** for idempotent retries, meaning that if a worker fails and the task is retried, the same output will be generated, preventing duplicate or inconsistent data. This distributed, lock-based orchestration is a robust and scalable way to manage the complex, long-running tasks involved in preparing the dataset for training.

#### 2.3.3. Final Validation and Shard Creation

The final stage of the data pipeline is to perform a last round of validation checks on the cleaned and processed data and to create the final sharded datasets for training. This involves several steps. First, a validation script is run to check for any missing metadata or inconsistencies in the data. This script verifies that every audio segment has a corresponding entry in the metadata file and that all the required fields are present. Next, the data is sharded into smaller, more manageable chunks. This is done to facilitate distributed training, as each training process can work on its own shard of the data. The sharding process is deterministic and idempotent, meaning that the same shard will always be created from the same input data, which is important for ensuring reproducibility. The final sharded datasets, in WebDataset format, are then uploaded back to a dedicated R2 bucket, ready to be pulled by the training jobs. This final validation and sharding step is crucial for ensuring that the data is in a clean, consistent, and ready-to-use state before it is fed into the TTS model.

## 3. Model Architecture and Training Strategy

The development of a human-like TTS LLM requires a sophisticated model architecture and a multi-stage training strategy that can effectively leverage the massive, curated dataset. The proposed approach is heavily inspired by the successful methodologies of state-of-the-art models like **Inworld TTS-1** and **FireRedTTS-2**, which have demonstrated the power of large-scale pre-training followed by targeted fine-tuning and reinforcement learning. The core of the architecture will be a powerful LLM backbone that learns to generate discrete audio tokens from a neural codec. The training process is designed to be a curriculum, starting with learning the fundamental properties of speech from the entire filtered dataset, then specializing on high-quality, transcribed data, and finally, being aligned with human preferences for naturalness and quality. This section outlines the key components of this strategy, including the selection and training of the audio codec, the multi-stage TTS LLM training pipeline, and the incorporation of advanced features like multi-speaker dialogue and emotional control.

### 3.1. Audio Codec Selection and Training

The choice of audio codec is a foundational decision that will impact the entire TTS system. The codec is responsible for compressing the high-dimensional audio waveform into a low-dimensional sequence of discrete tokens, which can then be modeled by the LLM. The quality of the codec directly affects the upper bound of the fidelity of the synthesized speech. Therefore, a careful evaluation of available codecs is necessary to select the one that offers the best trade-off between compression rate, reconstruction quality, and computational efficiency for the project's specific needs.

#### 3.1.1. Candidate Codecs: X-Codec 2.0, Mimi, SNAC, BiCodec, Encodec

The project will evaluate a range of state-of-the-art neural audio codecs that are suitable for TTS applications. The primary candidates include:

*   **X-Codec 2.0**: A recent and highly performant codec known for its high-quality reconstruction and efficiency, which has shown strong results in TTS benchmarks.
*   **Mimi**: A codec developed by Kyutai, designed for high-fidelity audio generation with a focus on efficiency and low latency.
*   **SNAC (Speech Neural Audio Codec)** : A codec specifically designed for speech, offering a good balance between quality and compression.
*   **BiCodec**: A bidirectional codec that can be used for both encoding and decoding, potentially offering more flexibility in the training process.
*   **Encodec**: A widely used and well-established codec from Meta, known for its high-quality reconstruction, though it can be computationally intensive.

The evaluation will be based on several criteria, including the **reconstruction quality** (measured by metrics like PESQ and STOI), the **token rate** (lower is better for LLM context length), the **codebook size**, and the **robustness** to the diverse acoustic conditions present in the Indic language dataset. The goal is to select a codec that can faithfully reconstruct the nuances of the human voice, including its emotional and prosodic variations, while keeping the token sequence short enough to be efficiently processed by the LLM.

#### 3.1.2. Training Strategy on Filtered, High-Confidence Audio

Once a codec is selected, it will be trained on the curated, high-confidence audio dataset created through the GSE-based filtering process. Training the codec on clean, high-quality data is crucial for ensuring that it learns a robust and accurate representation of speech. The training process will involve feeding the audio segments into the codec and optimizing its parameters to minimize the reconstruction loss. This will be a large-scale training job, requiring significant GPU resources to process the tens of thousands of hours of audio data. The resulting trained codec will serve as the "vocabulary" for the TTS LLM, translating between the continuous audio space and the discrete token space of the language model. The quality of this codec will be a direct determinant of the final TTS model's performance, making this a critical step in the overall pipeline.

#### 3.1.3. Segment Length Strategy for Training (6-second windows)

To manage the computational load and to create a consistent training experience, a specific strategy for handling audio segments of varying lengths has been devised. The strategy is to standardize on a **6-second window** for all training samples. This is achieved through a multi-bucket approach:

*   **Segments < 3 seconds**: These are discarded as they are too short to contain meaningful prosodic information.
*   **Segments 3-6 seconds**: These are padded to a 6-second duration during the training process.
*   **Segments 6-12 seconds**: A random 6-second window is extracted from these segments for each training step.
*   **Segments > 12 seconds**: These are split into multiple 6-second segments to ensure that no audio data is wasted.

This strategy ensures that the model is always trained on fixed-length inputs, which simplifies the training process and allows for efficient batching. The choice of a 6-second window is a trade-off between providing enough context for the model to learn long-range prosodic dependencies and keeping the computational requirements manageable.

### 3.2. TTS LLM Training Pipeline (Inspired by Inworld TTS & FireRedTTS)

The training of the TTS LLM will follow a multi-stage pipeline, inspired by the successful strategies of models like Inworld TTS-1 and FireRedTTS-2. This approach allows the model to first learn the fundamental properties of speech from a massive amount of unlabeled data, then to specialize on the task of text-to-speech with high-quality labeled data, and finally to be fine-tuned for human preference. This staged approach is crucial for building a robust and high-quality TTS system.

| Stage | Name | Data | Objective | Key Techniques |
| :--- | :--- | :--- | :--- | :--- |
| **1** | **Pre-training** | Large-scale, filtered audio (codec tokens) | Learn fundamental speech representations | Next-token prediction on audio tokens |
| **2** | **Supervised Fine-Tuning (SFT)** | High-quality, transcribed audio-text pairs | Learn the text-to-speech mapping | Cross-entropy loss on audio tokens, conditioned on text |
| **3** | **Reinforcement Learning (RL) Alignment** | Smaller, high-quality dataset with human feedback | Optimize for human preference (naturalness, quality) | Group Relative Policy Optimization (GRPO), composite rewards |

*Table 1: A summary of the three-stage TTS LLM training pipeline, outlining the data, objectives, and key techniques for each stage.*

#### 3.2.1. Stage 1: Pre-training on Large-Scale, Filtered Audio Data

The first stage is **pre-training**, where the LLM is trained on the massive, curated dataset of high-confidence audio segments. In this stage, the model learns to predict the next audio token in a sequence, without any text conditioning. This is an unsupervised learning task that allows the model to develop a deep understanding of the statistical properties of human speech, including phonetics, prosody, and speaker characteristics. This stage is analogous to the pre-training of a text-based LLM on a large corpus of text. The goal is to create a powerful "speech foundation model" that has a strong prior on what constitutes natural-sounding audio. This pre-trained model will then serve as a robust starting point for the subsequent, more specialized training stages.

#### 3.2.2. Stage 2: Supervised Fine-Tuning (SFT) on High-Quality Transcribed Data

The second stage is **Supervised Fine-Tuning (SFT)** , where the pre-trained model is trained on a smaller, high-quality dataset of audio-text pairs. This is where the model learns the core task of text-to-speech synthesis. The model is conditioned on the input text (the transcript) and is trained to generate the corresponding sequence of audio tokens. This stage requires a very clean and accurate transcribed dataset, which can be obtained through the alternative transcription methods discussed earlier (e.g., YouTube CCs, Wikipedia alignment). The SFT process will fine-tune the model's parameters to align the generated speech with the semantic content of the text, teaching it correct pronunciation, intonation, and rhythm. The quality of the SFT data is paramount, as any errors in the transcripts will be learned by the model.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Previewing reference files**

I'm planning to quickly review claude.md and gpt_research.md to extract key points relevant to the task before synthesizing.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Bharath, this is an incredibly well-thought-out pipeline, and you're asking exactly the right questions at this stage. Let me give you a comprehensive strategy for transcription that addresses all your concerns about scripts, code-mixing, emotions, and validation.

## Core Transcription Strategy

### Multi-Pass Architecture with Validation

```
┌─────────────────────────────────────────────────────────────────────────┐
│                    TRANSCRIPTION PIPELINE                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                          │
│  Audio Segment                                                           │
│       │                                                                  │
│       ▼                                                                  │
│  ┌─────────────────┐                                                    │
│  │ Gemini-3-Flash  │──────────────────┐                                 │
│  │ (Primary ASR)   │                  │                                 │
│  └────────┬────────┘                  │                                 │
│           │                           │                                 │
│           ▼                           ▼                                 │
│  ┌─────────────────┐         ┌─────────────────┐                       │
│  │ Forced Aligner  │         │ Audio Classifier │                       │
│  │ (MFA/Custom)    │         │ (emotion2vec)    │                       │
│  └────────┬────────┘         └────────┬────────┘                       │
│           │                           │                                 │
│           ▼                           ▼                                 │
│  ┌─────────────────────────────────────────────┐                       │
│  │         Confidence Aggregation              │                       │
│  │  • Alignment Score (phoneme boundaries)     │                       │
│  │  • Event Tag Agreement                      │                       │
│  │  • Language Detection Confidence            │                       │
│  └────────────────────┬────────────────────────┘                       │
│                       │                                                 │
│           ┌───────────┴───────────┐                                    │
│           │                       │                                    │
│     Score > 0.85            Score < 0.85                               │
│           │                       │                                    │
│           ▼                       ▼                                    │
│     ┌──────────┐          ┌─────────────────┐                         │
│     │  ACCEPT  │          │ Gemini-3-Pro    │                         │
│     └──────────┘          │ (Arbitration)   │                         │
│                           └────────┬────────┘                         │
│                                    │                                   │
│                                    ▼                                   │
│                           ┌─────────────────┐                         │
│                           │ Final Decision  │                         │
│                           │ • Accept        │                         │
│                           │ • Flag for      │                         │
│                           │   manual review │                         │
│                           │ • Discard       │                         │
│                           └─────────────────┘                         │
└─────────────────────────────────────────────────────────────────────────┘
```

---

## The Hallucination Problem & Solutions

You're absolutely right that temperature=0.0 doesn't eliminate creativity. Here's why and what to do:

### Why Temperature=0 Isn't Enough

LLMs have learned patterns like "completing thoughts" or "making sentences grammatically correct." Even at temperature=0, the model picks the *most likely* token, which might still be a hallucinated word that "fits" better than the actual spoken word.

### Multi-Layered Hallucination Prevention

**1. Prompt Engineering (Necessary but Insufficient)**
```
You are a VERBATIM transcription system. Your task is to transcribe EXACTLY what is spoken.

CRITICAL RULES:
- Transcribe ONLY sounds you hear. Never add words to "complete" a thought.
- If speech is unclear, use <unclear> tag, DO NOT guess.
- If speech is cut off, transcribe exactly what was said, don't complete it.
- Transcribe filler words: "um", "uh", "hmm" exactly as spoken.
- Transcribe false starts: "I was go- I was going" exactly.
- DO NOT correct grammar. If speaker says "I goes there", write "I goes there".
- DO NOT add punctuation that changes meaning.

OUTPUT FORMAT: [Your structured JSON]
```

**2. Forced Alignment Validation (Critical)**

This is your primary defense against hallucinations. Here's the strategy:

```python
# Pseudo-pipeline for validation
def validate_transcript(audio_path, transcript, language):
    # 1. Get phoneme-level alignment
    alignment = forced_aligner.align(audio_path, transcript, language)

    # 2. Calculate metrics
    alignment_score = alignment.confidence
    coverage = aligned_duration / total_audio_duration
    gap_ratio = sum(unaligned_gaps) / total_duration

    # 3. Flag issues
    issues = []
    if coverage < 0.90:
        issues.append("low_coverage")
    if gap_ratio > 0.15:
        issues.append("alignment_gaps")
    if alignment_score < 0.80:
        issues.append("low_confidence")

    return {
        "valid": len(issues) == 0,
        "score": alignment_score,
        "issues": issues,
        "word_alignments": alignment.words  # For debugging
    }
```

**3. Language-Specific Forced Aligners**

| Language | Recommended Aligner | Notes |
|----------|-------------------|-------|
| Hindi | MFA with Hindi acoustic model | Train custom on 50-100h verified data |
| Telugu/Tamil/Kannada | Custom trained | MFA lacks good Dravidian support |
| Bengali/Gujarati/Punjabi | MFA with custom lexicon | Need phoneme dictionary |

**Should you train your own aligners?**

**Yes, absolutely.** Here's the bootstrap strategy:

1. Take 500-1000 segments per language where you have high confidence (e.g., from clean studio recordings or where multiple ASR systems agree)
2. Manually verify a subset (~100 segments)
3. Train language-specific acoustic models using MFA or Kaldi
4. Use these to validate the Gemini transcripts

---

## Script Strategy: The Complete Framework

This is where most teams mess up. Let me give you a robust framework.

### The Fundamental Decision: Native Script vs Romanized

**For TTS training, native script is superior because:**
- Unambiguous pronunciation (Hindi "kh" = ख or क्ह?)
- Consistent grapheme-to-phoneme mapping
- No transliteration variance (is it "ghar" or "ghr" or "gher"?)

**However, you need BOTH for flexibility.**

### Recommended Output Schema

```json
{
  "segment_id": "videoID_seg_001",
  "duration_ms": 4523,
  "language_primary": "te",

  "transcripts": {
    "native_verbatim": "అరే అలా కాదు, దీన్ని హిందీలో 'मैं सेब खाता हूँ' అంటారు",

    "native_normalized": "అరే అలా కాదు, దీన్ని హిందీలో 'मैं सेब खाता हूँ' అంటారు",

    "roman_verbatim": "are ala kaadu, deenni hindi lo 'main seb khaata hoon' antaaru",

    "roman_normalized": "are ala kadu, dinni hindi lo 'main seb khata hun' antaru",

    "with_events": "అరే [pause] అలా కాదు, దీన్ని హిందీలో [code_switch:hi] 'मैं सेब खाता हूँ' [/code_switch] అంటారు",

    "with_language_tags": "<te>అరే అలా కాదు, దీన్ని హిందీలో '<hi>मैं सेब खाता हूँ</hi>' అంటారు</te>",

    "phoneme_sequence": "a r eː | a l aː | k aː d u | d iː n n i | ..."
  },

  "metadata": {
    "contains_code_switch": true,
    "code_switch_languages": ["te", "hi"],
    "contains_roman_in_native": false,
    "detected_events": [
      {"type": "pause", "start_ms": 423, "end_ms": 678}
    ],
    "alignment_score": 0.92,
    "transcription_confidence": 0.89
  }
}
```

### Handling Code-Mixed Scenarios

Let me address your specific examples:

**Scenario 1: Roman Hinglish**
```
Input: "are bhaai, kya kar rhe ho?"
```

This is Hindi written in Roman script. Your model needs to:
1. Recognize this is NOT English
2. Know to pronounce it as Hindi

**Solution:** Language tagging with script indicator
```json
{
  "native": "अरे भाई, क्या कर रहे हो?",


Summary: auto


**Deciding to review research notes**

I'll quickly check gpt_research.md to capture any additional relevant points that could inform the approach before proceeding.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Strategy for Accurate Multilingual Transcription for TTS Data
High-Accuracy Transcription Pipeline
1. Multi-ASR Ensemble and Tiered Models: To achieve high transcription accuracy on ~150k hours across 12 languages, a multi-model approach is ideal. Run multiple ASR systems in parallel and fuse their outputs. For example, Alibaba’s CosyVoice 3 pipeline passes audio through several recognizers and uses cross-validation to get a reliable transcript
. Research on pseudo-labeling shows that combining three ASR models (e.g. Whisper, Nemo, Icefall) and taking a word-level consensus yields near-human accuracy
. In practice, you can adopt a tiered strategy: use a fast model (like a “Gemini-3-Flash” or Whisper-large) for all data, then route difficult segments (low confidence or high disagreement) to a stronger model (“Gemini-3-Pro”) for re-transcription. This hybrid approach reserves the expensive model for the hardest cases, maximizing efficiency. 2. Confidence Checking and Forced Alignment: Implement a validation step to flag uncertain transcripts. Many ASR models provide a confidence score per word or utterance. Additionally, you can run a forced aligner (e.g. Montreal Forced Aligner or a Wav2Vec-based aligner) to align the transcript to the audio. If alignment fails or many words cannot align, it indicates potential errors. Forced aligners are typically language-specific, so you may need acoustic models for each language (MFA offers pre-trained models for several languages, or you can train your own). Flag low-confidence or poorly aligned segments for review or re-processing with a more powerful model. This two-pass validation (ASR confidence + alignment) ensures that only high-quality transcriptions enter your dataset. 3. Cross-Validation with Multiple Models: Expand on the ensemble idea by comparing outputs from different ASRs. If two transcripts perfectly agree (zero CER difference), the text is likely correct
. If they disagree, identify the “confusion regions” (words with mismatches) and handle them specially
. One simple approach is majority voting at the word level when three models are used (choose the word that two of the three agree on)
. For more complex reconciliation, consider using an LLM in text-only mode: feed it the multiple candidate transcripts with markers highlighting differences, and prompt it to choose the most plausible combination. (In fact, a study found that fine-tuning an LLM on ASR outputs can learn to resolve these disagreements effectively
.) By leveraging multi-ASR fusion, you can reduce hallucinations and errors that any single model might produce, achieving transcripts close to human quality
. 4. Strict Prompting to Prevent Hallucination: When using a large language model (LLM) for transcription, constrain its behavior through the prompt and decoding settings. Clearly instruct the model not to generate anything other than the exact speech content. For example, a system prompt might say: “Transcribe the audio exactly as spoken, without paraphrasing or adding extra commentary. Use appropriate language script and include markers like [laugh] only if truly present.” Providing a few examples of correct transcriptions vs. incorrect (hallucinated) ones can reinforce this guideline. Use a low sampling temperature (0.0 or near 0) to minimize randomness. Also, request output in a structured format (like JSON) – this naturally limits free-form creativity. For instance, asking the LLM to output: {"verbatim": "...", "roman_normalized": "...", "verbatim_emotions": "..."} leaves little room for imaginative tangents. Big labs often rely on such prompt engineering and even slight fine-tuning of LLMs on transcription tasks to keep them factual. If the model is still too “chatty,” consider fine-tuning it briefly on a small supervised set of audio & accurate transcript pairs to bias it toward literal transcription (essentially instructing it on the task before scaling to all data).
Handling Multilingual and Code-Switched Speech
1. Language Identification and Script Selection: A key challenge is code-switching – many of your segments mix languages (e.g. Hinglish: “Are bhai, kya kar rahe ho?”). The transcription system should output each portion in the correct script for that language. The best practice is to identify the spoken language at the word or phrase level and use that language’s writing system. For example, if a Hindi speaker says “अरे भाई, क्या कर रहे हो?”, the transcript should be in Devanagari, even if similar phonetics could be written in English letters. Likewise, Telugu speech should be in Telugu script. Most multilingual ASR models (like Whisper) will automatically output in the native script of the language if they’re trained on it. For code-switched utterances, you might need to post-process or instruct the model explicitly: e.g., “Use Devanagari for Hindi words, Tamil script for Tamil, and Latin script for English.” If the ASR can’t handle this natively, a fallback is to use a language-tagging approach – insert markers like <lang hi>...</lang> around Hindi text, then convert to Devanagari with a transliteration tool later. However, direct script output is preferable to avoid an extra conversion step (which can introduce errors). Notably, CosyVoice 3 emphasized robust text normalization across languages and text format diversity, ensuring the model saw various scripts and styles during training
. This means your TTS model will be more robust if it’s trained on transcripts that faithfully represent each language in its own script, even within a single sentence. 2. Transliteration for Romanized Segments: In some cases, speakers might spell things out or use English for local terms. If your transcripts end up with romanized native-language text (e.g. “arey anna, em chestunnav?” in Latin letters), consider transliterating that to the native script (“అరే అన్నా, ఏమి చేస్తున్నావ్?” in Telugu) for consistency. There are libraries and models for Indic transliteration which you can use in an automated pipeline. An alternative strategy is to include a special notation for such cases (the “te_EN” approach you mentioned, to denote Telugu-in-English-letters). However, adding such tags in the training data could complicate the model’s input format. It’s usually cleaner to have unified transcripts in one script per language. If an English word is truly being used (like a name or technical term), you can leave it in English. But if it’s just a phonetic Hindi phrase typed in English letters, convert it to हिंदी. The goal is to preserve the language identity of each word so the TTS model knows how to pronounce it correctly. Big multilingual TTS systems often rely on either the script or explicit language tokens to handle code-switching; for example, some use inline tokens like <|hi|> to force Hindi pronunciation for a following Latin script text. In your case, since you’re generating transcripts from scratch, it’s optimal to output the correct script and avoid the need for such tokens. You can always fall back to language tags if needed for ambiguous cases, but try to minimize mixed-script scenarios by resolving them in the transcript generation step. 3. Text Normalization (Numbers, Acronyms, etc.): Don’t forget to normalize non-speech content in transcripts. This includes expanding numbers, dates, acronyms, and formatting. A phrase like “I have 12,345 rupees” should be transcribed as “I have twelve thousand three hundred forty-five rupees” (or the equivalent in the target language) to avoid the TTS model having to learn how to say numeric digits. You already noted you can handle numeric normalization with regex or rules – that’s a good idea. Also consider common ITN (Inverse Text Normalization) cases: expanding abbreviations (“Dr.” to “Doctor”), units, etc., in whatever language context they appear. Consistent normalization is important so that your model training data has a uniform, speech-like text form (no stray symbols or untranslated English tokens unless intended). As CosyVoice 3 noted, incorporating varied text normalization and formats during training improved robustness
 – so you might even include both a “verbatim” version and a “normalized” version of transcripts in training (though primarily you’ll train on normalized transcripts for TTS). In summary, ensure each transcript line reads like natural, fully-spoken text in that language’s orthography.
Incorporating Emotion and Non-Speech Markers
1. Annotating Laughter and Other Events: To create a truly human-like TTS, you may want to include phenomena like laughter, coughs, and emotional tone indicators in the training data. This means inserting special tokens like [laugh], [cough], or emotion tags (e.g. [sad], [excited]) at the appropriate places in transcripts. Many state-of-the-art models have done exactly this. For example, CosyVoice 3’s training data included fine-grained inline tags such as “[laughter]” to indicate non-verbal sounds
. Inworld’s TTS-1 similarly introduced a set of markup tokens for speaking style and non-verbal vocalizations during a later fine-tuning stage
. These tags let the model learn to produce those cues in audio. To get these annotations, you have a couple of options:
Automatic detection: Use an audio-based classifier to detect laughter, applause, music, etc. You already use PANN for music detection; consider using a pre-trained sound event detection model for laughter or crowd noises. If the audio segment triggers a high confidence for laughter, insert a [laugh] token at that point in the transcript. Some ASR models (or Whisper with certain settings) might also insert tags like “[LAUGHTER]” – if so, leverage that output.
LLM-based annotation: Since you plan to use a powerful model for transcription, you could instruct it to include tags for audible non-speech events. For instance: “Transcribe the speech, and if the speaker laughs or coughs, insert [laugh] or [cough] at that position.” A well-trained model should do this reasonably, but be cautious: the model might hallucinate an event tag if it thinks something was funny or if there’s an ambiguous noise. To mitigate false positives, cross-verify with audio: e.g., if the transcript has a [laugh] tag, ensure the audio segment indeed contains laughter (perhaps by a simple energy burst check or a secondary classifier). You can filter out tags that aren’t corroborated by audio evidence.
2. Defining a Controlled Tag Set: Limit the set of emotion and event tokens to a defined list – you mentioned about 10 stable tags (e.g. [happy], [sad], [angry], [laugh], [pause], etc.). Consistency is key: the model should see these tags frequently and in correct contexts. Big labs handle this by collecting or generating paired examples. For instance, Inworld’s team constructed a special dataset of ~100k paired samples where each neutral utterance was paired with a stylized version containing emotion tags (like an angry version with [angry] tag)
. This taught their model how tags correspond to delivery. In your case, since you already have a large amount of real speech, you can incorporate the tags in transcripts wherever applicable. The “verbatim_emotions” field you proposed would contain these annotations. During model training, you can decide to either train the base model with these tags included from the start, or introduce them in a fine-tuning stage:
Immediate integration: If your automated tagging is accurate, you can include tags in the transcripts from the get-go. The model will learn them as part of its language modeling. CosyVoice 3, for example, had a portion of its pretraining data (~5,000 hours) with instructions and emotions, meaning the base model already saw emotion tokens in context
.
Post-training fine-tune: Alternatively, you train the TTS model on plain speech first, then perform a smaller supervised fine-tuning (SFT) with a focused dataset that includes emotion/style tags. This is what Inworld did – after main training, they fine-tuned with LoRA on markup-enriched data to inject fine-grained control
. This approach can sometimes yield more stable base performance (since the model isn’t trying to learn tags from noisy data), then add controllability later.
Given your inclination to “get data at once”, a practical plan is to generate multiple transcript versions now: one pure verbatim, and one with emotion tags. You could even create an augmented training set where, say, 20% of samples have the emotion tags inserted (to teach the model the concept), and the rest are normal (to preserve base quality)
. This mirrors Inworld’s strategy of mixing neutral and marked-up examples. It’s wise to keep the tag vocabulary limited and standardized (no free-form descriptions, just [laugh] not [laugh_hysterically] unless you plan to consistently use a wide range). Simplicity will make it easier for the model to learn the association. 3. Quality Control for Tags: As you suspected, an LLM might occasionally hallucinate an emotion tag (e.g., insert [sad] because the content seems sad, even if the speaker’s tone was neutral). To control this, implement a filtering rule: non-verbal tags like [laugh] or [cough] should only be present if the audio truly contains that sound. For emotional tone tags like [angry] or [sarcastic], it’s harder to algorithmically verify, but you might base it on an emotion recognition model that classifies the speaker’s tone. If the confidence is below a threshold, remove or replace the tag. Essentially, treat LLM-generated emotion tags as suggestions that need verification. It’s better to have slightly under-tagged data than to train the model on hallucinated events.
Generating and Using Multi-Form Transcripts
You proposed outputting transcripts in multiple forms – verbatim native script, romanized, and with emotion tags. This is an excellent idea for flexibility. Here’s how to tackle it:
Verbatim (Native Script): This is the core transcription in the language’s own script, with basic punctuation. Use this for primary TTS and ASR training. Ensure it’s as accurate as possible and normalized. All the strategies above (ensemble ASR, alignment, etc.) apply chiefly to this output.
Romanized (Latin Script): Having a romanized version of each transcript can be useful for certain applications (like if you want to train an ASR that takes Latin characters for all languages, or simply to help non-native readers). To get this, you can either prompt the LLM to produce it (for instance, by asking it to fill the "roman_normalized" field) or generate it after-the-fact with a transliteration library. A large multilingual model will often know how to phonetically spell words in Latin script, but it might not follow a strict standard. You may get variations (“e.g. kam kar rahe ho vs kam kar rahe ho? punctuation or spelling differences). A rule-based transliterator (like ITRANS or ISO 15919 for Indic languages) would give consistent results, but those can sometimes be awkward for non-technical use. A compromise is to use the LLM’s output but do a spot check or minor normalization on it (for example, ensure all diacritics are removed if you want plain ASCII, etc.). This romanized text is mostly for reference – it likely won’t be used to train the final TTS model unless you plan to allow Latin-script input for languages like Hindi (which is uncommon for a production TTS). If code-mixed content includes true English words, they will appear unchanged here, which is fine.
Verbatim with Emotions (or “Rich Transcript”): This version would mirror the native script transcript but include the special tokens for laughs, emotions, background noises, etc. It might also include enhanced punctuation or casing if you decide to add that (for instance, maybe in the verbatim version you keep minimal punctuation, but in the enriched version you could add “…” for pauses or capitalize proper nouns – it depends on what your model is expected to handle). To generate this, the LLM can be prompted to include those markers (as discussed above). Another approach is a two-step process: get the clean verbatim transcript first, then run a second pass where you insert tags based on external detectors (e.g., insert [laugh] at 3.2s because a laughter was detected there, or prepend [angry] at the start if an emotion model says the speaker’s tone is angry throughout the segment). Doing it in two steps can actually be safer – it separates speech-to-text from paralinguistic tagging, potentially reducing transcription errors. However, a capable multimodal model could do both in one go. Choose the approach that gives the highest precision for tags.
When you have these multiple forms, you can store them in your metadata (e.g., in Supabase or alongside the audio in the .json). They’re all derived from the same audio, so you can reference them as needed. For TTS training, you’ll likely start with the plain verbatim transcripts (no tags) for the bulk of pretraining. Later, you might use the verbatim_emotions version in a specialized fine-tune to teach the model those tags and emotional style control
. The romanized text might not directly go into model training, but it’s a nice-to-have for analysis or for training an auxiliary model (like a language identification or as a fallback ASR input).
Best-Practice Takeaways
To summarize the strategy with some best practices from industry research:
Use multiple ASR engines and cross-validation to maximize transcription quality. This is a proven approach in CosyVoice 3, which used “cross-validated ASR transcription using multiple models” to create its 1M-hour training corpus
. Recent research goes further with ensemble decoding and LLM-based error correction, showing that an ensemble+LLM can match human transcription quality on large datasets
. Implement as much of this as feasible – even a simpler two-model vote can boost accuracy.
Rigorous filtering and QA: After transcription, apply filtering similar to large TTS projects. Inworld’s TTS-1, for instance, filtered out segments with likely transcription errors (e.g. extremely high or low speaking rates, or weird punctuation-only transcripts)
. You should discard or re-transcribe any segments that look suspicious (e.g., a 10-second audio that got only “[music]” as text, or an obviously wrong language). Using your metadata like audio-text length ratio is helpful – if an audio segment is 5 seconds but the transcript has 50 characters, something is off. This kind of filtering will ensure your final 150k hours are truly usable.
Handle code-mixing robustly: Make a decision and stick to it regarding script usage. Most likely, this means all Hindi in Devanagari, all Tamil in Tamil script, etc., even if the original speaker peppered English words – keep those English words in Latin script as they are English. If a speaker says a full English sentence inside a Hindi segment, just transcribe that sentence in English. Your TTS model (if truly multilingual) should then naturally switch to the English “voice” for that bit. Consistency in how you represent mixed-language content will be important for the model to learn code-switching behavior.
Leverage emotion markers as a controllability feature: Including things like [laugh] or <happy> in training data can significantly enhance the expressiveness of the TTS model. CosyVoice 3 and others explicitly added such markup to gain fine-grained control
. Plan how you’ll use these: if you include them in training from the start, ensure they are correct and not too sparse. Alternatively, create a smaller high-quality set of data with these annotations for a later fine-tune. You might even do both: a bit in pretraining, plus a focused fine-tune, as a belt-and-suspenders approach for the model to really learn it.
Finally, continuously evaluate WER/CER on a validation subset across languages as you iterate on the transcription process. If, say, the Telugu transcripts are coming out with 15% WER against a known test set, whereas Hindi is 5%, you might decide to route more Telugu data through the “Pro” model or do an extra QC pass for Telugu. This feedback loop will guide where to invest more effort (as you noted, perhaps using the larger model on “less feasible” languages). The end goal is a massive collection of transcripts that are as close to ground-truth as possible, in the right script, with optional metadata tags – a solid foundation for training a high-fidelity, multilingual, expressive TTS LLM. Sources:
Alibaba (Tongyi Lab) – CosyVoice 3 technical summary (1M hours, multi-model ASR pipeline, emotion tags)
.
Inworld AI – TTS-1 Technical Report (200k-hour filtered dataset, markup fine-tuning for emotions)
.
Y. Chen et al. – Multi-ASR Fusion for Pseudo-Labeling (ensemble of Whisper/Nemo/Kaldi with LLM refinement to approach human transcription quality)
.
Montreal Forced Aligner docs (usage of forced alignment for various languages).
General ASR best practices in handling code-switching and text normalization.


**Planning balanced transcript strengths**

I'm gathering positive points on transcript types from gemini-flash/pro models and synthesizing their strengths from multiple research files to suggest a hybrid strategy leveraging cross-validation and forced alignment, while ensuring a non-discriminative, balanced perspective.


**Outlining transcript types and model strengths**

I've detailed positive transcript types to request from Gemini Flash and Pro, clarified each model’s strengths, and structured a hybrid pipeline with validation plans and caution areas to aim for near 3% WER on Indic data realistically while supporting multi-view, multi-script, and code-mixed scenarios.


**Transcript Bundle (best positive points)**

- **`orthographic_mixed`**: Combines native script for Indic parts with Latin script for English, enabling effective handling of code-mix and broad generalization.
- **`orthographic_native`**: Provides canonical script transcription per language, aiding precise phonetic modeling, alignment, and evaluation; ideal primary training target.
- **`romanized`**: Offers consistent transliteration into Latin script, useful for debugging, search, fallback on roman input, and ASCII-only contexts.
- **`normalized_for_tts`**: Expands numbers and abbreviations into spoken forms, reducing ambiguity and lowering WER during normalized evaluation or TTS training.
- **`language_spans`**: Marks language and script spans with tags, supporting language identification, transliteration experiments, and code-mix handling within utterances.
- **`events`**: Separate event lists enable controllability (e.g., speaker turns, noises) without contaminating core text; accuracy improved by audio-based event detectors.

**Gemini Flash (positive points)**

- **High throughput**: Efficient processing suitable for large volumes and primary transcript generation.
- **Cost-effective**: Low computational expense, facilitating broad coverage and scalability.
- **Schema consistency**: Produces stable, structured JSON outputs aiding automated downstream use.
- **Strong on straightforward segments**: Excels with high-quality single-speaker or clear audio recordings.
- **Good baseline for all transcript types**: Adequate generation of core fields including multilingual and normalized transcripts.

**Gemini Pro (positive points)**

- **Advanced instruction adherence**: Handles complex prompts and strictly structured outputs more reliably.
- **Superior code-mix and diarization**: Better disambiguation in mixed languages and multi-speaker scenarios.
- **Robustness to low SNR and challenging audio**: Performs well with noisy, overlapping, or otherwise difficult segments.
- **Improved event tagging and transcription precision**: More consistent on diarization, event boundary placement, and complex annotations.
- **Ideal for escalation and arbitration**: Used to resolve Flash disagreements or for detailed meta-tasks (e.g., script mapping).

**Hybrid Strategy for Lowest WER**

- **Audio quality gating**: Pre-filter using signal-to-noise ratio (SNR), voice activity detection (VAD), and audio event presence.
- **Run LID and segmentation**: Identify languages and audio segments for targeted transcription passes.
- **Primary pass with Gemini Flash**: Generate structured transcripts with multi-view outputs and language spans.
- **Cross-check with local ASR models** (Whisper large v3, IndicConformer, MMS): Provide ensemble input to improve robustness.
- **Alignment with forced aligners** (CTC, IndicMFA): Validate transcript coverage and timing consistency.
- **Consensus and arbitration**: Employ ROVER or LLM-based voting to combine outputs; escalate uncertain segments to Gemini Pro.
- **Discard or manually audit irreconcilable audio**; store transcripts with normalized and multi-script metadata for training use.

**Validation and WER Targets**

- **Gold-standard human-labeled set**: Curate high-quality reference transcripts per language for calibration.
- **Open-source evaluation sets**: Use IndicSpeech, Common Voice, FLEURS, and OpenSLR Indic subsets for benchmarking.
- **Normalization-aware metrics**: Report WER on normalized/transliterated text and character error rate (CER) for script-rich languages.
- **Threshold tuning**: Adjust gating and arbitration thresholds to balance recall and precision on high-confidence segments.
- **Realistic WER goals**: <3% WER achievable on curated high-quality subsets; full in-the-wild podcasts expected around 5–8%.
- **Continuous drift monitoring**: Track model degradation or domain shifts with regular evaluation.

**High-Risk Areas Needing Care**

- **Code-mixed utterances with mixed scripts and romanization ambiguities** requiring clear span tagging and escalation.
- **Overlapping speech, music, and noisy backgrounds** that challenge speaker diarization and alignment.
- **Named entities, numbers, and domain-specific terms** prone to transcription errors or normalization mismatch.
- **Long segments and poor segmentation** increasing error propagation risks.
- **LLM-generated hallucinations or inconsistent output** necessitating consensus gating.
- **Script normalization issues** (e.g., Unicode normalization) affecting token matching and error metrics.

**Suggestions for Next Steps**

1. Define detailed JSON schemas and prompt templates for Gemini Flash and Pro to ensure consistent multi-view transcript outputs.
2. Establish gating thresholds, ROVER configuration, and escalation criteria for pipeline decision-making.
3. Assemble curated evaluation datasets with gold transcripts covering diverse Indic languages and audio conditions.

This approach balances each model’s strengths, enabling a hybrid pipeline designed to minimize WER realistically while supporting multilingual, code-mixed, and multi-script scenario transcription needs.


I read your question and the model notes. Here is a balanced, honest plan that keeps every model’s strengths and aims for a realistic path to <3% WER on a curated subset.

**Transcript Bundle (what to ask Gemini to output)**
- **`orthographic_mixed`**: native script for Indic spans + Latin for English; best for real-world code‑mix and user input.
- **`orthographic_native`**: all content in the canonical script per language; strongest for pronunciation fidelity and alignment.
- **`romanized`**: consistent transliteration; great for search/debug and roman‑input robustness experiments.
- **`normalized_for_tts`**: spoken‑form expansion of numbers/abbrev; reduces ambiguity and stabilizes training WER.
- **`language_spans`**: per‑span `lang` + `script`; enables clean handling of quoted Hindi inside Telugu, etc.
- **`events`** (separate list): optional; enables controllability later without polluting core text.

**Gemini Flash: positive points and best use**
- **Throughput**: fast and cheap enough to cover your full corpus and multiple transcript views.
- **Schema stability**: reliable JSON adherence for structured fields at scale.
- **Good on easy audio**: single‑speaker, high‑SNR segments get strong results with strict prompts.
- **Ideal first pass**: generates candidate transcripts + spans for the ensemble to validate.
- **Metadata tasks**: efficient for LID hints, code‑mix detection, and simple tag suggestions.

**Gemini Pro: positive points and best use**
- **Instruction fidelity**: better at strict verbatim rules and complex multi‑field outputs.
- **Code‑mix disambiguation**: stronger at tricky language switches or mixed‑script quotes.
- **Hard audio**: more robust to noise, overlap, and long‑form context.
- **Arbitration**: best for segments where models disagree or alignment fails.
- **Higher‑value tagging**: safer for emotion/event tags when combined with detector gating.

**Hybrid strategy to chase ultra‑low WER**
- **Pre‑gate audio**: drop/flag multi‑speaker, music, or low‑SNR segments using your existing detectors.
- **Flash first pass**: get full transcript bundle with `<unk>` for uncertain spans.
- **Local ASR pass**: run Whisper‑large‑v3 + IndicConformer (and optionally MMS) for cross‑checks.
- **Forced alignment**: CTC or IndicMFA alignment score as the primary truth‑gate.
- **ROVER/consensus**: build a confusion network and vote; resolve only high‑agreement words.
- **Escalation**: if alignment/consensus fails, send to Pro; if still weak, drop or keep for non‑text pretraining.

**Validation plan for <3% WER (honest but aggressive)**
- **Gold set**: 1–2k human‑checked segments per language; this is the only honest way to prove <3%.
- **Open‑source eval**: run the same pipeline on Common Voice / FLEURS / OpenSLR Indic subsets for calibration.
- **Dual metrics**: report raw WER and normalized WER; track CER for script‑heavy languages.
- **Threshold tuning**: use alignment coverage + cross‑ASR agreement to tune accept/drop thresholds.
- **Expectation setting**: <3% WER is realistic only on a curated “gold” slice; the full corpus will be higher.

**Where to be extra careful**
- **Romanized code‑mix**: LID confusion and wrong script are common; rely on `language_spans`.
- **Overlaps + music**: WER spikes; treat these as hard negatives even if text looks plausible.
- **Named entities**: people/places/brands are failure hotspots; allow `<unk>` or tag for review.
- **Numbers/abbrev**: inconsistent normalization inflates WER; keep both verbatim and normalized views.
- **Long segments**: errors accumulate; keep shorter windows for transcription, align on the original audio.

If you want, I can help with:
1. A concrete JSON schema and prompt pair tuned separately for Flash vs Pro.
2. ROVER/consensus rules and acceptance thresholds based on alignment + agreement.
3. A minimal gold‑set sampling plan per language to validate the <3% target.