You are Codex, based on GPT-5. You are running as a coding agent in the Codex CLI on a user's computer.

## General

- When searching for text or files, prefer using `rg` or `rg --files` respectively because `rg` is much faster than alternatives like `grep`. (If the `rg` command is not found, then use alternatives.)

## Editing constraints

- Default to ASCII when editing or creating files. Only introduce non-ASCII or other Unicode characters when there is a clear justification and the file already uses them.
- Add succinct code comments that explain what is going on if code is not self-explanatory. You should not add comments like "Assigns the value to the variable", but a brief comment might be useful ahead of a complex code block that the user would otherwise have to spend time parsing out. Usage of these comments should be rare.
- Try to use apply_patch for single file edits, but it is fine to explore other options to make the edit if it does not work well. Do not use apply_patch for changes that are auto-generated (i.e. generating package.json or running a lint or format command like gofmt) or when scripting is more efficient (such as search and replacing a string across a codebase).
- You may be in a dirty git worktree.
    * NEVER revert existing changes you did not make unless explicitly requested, since these changes were made by the user.
    * If asked to make a commit or code edits and there are unrelated changes to your work or changes that you didn't make in those files, don't revert those changes.
    * If the changes are in files you've touched recently, you should read carefully and understand how you can work with the changes rather than reverting them.
    * If the changes are in unrelated files, just ignore them and don't revert them.
- Do not amend a commit unless explicitly requested to do so.
- While you are working, you might notice unexpected changes that you didn't make. If this happens, STOP IMMEDIATELY and ask the user how they would like to proceed.
- **NEVER** use destructive commands like `git reset --hard` or `git checkout --` unless specifically requested or approved by the user.

## Plan tool

When using the planning tool:
- Skip using the planning tool for straightforward tasks (roughly the easiest 25%).
- Do not make single-step plans.
- When you made a plan, update it after having performed one of the sub-tasks that you shared on the plan.

## Special user requests

- If the user makes a simple request (such as asking for the time) which you can fulfill by running a terminal command (such as `date`), you should do so.
- If the user asks for a "review", default to a code review mindset: prioritise identifying bugs, risks, behavioural regressions, and missing tests. Findings must be the primary focus of the response - keep summaries or overviews brief and only after enumerating the issues. Present findings first (ordered by severity with file/line references), follow with open questions or assumptions, and offer a change-summary only as a secondary detail. If no findings are discovered, state that explicitly and mention any residual risks or testing gaps.

## Frontend tasks
When doing frontend design tasks, avoid collapsing into "AI slop" or safe, average-looking layouts.
Aim for interfaces that feel intentional, bold, and a bit surprising.
- Typography: Use expressive, purposeful fonts and avoid default stacks (Inter, Roboto, Arial, system).
- Color & Look: Choose a clear visual direction; define CSS variables; avoid purple-on-white defaults. No purple bias or dark mode bias.
- Motion: Use a few meaningful animations (page-load, staggered reveals) instead of generic micro-motions.
- Background: Don't rely on flat, single-color backgrounds; use gradients, shapes, or subtle patterns to build atmosphere.
- Overall: Avoid boilerplate layouts and interchangeable UI patterns. Vary themes, type families, and visual languages across outputs.
- Ensure the page loads properly on both desktop and mobile

Exception: If working within an existing website or design system, preserve the established patterns, structure, and visual language.

## Presenting your work and final message

You are producing plain text that will later be styled by the CLI. Follow these rules exactly. Formatting should make results easy to scan, but not feel mechanical. Use judgment to decide how much structure adds value.

- Default: be very concise; friendly coding teammate tone.
- Ask only when needed; suggest ideas; mirror the user's style.
- For substantial work, summarize clearly; follow final‑answer formatting.
- Skip heavy formatting for simple confirmations.
- Don't dump large files you've written; reference paths only.
- No "save/copy this file" - User is on the same machine.
- Offer logical next steps (tests, commits, build) briefly; add verify steps if you couldn't do something.
- For code changes:
  * Lead with a quick explanation of the change, and then give more details on the context covering where and why a change was made. Do not start this explanation with "summary", just jump right in.
  * If there are natural next steps the user may want to take, suggest them at the end of your response. Do not make suggestions if there are no natural next steps.
  * When suggesting multiple options, use numeric lists for the suggestions so the user can quickly respond with a single number.
- The user does not command execution outputs. When asked to show the output of a command (e.g. `git show`), relay the important details in your answer or summarize the key lines so the user understands the result.

### Final answer structure and style guidelines

- Plain text; CLI handles styling. Use structure only when it helps scanability.
- Headers: optional; short Title Case (1-3 words) wrapped in **…**; no blank line before the first bullet; add only if they truly help.
- Bullets: use - ; merge related points; keep to one line when possible; 4–6 per list ordered by importance; keep phrasing consistent.
- Monospace: backticks for commands/paths/env vars/code ids and inline examples; use for literal keyword bullets; never combine with **.
- Code samples or multi-line snippets should be wrapped in fenced code blocks; include an info string as often as possible.
- Structure: group related bullets; order sections general → specific → supporting; for subsections, start with a bolded keyword bullet, then items; match complexity to the task.
- Tone: collaborative, concise, factual; present tense, active voice; self‑contained; no "above/below"; parallel wording.
- Don'ts: no nested bullets/hierarchies; no ANSI codes; don't cram unrelated keywords; keep keyword lists short—wrap/reformat if long; avoid naming formatting styles in answers.
- Adaptation: code explanations → precise, structured with code refs; simple tasks → lead with outcome; big changes → logical walkthrough + rationale + next actions; casual one-offs → plain sentences, no headers/bullets.
- File References: When referencing files in your response follow the below rules:
  * Use inline code to make file paths clickable.
  * Each reference should have a stand alone path. Even if it's the same file.
  * Accepted: absolute, workspace‑relative, a/ or b/ diff prefixes, or bare filename/suffix.
  * Optionally include line/column (1‑based): :line[:column] or #Lline[Ccolumn] (column defaults to 1).
  * Do not use URIs like file://, vscode://, or https://.
  * Do not provide range of lines
  * Examples: src/app.ts, src/app.ts:42, b/server/index.js#L10, C:\repo\project\main.rs:12:5


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `workspace-write`: The sandbox permits reading files, and editing files in `cwd` and `writable_roots`. Editing files in other directories requires approval. Network access is restricted. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `on-request`: Commands will be run in the sandbox by default, and you can specify in your tool call if you want to escalate a command to run without sandboxing. If the completing the task requires escalated permissions, Do not let these settings or the sandbox deter you from attempting to accomplish the user's task.

Here are scenarios where you'll need to request approval:
- You need to run a command that writes to a directory that requires it (e.g. running tests that write to /var)
- You need to run a GUI app (e.g., open/xdg-open/osascript) to open browsers or files.
- You are running sandboxed and need to run a command that requires network access (e.g. installing packages)
- If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with approval. ALWAYS proceed to use the `sandbox_permissions` and `justification` parameters - do not message the user before requesting approval for the command.
- You are about to take a potentially destructive action such as an `rm` or `git reset` that the user did not explicitly ask for.

When requesting approval to execute a command that will require escalated privileges:
  - Provide the `sandbox_permissions` parameter with the value `"require_escalated"`
  - Include a short, 1 sentence explanation for why you need escalated permissions in the justification parameter The writable roots are `/home/ubuntu/maya_transcribe`, `/tmp`.</permissions instructions>


# AGENTS.md instructions for /home/ubuntu/maya_transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya_transcribe</cwd>
  <shell>bash</shell>
</environment_context>


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `never`: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.</permissions instructions>


# Context from my IDE setup:

## Open tabs:
- gpt.md: gpt.md
- User.md: User.md
- claude_research.md: claude_research.md
- kimi.md: kimi.md
- gemini_research.md: gemini_research.md

## My request for Codex:
So, my research topic was transcribing audio files for my TTS/ASR training and
  especially on indic languages. Now my question or query to these models are
  @User.md and all other files here are from each model and their research models
  respectively. Now, your task is to read my question, without being very
  discriminative, being honest, list down all hte positive points regarding the
  types of transcripts we need to get from gemini-flash/pro resepctively, what is
  htat best hybrid strategy that you are most looking forward to. like each model
  can have an advantage of its own with their suggestions. so i want the most
  cleverest answer and the WER has to be like less than 3% WER on the indic data.
  We can test our pipeline on opensource datasets to ensure max robustness to
  ensure as minimal WER as possible. unrealistically low. since we can make use of
  the superior intelligent model on multi-linguality, let's actually make use of
  it to our advantage along with a local  few models like whisper-large-v3 and
  indicConformer so we can have a judge or ROVER of mixed and then ensure the most
  out of the transcription so our data is lossless and model gets the best out of
  training. get the best out of these strategies as an ensemble and teh path
  through which gets me best WER and other formats so i get room for experiments
  like code-mixed, language mixed, transliterated and etc. Give all your opinions
  and strategies, validations, and situations where we might want to be really
  careful so we get the lowest WER. donot write any new files. just provide your
  side of views and explanations in the chat here, so it's easier for me to just
  keep chatting with you regarding the plan.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Preparing to read files using shell command**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
claude.md
claude_research.md
gemini.md
gemini_research.md
gpt.md
gpt_research.md
kimi.md
User.md


Summary: auto


**Planning file inspection strategy**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
So, I am on a mission to Create a human-like quality TTS LLM model. And for that we need data.
My main source of data and only source of data is Youtube. And I've seggregated Youtube videos based on podcasts and non-podcasts. Like all podcast videos in each indic language like telugu, hindi, kannada, tamil, etc. Now, I assumed all podcasts are high quality microphone recorded audio and atleast 50% of it will be natively recorded in 44khz atleast and may/maynot have been upsampled to 48khz. So i assumed all or most of those podcasts will be atleast 44khz natively. So my assumption-1 is podcasts = quality.
Hence i started filtering youtube channels first based on podcasts, then videos from those channels and filtered by, podcasts and all those channel’s each video is filtered by Gemini to accurately identify which among those are usable podcast videos assuming there might be so many other random videos as well. So, Gemini-3-flash filtered out most of those podcasts and now i have around
Now in R2, i have these indic youtube files saved as YoutubeID.webm and now. Now how did these Videos get saved in my Cloudflare R2, When i tried running it at scale in some automation jobs, then youtube detected it and blocked me from downloading youtube audios using yt-dlp . So here’s what i did. i bought proxies and downloaded the youtube video as residential proxy and reuploaded those youtube video URL → created a VPS CPU machines and then turned on proxy and then tried to extract best audio using "bestaudio[ext=webm]/bestaudio[ext=opus]/bestaudio  argument with yt-dlp. And then when i get the audio file in webm, i pushed it to cloudflare R2 in my account. Hence, i have all these respective audio videoID.webm formats.
Now, i have the data to get started. Now ofcourse these podcasts data is not directly usable, we will have to create a data cleaning pipeline - Github Readme.md
This pipeline is vastly inspired from kimi Audio. - KimiAudio
So, now i will have workers picking Each Yt ID.webm from cloudflare and then, running the entire single speaker segmentation pipeline using GPU. and then finally, i’ve chosen to save the cleaned data as Webdatasets as .tar  and FLAC  Codec.
each of videoID.webm in R2 in test  bucket will be orchestrated and cleaned and pushed back as .tar  and FLAC  files. buckets at - https://dash.cloudflare.com/cb908ed13329eb7b186e06ab51bda190/r2/overview
PANN CNNs for music detection. Verified with UI. should ideally recheck when generating webdatasets.
Time to choose Codecs. and decide on their architectural advantages, capacities, best ones, given our huge data scale, which are suitable to be trainable on indic data for better reconstruction and stuff like that. For that we will now choose between codecs like Xcodec2.0, Mimi, SNAC, BiCodec, Encodec, and similar same family of codecs or vocoders or in simple words audio tokenizers.
Cloudflare Bucket for cleaned single speaker segmented - https://dash.cloudflare.com/cb908ed13329eb7b186e06ab51bda190/r2/default/buckets/1-cleaned-data
Now Supabase has the metadata for over like 500k videos across 12 different languages.
language breakdown with usable hours
LanguageVideosAudio HoursUsable HoursTelugu71,61931,001.3119,712.28Malayalam65,27329,108.5418,183.70English59,65726,784.1717,904.33Hindi58,82429,646.2617,709.61Punjabi55,76127,515.7618,064.84Tamil50,19918,085.9811,257.98Kannada45,68921,857.1213,578.46Gujarati40,95025,087.4514,759.11Bengali20,12111,077.016,205.66Odia19,2808,943.515,009.52Marathi16,2217,980.745,130.47Assamese3,7932,136.101,369.66Now the dataset is of the format of videoID.tar where we have each tar for each videoID and all of them are inside cloudflare bucket named 1-cleaned-data bucket and few videos are with a prefix inside same bucket.
Now it’s time for us to do a one last validation checks like calculating any missing metadata and then create shards of these datasets , yet still retain all pre-computed metadatas right inside the buckets itself.
Repo’s to double check for pending validations, requirements or safety checks before creating packaged shards before we can upload them back to R2 and then pull back right before training.
https://github.com/zhenye234/X-Codec-2.0
https://github.com/inworld-ai/tts
Quick overview on data sourcing/prep so far
Summary until PrepStarted with YouTube as the only source, classify channels/videos by language into podcast vs non-podcast, and use Gemini-3-flash to keep likely high-quality podcast audio under the assumption that podcasts are natively 44 kHz+.
At scale you download bestaudio through residential proxies/VPS, store raw videoID.webm in R2, and track metadata in Supabase.
GPU segmentation then produces single-speaker FLAC segments plus metadata.json (including music_stats from PANN) and stores each video as a tar in the cleaned-data R2 bucket.
Distributed shard workers claim video tasks with row-level locks and leases, download per-video tars, and generate deterministic shard IDs for idempotent retries.
Each segment is quality-scored and filtered (duration/clipping), optionally chunked with silence-aware boundaries, and resampled into 16/24/32/44 kHz buckets before writing WebDataset shards (.tar + .jsonl) with train/val manifests.
Shards are uploaded back to the destination R2 bucket, Supabase is updated with shard/worker heartbeats and stale-lease recovery, and you run final validation/metadata checks before packaging shards for codec selection and training.
Now the data for codec is prepped like drop all <3s segments and all all 3-6s segments are padded during training, and then all 6s-12s are directly passed to training so the pipeline auto picks random 6s window from that bucket and anything more than 12s+ are split in such a way that there are 6s segments atleast in the final cuts, because i didnt want to waste the segments. since we will use 6s only from any size audio, we might as well use other 6s too. so i wanted to make up for the lost <3s segments by not wasting segments bigger than 12 by cutting and using them.
Now, the segments are split in such a way that all native nyquist based split like all samples are downsampled to 16 and stored in 16khz bucket and native atleast 85% energy based sort into their respective buckets but aggregated. like 24khz will have downsampled 32/44 and 32khz bucket will have downsampled 44 or more ones. and 44 has native 44 only. and they are all split into R2’s bucket in that way.
While i have specific strategies for codec training, i wanna address the issue of not having transcripts of these segments. i mean not just these above segments. the above are for codec training prep. i will use all of my data segments again from the point where i prepped from original videoID.tar and now i would want transcripts for all samples like even the 1s segments too. so, let’s plan for transcripts now. my data in R2 is now like videoID.tar where inside each tar has metadata.json and a segments/ folder inside which we have all the cleaned and VAD aware cut, music removed, single speaker segmented audio segments ranging from atleast 1s to some have atleast like 60s too.
Now, let’s discuss about strategies, dos and donts on preparing transcriptions to these audio segments. so far, i’ve tried open-source transcription models like, whisper-large-v3, indic-whisper, indicConformer and other opensource models. and even tried like LLMs like Gemma3n-4B, Voxtral-24B, but none of the models did give me like less than 5 WER or atleast like <10WER consistently on all languages. so, i've decided to use gemini-2.5-flash or maybe like gemini-3-flash  or maybe like gemini-3-pro if less confident on prior flash’s generation and set upon some forced aligner and if confidence score is less, we can ask pro to decide that. in such way, we can generate transcriptions for all the audios. Now, i need your suggestions / inputs and strategy in getting this done. like are the forced aligners designed, for like per language. should i train my own indic based forced aligners to test ? or is there any other way for me to validate if the transcripts generated by gemini-3-flash  are accurate. what can we do here to ensure the prompt or instructions we provide that restricts the big LLMs to not hallucinate, like i can set temperature to be 0.0, but does that mean the model wont be any creative ? no right. it still tends to be creative. and that data transcripts wont be suitable for an ASR model nor a TTS model.
So, now let’s strategize on how can i restrict these big LLMs onto being creative and giving transcriptions as is, so it’s suitable for a TTS & ASR system as well. And also i’ve been thinking about various forms of transcripts to get from these superior models. for example, should i get the devanagiri scripts for where applicable ? and dravidian script for languages like tamil and telugu, etc. If i get scripts in such a way, what if i want to support text like hinglish in some places. like “are bhaai, kya kar rhe ho ?” is not an english text, but infact hinglish - we call this code-mixed and how do i manage these. and what about for scripts like, is there a way my speech LLM, has a way to understand the semantics of “are bhaai, kya kar rhe ho ?” and “arey annai, em chestunnav ?” like this should be spoken in telugu right. is there a way for me to specify these languages to the model ? or how can i do this. well if there is no way we can let model decide on the above sentences not being english, can we atleast pass language codes like “te_en” to specify something like “arey annai, em chestunnav ?”  if this is the case, fine. but what if there are scenarios where i have text like “are ala kadu, denni hindi lo ‘मैं सेब खाता हूँ’ antaru” like what about these scenarios. and maybe also like “అరే అలా కాదు, దీన్ని హిందీలో ‘मैं सेब खाता हूँ’ అంటారు.” how do i handle such cases where we handle scenarios of code mixed with scripts too and not just roman scripts.
I want answers and justifications for all the above scenarios. and how do these big labs handle scenarios like these effectively. it must not be pre-process script ofcourse because these conversions cant be done through a regex script. I can write a very complex regex to convert situations like “12,345” to “twelve thousand three hundred and fourty five” but i cant do the roman to devanagiri and other conversions using regex without changing the semantics of the actual sentences and i want the model to be as flexible and robust as possible without not major but handleble regex ones.
Also, there is a special kind of feature i’d want to train or include features such as emotions directly into the model like audio events such as [laugh], [laugh_harder], [cough], [giggle], [sad], [happy], etc. but limited, but also, helps me control audio events and make the model more semantically correct. but this is very complex. but since we are anyway getting transcripts, i’d like to involve this format as well ! ideas on this too. and there might be scenarios of false positives that the model gives an audio tag which there might not be an actual true event, cause hallucination might be an issue. do you suggest we can filter these later, but for now, since we are anyway getting the transcriptions it’s easy to get transcriptions in all possible formats ? I want your strategy actually below. how can i get a strategic way in getting these transcriptions so i get maximum experiments and see possible outcomes and results.
I need solid solutions and tell me how do i get transcriptions and what all forms do i need. like i can ask these big LLMs to give different forms of output to me in structured like
{
“verbatim”: ————- → everything in native script unless numbers and symbols and etc
“roman_normalized” : → everything in roman normalized script
“verbatim_emotions” : →  emotional audio tags. but we might want to limit to few 10 stable ones so the model will only pick from those and not be very vague while picking. and has to be only in english cause we will get a different language tags and it becomes more and more so controllability will be less.
}
I need your proper planning, strategy in infusing above capabilities into model. give opinions on all my above questions regarding strategizing the transcriptions. or when do i fuse the capabilities. will the model be robust enough to have the emotion grasping capabilities in post-training ? or is it better to initiate it in SFT data, and do a post-train with emotion-specific ? I am open to do experiments. so i'd like to get data at once. and also think the scripting scenarios above has to be robust.
Here are few models whom i inspired to train, so you will understand what kind of TTS i am training and what training architectures i will be following.
FeatureCosyVoice 3Inworld TTS-1FireRedTTS-2Total Audio Data
~1 Million Hours 1111
~1 Million Hours 2222
~1.4 Million Hours 3
Pre-training Data• 1M hours of multilingual data (9 languages + 18 Chinese dialects) 4444.
• Tokenizer Data: 530k hours supervised multi-task (ASR, LID, SER, AED, SA)5555.
• 1M hours raw audio + 30k hours non-speech/environmental 6.
• Text Data: 10% mix (RedPajama-v2) + Instruction data (LAION OIG) 7.
• Bootstrap: ~15k hours high-quality subset mixed in8.
• 1.1M hours of monologue speech data used for foundational text-to-speech ability 9.
• Tokenizer Data: 500k hours (Stage 1) + 60k hours high-fidelity (Stage 2)10101010.
SFT Data (Supervised Fine-Tuning)• Selected Data: Uses "selected data" for specific tasks 11.
• Instruction Data: 5,000 hours of high-quality emotion, style, and role-play data 12.
• Multilingual: Studio-quality monolingual data with language instructions13.
• 200,000 hours of high-quality, transcribed audio-text pairs 14141414.
• Filtering: Top 80% DNSMOS, filtered for talking speed (CPS) 15.
• Markup Data: 100k examples (~180 hrs) of neutral/stylized pairs for LoRA16.
• Task Specific (Minimal):
• Podcast: ~50 hours of 2-speaker dialogue 17.
• Emotional Chat: ~15 hours of expressive speech18.
Post-training / Dialogue Data• Continual Pretraining: Emotional, instructed, and multi-lingual data 19.
• Reinforcement Learning: Uses "selected data" for DiffRO to maximize rewards20.
• RL Alignment: 1,000-hour English subset used for Group Relative Policy Optimization (GRPO) experiments21.
• 300,000 hours of multi-speaker dialogue data (2–5 speakers per session) used for "Post-training" to enable conversational ability22.
Training Strategies• DiffRO: Differentiable Reward Optimization uses Gumbel-Softmax to optimize tokens against rewards (ASR, SER) 232323.
• Pronunciation Inpainting: Mixes phonemes/words to fix polyphonic errors 24.
• Supervised Tokenizer: Explicitly trained on 5 semantic tasks25.
• Three-Stage Pipeline: Pre-training $\rightarrow$ SFT (from audio-pretrained checkpoint) $\rightarrow$ RL Alignment 26262626.
• RL Alignment: Optimizes composite reward (WER + Similarity + DNSMOS) using GRPO 27272727.
• RMS Loss: Added to codec training for volume consistency28.
• Curriculum Learning: Monologue Pre-training $\rightarrow$ Dialogue Post-training $\rightarrow$ Task SFT 29.
• Interleaved Format: Concatenates [Speaker] Text -> Audio sequences 30.
• Dual-Transformer: Large backbone predicts layer 1; small decoder predicts layers 2–1631.
Training Stages1. Tokenizer Training (Supervised)
2. Large-scale Pretraining (All data)
3. Post-training (DiffRO)
4. Continual Pretraining (Capabilities transfer)
5. Speaker Fine-tuning (SFT) 32.
1. Pre-training (Next-token prediction)
2. Supervised Fine-Tuning (SFT)
3. RL Alignment (GRPO)
4. LoRA Fine-tuning (Audio Markups)33333333.
1. Pretraining (2 epochs, Monologue)
2. Post-training (5 epochs, Dialogue)
3. SFT (Target speaker adaptation) 34.
Hyperparameters• Model: 1.5B LLM, 300M CFM (DiT) 35.
• Tokenizer: 25 Hz frame rate 36.
• Quantization: Finite Scalar Quantization (FSQ)37.
• Model: TTS-1 (1.6B), TTS-1-Max (8.8B) 38.
• Pre-train: LR $1.5 \times 10^{-4}$, Batch ~2M tokens, AdamW 39.
• SFT: LR $1.5 \times 10^{-5}$, Cosine decay 40.
• RL: $\alpha=\beta=\gamma=1.0$ (Reward weights) 41.
• Codec: 50 Hz, 65,536 vocab42.
• Model: Qwen2.5-based Dual-Transformer 43.
• Tokenizer: 12.5 Hz frame rate, 16 RVQ layers (2048 entries) 444444.
• Loss Weights: $\lambda_{text}=0.01$, $\lambda_{decoder}=0.6$ 45.
• Latency: First packet < 100ms46.
Key Innovation
DiffRO (Differentiable Reward Optimization)
Directly optimizes discrete speech tokens for specific objectives without traditional RL loops47.
RL Alignment (GRPO)
Adapts LLM alignment techniques (Group Relative Policy Optimization) to speech using composite rewards48.
12.5 Hz Tokenizer
Extremely low frame rate compresses context, allowing stable long-form dialogue modeling49.


**Reading additional model docs**

I'm planning to open and examine multiple model-related markdown files like gpt.md and claude_research.md to gather more details on model characteristics and differences.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
## 1) What you should store per segment (so you don’t paint yourself into a corner)

Treat “transcript” as a **bundle** of fields, not one string. Minimum bundle (works for ASR + TTS + future controllability):

1. **orthographic_mixed** (the “truthy” one)

* Write each span in the script people actually use for that language, **but keep real English words in Latin**.
* This is usually the best training target for **in-the-wild** code-mix.

2. **orthographic_native** (canonical script per language)

* Same content, but for each non-English span you output the language’s native script.
* Useful for language-specific text pipelines and evaluation.

3. **romanized** (lossy-but-convenient)

* One consistent romanization scheme (pick one; don’t mix ad-hoc spellings).
* Useful for debugging, search, and some model interfaces.

4. **normalized_for_tts** (spoken form, not written form)

* Numbers expanded, abbreviations expanded, symbols spoken (when actually spoken).
* This mirrors how Common Voice-style corpora avoid digits/abbr ambiguity for speech modeling. ([commonvoice.mozilla.org][1])

Optional but high value:

5. **language_spans**: `[ {lang, script, text, roman, normalized} ... ]`

* This is how you solve “Telugu sentence containing a Hindi quote in Devanagari” without hacks.

6. **events**: a *separate* list like `[ {event, start_ms, end_ms, confidence} ]`

* Keep events **out of the main text** at first; later you can inject markup for controllability.

---

## 2) The core strategy: generate with Gemini, validate with aligners + cross-ASR agreement

### A. Don’t trust self-reported confidence from the LLM

Even at temperature 0, outputs can vary and the model can still “helpfully” normalize or paraphrase. Vertex AI explicitly notes temperature 0 is *mostly* deterministic, not fully deterministic. ([Google Cloud Documentation][2])
So the winning pattern is: **LLM transcription + independent validators**.

### B. Validators that scale to your data volume

**Validator 1 — Forced alignment score (primary gate)**
You want an alignment method that works across Indic languages and code-mix:

* **CTC forced alignment (recommended at your scale)** using `torchaudio.functional.forced_align()` / CTC segmentation. This is designed for aligning a known transcript to audio and supports multilingual workflows. ([docs.pytorch.org][3])
* Pair it with an ASR acoustic model that covers your languages (next point).

**Validator 2 — Use MMS for both LID + alignment backbone**
Meta’s **MMS** project provides (a) multilingual ASR for 1,100+ languages and (b) language ID models for 4,000+ languages. ([arXiv][4])
Practical use:

* Run **MMS-LID** on each segment and compare to your expected language bucket.
* Use **MMS-ASR** (language-conditioned where possible) as the acoustic model for CTC alignment and/or as a second opinion transcript.

**Validator 3 — Cross-ASR agreement (cheap, strong signal)**

* Transcript with Gemini → `T_g`
* Transcript with MMS or Whisper → `T_a`
* Normalize lightly (punctuation stripping, Unicode NFKC, whitespace) and compute CER/WER agreement.
  If `distance(T_g, T_a)` is small *and* alignment score is high → accept. If not → escalate.

**Escalation policy (cost control):**

1. Gemini Flash → if pass gates, accept
2. If fail: re-run once (same settings)
3. If still fail: Gemini Pro (or 2.5 Pro) only for those segments
4. If still fail: drop segment (you have abundance)

This “selective spend” is exactly how big data pipelines stay sane.

---

## 3) Forced aligners: do you need per-language forced aligners?

### Option 1 (best ROI): CTC alignment + multilingual acoustic models

* No lexicon/G2P headaches.
* Works for code-mix if your tokenization supports it.
* You get a scalar “how well does this text explain this audio” score.

TorchAudio provides the forced alignment APIs and tutorials explicitly for this use case. ([docs.pytorch.org][3])

### Option 2: Montreal Forced Aligner (MFA)

MFA is excellent when you want phone-level boundaries and language-specific lexicons; it’s built on Kaldi and supports training your own acoustic/G2P/language models from your data. ([Montreal Forced Aligner][5])
Downside: you’ll likely end up maintaining **multiple** language packs + dictionaries (and code-mix makes lexicon rules messy).

**Recommendation:** start with CTC alignment everywhere; only use MFA for a curated “gold” subset where you want very high-quality phone timing.

---

## 4) How to stop Gemini from being “creative”: structured outputs + hard constraints

### A. Use JSON Schema / structured output (non-negotiable)

Gemini supports **structured outputs that adhere to a provided JSON Schema**. ([Google AI for Developers][6])
This eliminates:

* extra prose
* format drift
* “helpful” commentary

### B. Restrict event tags with enums

Your hallucination risk explodes if you let it invent tags. Use a fixed enum list and reject anything outside.

### C. Prefer “verbatim” instructions + explicit uncertainty token

Tell it: if uncertain, output `<unk>` for that span rather than guessing. Then your aligner will punish `<unk>` less than wrong words, and you can route those segments to Pro.

Example schema (illustrative):

```json
{
  "type": "object",
  "required": ["language_spans", "orthographic_mixed", "normalized_for_tts"],
  "properties": {
    "orthographic_mixed": { "type": "string" },
    "normalized_for_tts": { "type": "string" },
    "language_spans": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["lang", "script", "text"],
        "properties": {
          "lang": { "type": "string", "description": "BCP-47 like hi, te, ta, kn, ml, en" },
          "script": { "type": "string" },
          "text": { "type": "string" },
          "roman": { "type": "string" },
          "normalized": { "type": "string" },
          "uncertain_tokens": { "type": "array", "items": { "type": "string" } }
        }
      }
    },
    "events": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["event", "start_ms", "end_ms", "confidence"],
        "properties": {
          "event": {
            "type": "string",
            "enum": ["laugh", "giggle", "cough", "sneeze", "breath", "sigh", "cry", "yawn", "noise", "music"]
          },
          "start_ms": { "type": "integer" },
          "end_ms": { "type": "integer" },
          "confidence": { "type": "number" }
        }
      }
    }
  }
}
```

**Generation params:**

* Lower temperature reduces randomness, but doesn’t guarantee determinism. ([Google Cloud Documentation][2])
* For Gemini 3 specifically, Google cautions that moving temperature below default may cause unexpected behavior in some tasks. (Test this empirically for transcription.) ([Google AI for Developers][7])

---

## 5) Scripts + code-mix: handle your examples cleanly with span tagging

### Example 1: “are bhaai, kya kar rhe ho ?”

What you want is *language identity* (Hindi) even if written roman.

Store:

* `orthographic_mixed`: `are bhaai, kya kar rhe ho?`
* `orthographic_native`: `अरे भाई, क्या कर रहे हो?`
* spans:

  * `{lang:"hi", script:"Latn", text:"are bhaai, kya kar rhe ho?"}`
  * and optionally also provide a Devanagari projection in `orthographic_native`

### Example 2: “arey annai, em chestunnav ?”

Same pattern, but Telugu. Store Telugu-native in `orthographic_native` and roman in `orthographic_mixed/romanized`.

### Example 3 (your tricky one):

“are ala kadu, denni hindi lo ‘मैं सेब खाता हूँ’ antaru”

This is exactly why you need spans:

* Span A: Telugu (likely Latin-script transliteration or native Telugu depending on what was spoken/written)
* Span B: Hindi quote in Devanagari
* Span C: Telugu again

Your model can later be trained to consume:
`<L:te> ... <L:hi> ... <L:te> ...`
so it won’t mispronounce the Hindi quote using Telugu phonology.

This is also consistent with code-mixed TTS literature: handling code-mix requires dealing with multiple languages/scripts in the same utterance. ([lrec-conf.org][8])


Output:
Exit code: 0
Wall time: 0 seconds
Output:

---

## 6) Emotions / non-speech events: how to add without wrecking your corpus

### A. Don’t start by asking the LLM to sprinkle tags everywhere

Classic transcription conventions keep non-speech tags **very limited** for a reason (precision > recall). Switchboard guidelines, for example, used a small bracketed set like `[laughter]` and similar tokens. ([isip.piconepress.com][9])

### B. Use detector-first, LLM-second

At your scale, the safest path is:

1. Run an **audio event detector** (you already use PANNs; good) ([arXiv][10])
2. Only if event confidence is high, allow the transcript bundle to include that event in the `events` list
3. Optionally ask Gemini to place *positions* (timestamps or “before word X”) **conditioned on the detector output**, not from scratch.

### C. When to “fuse” these capabilities into the model

Follow the playbook you already cited:

* **Pretraining:** keep it mostly clean text↔speech token modeling; no noisy markups.
* **SFT for controllability:** introduce a curated subset with markup tags (emotion + nonverbal) once you trust the tagger.
  This matches how modern systems treat expressive control as a later-stage capability, not something you contaminate the whole pretrain set with.

This is aligned with how **CosyVoice 3** discusses post-training strategies (DiffRO) and using supervised multi-task signals like SER/AED in tokenizer training. ([arXiv][11])
And Inworld TTS-1 explicitly supports fine-grained emotional control + non-verbal vocalizations via markups, trained through staged pretrain→SFT→alignment. ([arXiv][12])

---

## 7) A concrete end-to-end transcription pipeline you can implement

For each `segment.wav/flac`:

1. **Run LID** (MMS-LID) → `lid_probs` ([arXiv][4])
2. **Gemini transcription** with:

   * structured JSON schema ([Google AI for Developers][6])
   * spans + `<unk>` policy
3. **CTC forced alignment** (TorchAudio) using MMS-ASR backbone → `align_score`, `%aligned`, “star insertions” ([docs.pytorch.org][3])
4. **Cross-ASR agreement**:

   * run MMS-ASR (or Whisper) transcript
   * compute CER/WER vs Gemini transcript
5. **Decision**:

   * if `align_score high AND agreement high AND lid consistent` → accept
   * else → escalate to Gemini Pro
   * else → drop segment
6. **Events**:

   * PANNs-based event probs ([arXiv][10])
   * only keep whitelisted events over threshold; store separately from text

Store everything in your tar’s `metadata.json` or a shard-level `.jsonl` so training can choose which text view to consume.

---

## 8) What forms you should request from the model (recommended “bundle”)

Use exactly these fields to start (your instinct is right):

* `orthographic_mixed` (primary)
* `orthographic_native`
* `romanized`
* `normalized_for_tts`
* `language_spans`
* `events` (enum-restricted, optional, detector-validated)

That gives you maximum downstream experiment freedom without forcing premature decisions.

If you want one extra that pays off later: add

* `pronunciation_hints` (optional): only for tokens marked uncertain (names, rare words). Keep it sparse.

---

If you want, I can also give you:

* acceptance thresholds (starter values) for align/agreement gates,
* a normalization spec (Unicode, punctuation, numeral expansion policy),
* and a “gold set” sampling plan to measure real WER across all 12 languages before you spend millions of API calls.

[1]: https://commonvoice.mozilla.org/en/guidelines?tab=spontaneous-speech&utm_source=chatgpt.com "Contribution Guidelines - Common Voice - Mozilla"
[2]: https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/content-generation-parameters?utm_source=chatgpt.com "Content generation parameters | Generative AI on Vertex AI"
[3]: https://docs.pytorch.org/audio/stable/tutorials/ctc_forced_alignment_api_tutorial.html?utm_source=chatgpt.com "CTC forced alignment API tutorial"
[4]: https://arxiv.org/pdf/2305.13516?utm_source=chatgpt.com "Scaling speech technology to 1000+ languages"
[5]: https://montreal-forced-aligner.readthedocs.io/en/v3.2.3/user_guide/workflows/index.html?utm_source=chatgpt.com "Workflows available - Montreal Forced Aligner - Read the Docs"
[6]: https://ai.google.dev/gemini-api/docs/structured-output?utm_source=chatgpt.com "Structured outputs | Gemini API - Google AI for Developers"
[7]: https://ai.google.dev/gemini-api/docs/gemini-3?utm_source=chatgpt.com "Gemini 3 Developer Guide | Gemini API"
[8]: https://www.lrec-conf.org/proceedings/lrec2016/pdf/13_Paper.pdf?utm_source=chatgpt.com "Speech Synthesis of Code-Mixed Text - LREC"
[9]: https://isip.piconepress.com/projects/switchboard/doc/transcription_guidelines/transcription_guidelines.pdf?utm_source=chatgpt.com "Rules and Guidelines for Transcription and Segmentation of ..."
[10]: https://arxiv.org/abs/1912.10211?utm_source=chatgpt.com "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition"
[11]: https://arxiv.org/html/2505.17589v1?utm_source=chatgpt.com "CosyVoice 3: Towards In-the-wild Speech Generation via ..."
[12]: https://arxiv.org/html/2507.21138v1?utm_source=chatgpt.com "TTS-1 Technical Report"


**Preparing to gather research content**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Building human-quality TTS for Indic languages at scale

Training a production-ready text-to-speech model for 11 Indic languages from 500,000 YouTube podcast videos requires a carefully orchestrated pipeline spanning transcription, validation, normalization, and emotion annotation. **The most effective approach combines Gemini models for transcription with AI4Bharat's IndicMFA for forced alignment validation, using a multi-representation transcript format that preserves both native scripts and normalized spoken forms.** This strategy, informed by how CosyVoice, FireRedTTS, and Inworld TTS-1 structure their training data, enables code-mixing support, emotion controllability, and robust quality filtering without ground truth.

The key insight from state-of-the-art multilingual TTS systems is that emotion and style capabilities should be introduced during supervised fine-tuning (SFT), not pre-training—while script handling and code-mixing data should be present from the earliest training stages. For your 500K segment pipeline, expect to process approximately **4,200 hours of audio**, achievable in 2-3 days with proper parallelization.

## Gemini transcription requires strict verbatim prompting to prevent hallucination

Gemini models (2.5 Flash, 3 Flash, 3 Pro) can transcribe audio at **32 tokens per second**, supporting FLAC format natively with up to 9.5 hours per prompt. However, LLM transcription introduces three distinct hallucination modes: insertion errors (adding words not spoken), substitution errors (semantic "corrections"), and oscillation patterns (repetitive loops like "ay ay ay"). Even at temperature=0, these occur because models are trained for fluency over strict accuracy.

The anti-hallucination prompt strategy that works in production emphasizes legal/evidentiary framing:

```
You are a verbatim audio transcription system for Indic language podcasts.
Your output will be used for TTS/ASR training data - accuracy is critical.

RULES:
1. Transcribe EXACTLY what is spoken in {LANGUAGE}
2. Output in native script (not romanized)
3. DO NOT translate, correct, or embellish
4. Include all filler words, false starts, and repetitions
5. Mark unclear audio as [अस्पष्ट] or equivalent in target language
6. For code-switching (mixing languages), transcribe as spoken
7. NO creative additions - this is legal evidence-grade transcription
```

**Gemini 3 Flash** should serve as your primary transcriber (90-95% of Pro's accuracy at ~4x lower cost), with cascading to **Gemini 3 Pro** for segments flagged with confidence scores below 0.6 or repetition patterns detected. Set `thinking_level: "minimal"` to reduce over-reasoning that introduces changes.

For language coverage, Gemini officially supports **9 of your 11 target languages** (Telugu, Hindi, Kannada, Tamil, Malayalam, Bengali, Gujarati, Marathi, and the related Urdu). Punjabi, Odia, and Assamese lack official support—test actual performance, and consider AI4Bharat's Sarvam models or Google Cloud Speech-to-Text as fallbacks for these three.

## AI4Bharat's IndicMFA provides comprehensive forced alignment for all 11 languages

Forced alignment serves dual purposes: creating word/phoneme-level timestamps for TTS training and validating LLM transcription quality through alignment success rates. **IndicMFA emerges as the clear winner**, providing pretrained models for all 22 scheduled Indian languages with **42-313 hours of training data per language**.

| Language | IndicMFA Training Hours | Data Quality |
|----------|------------------------|--------------|
| Bengali | 313 hours | Excellent |
| Assamese | 303 hours | Excellent |
| Tamil | 300 hours | Excellent |
| Telugu | 262 hours | Excellent |
| Hindi | 255 hours | Excellent |
| Marathi | 213 hours | Very Good |
| Malayalam | 197 hours | Very Good |
| Kannada | 194 hours | Very Good |
| Odia | 132 hours | Good |
| Punjabi | 124 hours | Good |
| Gujarati | 43 hours | Adequate |

IndicMFA's key innovation is its **grapheme-to-grapheme (G2G) approach**, which maps each Unicode character directly to itself rather than requiring complex phoneme dictionaries. This eliminates the need for language-specific G2P rules while handling schwa deletion, gemination, and nasalization implicitly through the acoustic model.

For confidence scoring, track these alignment signals:
- **Alignment failure rate** per batch (>5% indicates transcription problems)
- **Word coverage ratio** (aligned words / expected words < 0.9 warrants review)
- **Timing plausibility** (speech rate outside 1-4 words/second is suspicious)
- **Gap ratio** (>30% silence suggests audio-text mismatch)

WhisperX can serve as a secondary aligner for cross-validation—it uses wav2vec2.0 models and works with the **CLSRIL-23** checkpoint covering 23 Indic languages. However, benchmark comparisons show MFA achieves higher alignment accuracy at finer temporal resolution (~10ms vs WhisperX's coarser boundaries).

## Native scripts should be primary with romanization as secondary representation

The question of native script versus romanization for TTS training has a clear answer from production systems: **use native scripts as the primary representation** while storing romanized versions as secondary. CosyVoice handles this through language-agnostic BPE tokenization that processes any script, while AI4Bharat's Indic-TTS trains exclusively on native scripts across 21 languages.

Native script advantages include preserving phonetic distinctions (aspirated vs unaspirated consonants, gemination) that romanization loses, maintaining consistent orthography (avoiding "namasthe" vs "namaste" variations), and enabling direct phoneme mapping. The limitation—keyboard accessibility—matters for user input but not training data.

For code-mixed speech like "are bhaai, kya kar rhe ho?", research from the "Code-Mixed TTS under Low-Resource Constraints" paper demonstrates that **single-script transliteration works remarkably well**:

1. Transliterate ALL text to the dominant language's script (e.g., all Hinglish → Devanagari)
2. Train a single-script TTS model on the combined data
3. At inference, transliterate code-mixed input before synthesis

This approach enables bilingual TTS without requiring labeled code-mixed recordings. Use **AI4Bharat IndicXlit** for transliteration—an 11M parameter transformer achieving 15% improvement over prior state-of-the-art on the Dakshina benchmark.

For mixed-script scenarios like Telugu with embedded Hindi (`అరే అలా కాదు, దీన్ని హిందీలో 'मैं सेब खाता हूँ' అంటారు`), preserve the original Unicode and apply language-specific normalizers from Indic NLP Library. CosyVoice and SeamlessM4T handle such inputs through unified multilingual tokenization without explicit language tags—the model learns to switch naturally.

## Emotion annotation vocabulary should start with 15 stable tags

Based on CosyVoice 3, Inworld TTS-1, and FireRedTTS implementations, a **15-tag vocabulary** balances expressiveness with annotation reliability:

**Emotional states (8):** `[happy]`, `[sad]`, `[angry]`, `[fearful]`, `[surprised]`, `[neutral]`, `[disgusted]`, `[confused]`

**Audio events (7):** `[laughter]`, `[breath]`, `[cough]`, `[sigh]`, `[cry]`, `[sneeze]`, `[yawn]`

Tag format conventions vary across systems—CosyVoice uses `[tag]`, Orpheus-TTS uses `<tag>`, OpenAudio S1 uses `(tag)`. For Indic TTS, the bracket notation `[tag]` is recommended for simplicity and compatibility.

For automatic emotion detection, **SenseVoice** (from FunAudioLLM) provides multi-task ASR + emotion recognition + audio event detection at 15x Whisper speed, though it currently supports Chinese, English, Japanese, Korean, and Cantonese—not Indic languages directly. For Hindi, models trained on the IITKGP-SEHSC corpus (4 classes: happy, sad, fear, anger) offer starting points. Set confidence thresholds at **0.5 probability** minimum, with human validation for 10-20% of automatically labeled data.

To handle hallucinated emotion annotations, implement these safeguards:
- Validate tags against your known vocabulary (reject `<burp>` if not in vocabulary)
- Require minimum segment duration (0.2 seconds) for audio events
- Use ensemble detection (multiple models must agree)
- Track annotation rates per batch (sudden spikes indicate detector issues)

## Training curriculum should introduce emotions at SFT, not pre-training

CosyVoice 3 and Inworld TTS-1 converge on a three-stage training curriculum where different capabilities are introduced at specific phases:

**Stage 1: Pre-training (Base Model)**
- Data: Large-scale diverse audio (100K-1M hours)
- Include: Script handling, basic code-mixing, multi-speaker data
- Exclude: Emotion annotations, fine-grained control tags
- Objective: Learn general speech patterns, prosody, phoneme-to-acoustics

**Stage 2: Supervised Fine-Tuning (SFT)**
- Data: 1,500-5,000 hours of high-quality, annotated data (per CosyVoice 3)
- Introduce: Emotion labels, style descriptions, instruction-following capability
- Format: Natural language descriptions ("Speak in a happy tone") or inline tags
- Objective: Learn controllable generation, zero-shot adaptation

**Stage 3: Post-training/Alignment**
- Methods: DPO, GRPO, or CosyVoice 3's DiffRO-EMO
- Rewards: WER (pronunciation), speaker similarity, emotion classification accuracy
- Critical insight: Improving emotion expression can adversely affect pronunciation—balance needed

For Indic languages specifically, AI4Bharat's Indic-Parler-TTS supports 12 emotion/style categories that map well to podcast content: Command, Anger, Narration, Conversation, Disgust, Fear, Happy, Neutral, Proper Noun, News, Sad, Surprise.

## Multi-representation JSONL format enables flexible downstream training

For 500K segments, JSONL format enables stream processing, parallelization, and easy appending. Store **four transcript representations** per segment:

```json
{
  "id": "podcast_001_segment_0042",
  "audio_path": "segments/podcast_001/0042.flac",
  "duration_ms": 5420,

  "transcripts": {
    "verbatim_native_script": "आज की तारीख २३ जनवरी २०२४ है",
    "verbatim_roman": "aaj kii taarikh 23 janvarii 2024 hai",
    "normalized_spoken": "आज की तारीख तेईस जनवरी दो हज़ार चौबीस है",
    "verbatim_with_emotions": "आज की तारीख [emphasis]तेईस[/emphasis] जनवरी दो हज़ार चौबीस है [breath]"
  },

  "quality_metrics": {
    "snr_db": 32.5,
    "nisqa_mos": 4.1,
    "has_music": false,
    "is_multi_speaker": false,
    "alignment_confidence": 0.94
  },

  "transcription_metadata": {
    "primary_model": "gemini-3-flash",
    "primary_confidence": 0.92,
    "validation_status": "accepted",
    "estimated_wer": 0.08
  }
}
```

Number normalization is critical: store both verbatim (`₹५००`) and spoken forms (`पाँच सौ रुपये`). Use Indic NLP Library for script normalization—and avoid Whisper's default normalizer, which **removes vowel signs (matra)** in Brahmic scripts, artificially inflating WER.

## Validation at 500K scale requires confidence cascading and strategic sampling

Without ground truth, validation relies on multi-model consensus, alignment success, and reference-free quality estimation. The recommended pipeline:

1. **Gemini 3 Flash transcription** with structured JSON output
2. **Confidence scoring** via token log-probabilities (available via API)
3. **Routing decisions:**
   - High confidence (≥0.9): Accept directly
   - Medium (0.75-0.9): Cross-validate with secondary model (IndicWhisper or MMS)
   - Low (<0.75): Escalate to Gemini 3 Pro or flag for review
4. **IndicMFA forced alignment** as final quality gate—alignment failure indicates transcription error
5. **NoRefER** for reference-free WER estimation using attention patterns

For human validation, use stratified sampling: review 0.1% of high-confidence segments, 2% of medium, and 10% of low-confidence segments. This achieves statistical coverage while focusing effort on risky data.

Audio quality filtering should apply these thresholds:
- **SNR ≥ 32 dB** for TTS-quality data (20-32 dB acceptable with enhancement)
- **Segment duration 4-10 seconds** optimal for attention-based TTS models
- **Single speaker** for voice cloning applications (multi-speaker for conversational TTS)
- **No background music** or significant reverberation

Tools for automated quality estimation include HuggingFace's DataSpeech (SNR, reverberation, speech rate) and NISQA/SQUIM for MOS prediction without reference audio.

## Conclusion: Production pipeline orchestration

The complete pipeline for your 500K YouTube podcast segments should flow through six stages: ingestion (extract FLAC, segment by VAD), audio quality analysis (DataSpeech metrics), multi-model transcription (Gemini Flash primary, cascading to Pro), validation and consensus (alignment + confidence scoring), text normalization (Indic NLP Library), and output formatting (JSONL with multi-representation transcripts).

Parallelize using Flyte map tasks or Ray, with per-batch checkpointing every 100-500 batches. Store processed segments in **1-5 GB TAR archives** with JSONL manifests. Expected throughput: ~4,200 hours of audio processable in 2-3 days on a properly configured GPU cluster.

The novel insights from this research that diverge from common practice: IndicMFA's G2G approach eliminates the pronunciation dictionary bottleneck that has historically limited Indic forced alignment; single-script transliteration enables code-mixed TTS without labeled code-mixed recordings; and emotion capabilities should be introduced during SFT rather than pre-training, despite the intuition that more training data exposure would help. These patterns, validated across CosyVoice, FireRedTTS, and Inworld implementations, provide a production-ready blueprint for building human-quality Indic TTS at scale.


**Planning transcript strategy by reviewing research files**

I'll gather insights from gemini_research.md and related model files to identify transcript advantages and craft a hybrid approach targeting low WER on Indic languages using ensemble methods and multi-model validation.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Architectural Blueprint for a Generative Indic TTS System: Leveraging Large-Scale Podcast Data and Multimodal LLMs1. Introduction: The Paradigm Shift in Speech SynthesisThe domain of Text-to-Speech (TTS) synthesis is currently undergoing a fundamental paradigm shift, moving from statistical parametric synthesis and end-to-end neural pipelines (such as Tacotron and FastSpeech) toward generative Speech Language Models (SpeechLMs). This transition is driven by the recognition that human speech is not merely an acoustic signal reconstruction task but a semantic modeling challenge that requires deep understanding of context, paralinguistics, and conversational dynamics. For Indic languages—characterized by complex code-switching (e.g., Hinglish, Tanglish), rich morphological structures, and diverse prosodic patterns—this shift presents both a formidable challenge and an unprecedented opportunity.Traditional acoustic models have historically struggled with "in-the-wild" data, requiring clean, studio-recorded corpora to produce intelligible speech. However, the emergence of architectures such as Inworld TTS-1 1, CosyVoice 3 1, and FireRedTTS-2 1 demonstrates that scaling training compute and data volume allows models to learn directly from noisy, unstructured audio, provided the semantic conditioning is sufficiently robust. These models treat speech synthesis as a next-token prediction task, effectively bridging the gap between Large Language Models (LLMs) and audio generation.This report articulates a comprehensive strategy for developing a human-like Indic TTS system using uncurated YouTube podcast data. Podcasts represent a rich repository of naturalistic speech, containing the disfluencies, interruptions, laughter, and code-switching that characterize real human communication. However, utilizing this data requires a sophisticated curation pipeline. We propose leveraging the multimodal reasoning capabilities of Gemini 3.0 Pro and Flash to generate high-fidelity, verbatim transcriptions enriched with audio event tags. Furthermore, we delineate a hybrid training curriculum that synthesizes the best practices from state-of-the-art research—specifically, the use of Supervised Multi-task Training (SMT) for tokenization, interleaved text-speech formatting for context maintenance, and Low-Rank Adaptation (LoRA) for stylistic control—to determine the optimal phase for introducing code-switching and paralinguistic features.2. The Data Curation Pipeline: Multimodal Transcription with Gemini 3.0The quality of an autoregressive TTS model is strictly bounded by the fidelity of its training data. In the context of podcast data, "fidelity" refers not just to audio quality, but to the precision of the textual transcription and its alignment with the acoustic reality. Standard Automatic Speech Recognition (ASR) systems typically normalize speech, removing "umms," "ahs," and stuttering to produce readable text. For a generative TTS aiming to sound human, this normalization is destructive; it erases the very features the model needs to learn.2.1. The Role of Gemini 3.0: Verbatim Stenography vs. Semantic ReasoningGemini 3.0, with its massive context window (up to 1 million tokens in Flash and Pro) and multimodal native architecture, offers a distinct advantage over traditional ASR models.2 It can process audio and text simultaneously, allowing for "context-aware transcription" where the model uses semantic cues to resolve acoustic ambiguities—a frequent occurrence in code-mixed Indic speech.2.1.1. Model Selection: Gemini 3.0 Pro vs. Flash for ASRThe choice between Gemini 3.0 Pro and Gemini 3.0 Flash is critical and depends on the specific stage of the data pipeline.Gemini 3.0 Pro: This model excels in complex reasoning and instruction following.4 Benchmarks indicate it significantly outperforms previous iterations in multimodal understanding and long-context retrieval.5 For the initial generation of "ground truth" transcripts, Gemini 3.0 Pro is the recommended engine. Its "Deep Think" capabilities allow it to better adhere to complex formatting constraints, such as identifying specific speakers in a multi-party podcast or distinguishing between a cough and a laugh based on context.6Gemini 3.0 Flash: While significantly faster and more cost-effective (processed at ~218 tokens/second), evaluations suggest that Flash models can suffer from "timestamp hallucination" and compression issues.7 Users have reported that Gemini 3.0 Flash may compress timestamps (e.g., mapping a 5-minute audio to a 3-minute timeline) or round them to the nearest second, lacking the centisecond precision required for TTS alignment.7 Therefore, Flash should be reserved for secondary metadata tasks—such as topic classification, sentiment analysis, or filtering non-speech segments—rather than the primary verbatim transcription task where temporal precision is paramount.2.1.2. Prompt Engineering for Verbatim Fidelity and Audio EventsTo extract a training-ready corpus from podcasts, the system prompt must override the LLM's natural tendency to summarize or clean up text. The goal is to generate a "screenplay" of the audio.System Instruction Design:The prompt must enforce a strict schema that captures three parallel streams of information: linguistic content, paralinguistic events, and speaker diarization.Verbatim Directive: "You are an expert acoustic phonetician. Transcribe the provided audio file with absolute verbatim accuracy. Do not summarize. Do not correct grammar. Include all disfluencies, filler words (e.g., 'umm', 'uh', 'like'), repetitions, and stutters exactly as spoken. If a speaker trails off, mark it with '...'." 8Audio Event Tagging: The prompt must explicitly define the set of allowed audio tags to prevent vocabulary explosion in the TTS model. Based on Inworld TTS-1's methodology, valid tags should include [laugh], [sigh], [breath], [cough], [throat_clear], and [yawn].1 The instruction should specify: "Insert non-verbal tags exactly where the sound occurs in the temporal flow, even if it interrupts a word. Do not place them at the end of the sentence unless the sound occurs there." 10Speaker Diarization: "Identify distinct speakers and label them as Speaker A, Speaker B, etc. If the speaker changes mid-sentence, insert the new speaker tag immediately." 11This rigorous prompting transforms the LLM from a passive transcriber into an active annotator, creating a dataset where every acoustic event has a corresponding textual token.2.2. The Code-Switching Conundrum: Script Strategies for Hinglish/TanglishA defining characteristic of the target data is code-mixing—the fluid alternation between English and Indic languages. Handling this textually is one of the most significant engineering decisions in the pipeline.2.2.1. The Failure of Pure RomanizationA common approach is to transliterate all Indic text into the Roman (Latin) script (e.g., writing Hindi as "Main ghar ja raha hoon"). While this simplifies the character set, research indicates it is detrimental to TTS quality.Phonetic Ambiguity: The Roman script lacks the graphemic density to represent Indic phonemes accurately. For instance, the dental 't' (त) and retroflex 't' (ट) in Hindi are distinct phonemes but are both mapped to 't' in Roman transliteration. A model trained on this ambiguous data often suffers from "accent drift," producing anglicized pronunciations of Indic words.12LID Confusion: Language Identification (LID) models struggle to distinguish between Romanized Hindi, Romanized Urdu, and English, leading to poor routing in multilingual systems.122.2.2. The Hybrid Script RecommendationTo achieve human-like prosody and accurate pronunciation, this report recommends a Hybrid Script Strategy:English words are transcribed in the Roman script.Indic words are transcribed in their Native script (Devanagari for Hindi, Tamil script for Tamil).Prompting Implementation:The system instruction to Gemini 3.0 must explicitly enforce this: "Transcribe English words in the Roman script. Transcribe Hindi/Tamil words in their native script. Do not transliterate Indic words into Roman script unless the speaker explicitly spells them out." 12Why this works:This strategy effectively provides the TTS model with an explicit, token-level Language ID. When the model encounters Devanagari tokens, it switches its internal acoustic priors to the Indic phoneme space; when it encounters Roman tokens, it switches to English. This preserves the native pronunciation of Indic content while maintaining the correct stress patterns for English loanwords, directly addressing the "schwa deletion" and stress mapping challenges inherent in Hinglish TTS.142.3. Temporal Alignment: The Two-Pass ProtocolWhile Gemini 3.0 provides text and approximate timestamps, its precision is often insufficient for cutting training data, which requires alignment at the phoneme or frame level (<20ms). "Flash" models, in particular, have shown regression in timestamp accuracy.7 Therefore, a Two-Pass Alignment Protocol is necessary.2.3.1. Pass 1: Semantic Transcription (Gemini 3.0 Pro)Gemini generates the verbatim, code-mixed, event-tagged transcript. This provides the "Ground Truth" text sequence.2.3.2. Pass 2: Forced Alignment (MFA / Wav2Vec2)The Gemini transcript and the raw audio are fed into a Forced Alignment tool to generate precise start and end times for every phoneme and word.Montreal Forced Aligner (MFA): This is the industry standard for alignment. For code-mixed data, MFA must be initialized with a merged pronunciation dictionary that combines the CMU Dict (for English) with an Indic Lexicon generated via Grapheme-to-Phoneme (G2P) rules.15 MFA uses GMM-HMM acoustic models, which are highly robust for boundary detection in clean speech.Wav2Vec2-based Alignment: For podcast data with significant background noise or music where MFA might fail, a fine-tuned Wav2Vec2 model (like IndicWav2Vec 16) using CTC-segmentation is recommended.17 These models are more robust to noise and can align text even in challenging acoustic environments.This two-pass approach leverages the reasoning of Gemini for what was said and the signal processing of MFA/Wav2Vec2 for when it was said, creating a dataset with high semantic and temporal fidelity.ComponentRoleRecommended ToolKey ConstraintTranscriptionText GenerationGemini 3.0 Pro"Screenplay" Prompt; Hybrid ScriptDiarizationSpeaker IDGemini 3.0 ProExplicit Turn-Taking TagsEvent DetectionParalinguisticsGemini 3.0 Pro + Audio ClassifierVerify tags with YAMNet/BEATsAlignmentTemporal SlicingMFA / Wav2Vec2Merged EN/Indic Dictionary3. Tokenizer Architecture: The Foundation of ExpressivityIn the modern "SpeechLM" paradigm, the continuous audio waveform is discretized into a sequence of tokens using a Neural Audio Codec. The quality and semantic richness of these tokens define the upper bound of the TTS model's performance. If the tokenizer compresses a "sigh" into generic noise tokens, the LLM will never learn to generate a sigh, regardless of how well it is prompted.3.1. Limitations of Standard TokenizersStandard neural codecs like EnCodec or SoundStream are typically trained with a reconstruction objective (minimizing Mean Squared Error and perceptual loss). While effective for compression, this approach is "semantically blind." It treats paralinguistic events (laughter, breath) and background noise equivalently, often resulting in poor representation of the emotional content essential for human-like speech.3.2. Supervised Multi-task Training (The CosyVoice 3 Methodology)CosyVoice 3 introduces a critical innovation: Supervised Multi-task Training (SMT) for the tokenizer.1 Instead of training solely on reconstruction, the tokenizer's encoder is trained with auxiliary prediction heads for semantic tasks.Recommended Tokenizer Architecture:Base: A quantized autoencoder (e.g., X-codec2 or HiFi-Codec) using Residual Vector Quantization (RVQ) or Finite Scalar Quantization (FSQ).1Auxiliary Losses: The encoder representations should be fed into auxiliary heads to predict:Automatic Speech Recognition (ASR): Forces the latent tokens to retain phonetic/linguistic content.Speech Emotion Recognition (SER): Forces the latent tokens to encode emotional states (e.g., Happy, Sad, Angry).1Audio Event Detection (AED): Forces specific tokens to represent non-verbal events like laughter or breathing.Speaker Verification: Preserves speaker identity information in the tokens.Impact:This training regime produces "semantic audio tokens." A specific token sequence becomes strongly correlated with "laughter" or "questioning intonation" because the encoder was explicitly penalized for losing that information. When the downstream LLM generates these tokens, it is invoking rich, pre-learned acoustic concepts rather than just raw waveforms. This is the "bridge" that allows text tags like [laugh] to translate effectively into acoustic laughter.4. Training Curriculum: Staging Capability and ControlThe creation of a robust, controllable TTS system is not a monolithic process but a staged curriculum. Drawing from the methodologies of Inworld TTS-1, FireRedTTS-2, and CosyVoice 3, we propose a three-phase training strategy: Pre-training, Post-training, and Alignment.4.1. Phase 1: Pre-training (Capacity Building)Objective: Build a robust "World Model" of speech acoustics, text-audio alignment, and multilingual phonetics.Data: The full 1M+ hour corpus derived from YouTube podcasts and other sources.1Filtering: Unlike traditional TTS which requires studio-quality data, this phase should utilize the noisy, natural podcast data. Crucially, do not filter out breathing, hesitation, or minor background noise. The model must learn the statistical distribution of real human speech, not just "reading voice".1Input/Output: The model (a Decoder-only Transformer, e.g., 1B-3B parameters) is trained on the Next-Token Prediction task.Input: Text Tokens (Hybrid Script) + Speaker Prompts.Target: Audio Tokens (from the SMT tokenizer).Role of Audio Tags: In this phase, implicit learning is prioritized. While audio tags ([laugh]) can be present in the text, the primary goal is for the model to learn the correlation between text semantics and acoustic prosody (e.g., that the text "That's hilarious!" often precedes the acoustic tokens for laughter).4.2. Phase 2: Post-Training / Instruction Tuning (The Control Layer)This phase answers the user's specific query about when to introduce features. The consensus from Inworld TTS-1 1 and FireRedTTS-2 1 is that explicit control mechanisms should be refined during Post-training or Supervised Fine-Tuning (SFT).4.2.1. Handling Dialogue: The Interleaved Format (FireRedTTS-2)To model the conversational flow of a podcast (turn-taking, interruptions), the data must be structured to preserve context. FireRedTTS-2 proposes the Text-Speech Interleaved Format 1:Format:  <Text A> <Audio A> <Text B> <Audio B>...Mechanism:By feeding the model the history of the conversation (both text and audio tokens of previous turns), the generation of "Speaker B" is conditioned on the prosody and content of "Speaker A."Result: This enables features like Backchanneling (e.g., saying "hmm" or "yeah" at appropriate times) and Prosodic Matching (matching the energy or emotion of the previous speaker). This is impossible if the model is trained on isolated sentence pairs.4.2.2. Stylistic Control via LoRA (Inworld TTS-1)To enable explicit control over audio events ([laugh], [sigh]), Inworld TTS-1 utilizes Low-Rank Adaptation (LoRA) fine-tuning.1The "Paired Utterance" Strategy:Instead of flooding the model with tags during pre-training (which can lead to overfitting), specific styles are taught via contrastive pairs.Data Selection: Isolate segments containing specific events (e.g., laughter).Neutral Pairing: Pair these with "neutral" segments from the same speaker.Fine-Tuning: Train a LoRA adapter to distinguish between the prompts:[Neutral] Text... $\rightarrow$ Neutral Audio Tokens[Laughing] Text... $\rightarrow$ Laughing Audio TokensWhy LoRA?Fine-tuning the entire model on stylized data can cause Catastrophic Forgetting, where the model loses its general speaking ability or "forgetting" how to speak neutrally. LoRA isolates the "style vector" into a small set of trainable parameters, allowing the style to be triggered on demand without degrading the base model's stability.14.3. Phase 3: Alignment (The Quality Layer)The final phase addresses the stability issues inherent in autoregressive models, such as mumbling, repeating, or hallucinating audio.4.3.1. Differentiable Reward Modeling and GRPOCosyVoice 3 and Inworld utilize Reinforcement Learning (RL) to polish the output.1Methodology:Reward Model: Train a reward model to score generated audio based on:Intelligibility: ASR Word Error Rate (WER). This is critical for code-mixed speech to ensure the model doesn't garble the language switching.Speaker Similarity (SIM): Cosine similarity between the generated audio embedding and the target speaker embedding.Naturalness: DNSMOS scores (perceptual quality).Optimization: Use Group Relative Policy Optimization (GRPO) (Inworld) or a Differentiable Reward Loss (CosyVoice 3) to update the model.Process: The model generates multiple candidate audio clips for a given text. The reward model scores them. The policy is updated to increase the probability of the high-scoring candidates.Outcome:This phase "tightens" the distribution, suppressing artifacts and ensuring that the paralinguistic events (laughs/sighs) generated in the previous phase are natural and do not disrupt intelligibility.5. Summary Comparison of MethodologiesThe following table synthesizes the specific contributions of the referenced architectures to the proposed strategy:FeatureInworld TTS-1 CosyVoice 3 FireRedTTS-2 Recommended StrategyTokenizerStandard VQSupervised Multi-task (SMT)Streaming (12.5Hz)SMT (CosyVoice 3) for semantic richness.Dialogue ModelingContextual PromptingIn-context LearningInterleaved Text-SpeechInterleaved (FireRedTTS-2) for podcast flow.Style ControlLoRA Fine-tuningInstruction VectorsImplicit ContextLoRA (Inworld) for explicit [tag] control.Data Script---Hybrid Script (Native/Roman) for Hinglish.AlignmentGRPO (RL)Diff. Reward Model-GRPO (Inworld) for stability and intelligibility.6. ConclusionBuilding a human-like Indic TTS system is a convergence of data engineering and advanced modeling. The strategy outlined in this report leverages Gemini 3.0 Pro not just as a transcriber, but as a semantic annotator, generating a "screenplay" of audio events and code-switching that standard ASR misses. By adopting the Hybrid Script strategy, we resolve the phonetic ambiguities of Hinglish/Tanglish.The architectural recommendations synthesize the best of current research: using CosyVoice 3's Supervised Multi-task Tokenizer to ensure the model can represent paralinguistics, FireRedTTS-2's Interleaved Format to capture the flow of dialogue, and Inworld's LoRA/GRPO framework to provide control and stability. This holistic approach ensures the final model does not merely read text, but performs it with the cultural and emotional nuance characteristic of high-quality podcasts.


**Checking gemini.md contents**

I'm focusing on gemini.md next to see if it contains essential details relevant for the response before exploring other large files.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
This is an impressive engineering feat. You are essentially building an Indic-specialized dataset that rivals what Meta (Voicebox) and Alibaba (CosyVoice) have done, but with a smarter, leaner pipeline.

Given your scale (~120k hours) and your goal (human-like TTS with high controllability), **transcription is your single biggest point of failure.** If the text doesn't match the audio *perfectly* (including stuttering, laughs, and code-switching), your TTS model will hallucinate artifacts.

Here is the strategy to solve your Transcription, Code-Switching, and Validation challenges.

### **Phase 1: The "Strict-mode" Transcription Strategy**

You are right to be worried about LLM creativity. Temperature `0.0` is not enough. You need to constrain the model using **Structured Generation (JSON)** and a **"Critic" persona**.

**Don't use just one transcript.** You need a multi-view output to handle the code-switching and script issues effectively.

#### **1. The Transcription Schema (JSON)**

Instead of asking for one string, force Gemini to categorize the speech into specific buckets. This solves your "Hinglish vs. Native" dilemma by capturing **both**.

**Proposed JSON Structure:**

```json
{
  "transcription_verbatim_roman": "are bhaai, kya kar rhe ho? [laugh]",
  "transcription_verbatim_native": "अरे भाई, क्या कर रहे हो? [laugh]",
  "transcription_normalized_native": "अरे भाई, क्या कर रहे हो?",
  "primary_language_code": "hi",
  "is_code_mixed": true,
  "audio_events": ["laugh"],
  "speaker_emotion": "amused",
  "confidence_score": 9
}

```

#### **2. The "Anti-Hallucination" System Prompt**

You must instruct Gemini that it is **not** an assistant, but a **Forensic Transcriber**.

> **System Prompt:**
> "You are an automated ASR (Automatic Speech Recognition) system for a TTS dataset. Your ONLY goal is forensic accuracy.
> 1. **Verbatim is Law:** Transcribe exactly what is heard, including grammatical errors, repetitions (stuttering), and filler words (um, uh).
> 2. **Audio Events:** Detect human non-speech sounds and insert them as tags from this allowed list ONLY: `[laugh]`, `[breath]`, `[sigh]`, `[cough]`, `[clearing_throat]`, `[cry]`.
> 3. **Script Handling:**
> * `transcription_verbatim_roman`: Write exactly how a user would type this on a smartphone (Hinglish/Tanglish).
> * `transcription_verbatim_native`: Write the script native to the *spoken* language.
> * **Critical Rule for Code-Switching:** If a Telugu speaker says a Hindi phrase (e.g., "Main seb khata hoon"), write it in Devanagari inside the Telugu sentence for the `native` field, but use the Telugu script for the surrounding words.
>
>
> 4. **Numbers:** Convert "123" to spoken form "one hundred and twenty-three" (or native equivalent) in the `verbatim` fields.
> 5. **Hallucination Check:** If the audio is unintelligible or music, return an empty string. Do not invent text."
>
>

---

### **Phase 2: Solving the Code-Switching & Script Dilemma**

You asked how to handle: *“arey annai, em chestunnav?”* vs *“అరేయ్ అన్నయ్య, ఏం చేస్తున్నావ్?”*

**The CTO Strategy:**
You need **BOTH**. Modern TTS models (like CosyVoice/Inworld) use tokenizers that can technically handle mixed scripts, but training on "Romanized Indic" often degrades pronunciation quality because Roman characters are phonetically ambiguous for Indian languages.

* **Training Data Strategy:**
* **Primary Training:** Use `transcription_verbatim_native`. This grounds the model in the correct phonetics of the language (e.g., distinguishing retroflex sounds in Telugu/Hindi).
* **Robustness Training (10-15% of data):** Train on `transcription_verbatim_roman`. This teaches the model to handle "Hinglish" text input from users at inference time.
* **The "Telugu quoting Hindi" Scenario:**
* *Audio:* "antaru... main seb khata hoon"
* *Transcript (Native):* "... అంటారు 'मैं सेब खाता हूँ'"
* *Why:* Your tokenizer will likely be multilingual (e.g., 50k+ vocab size). It will see Telugu tokens, then Devanagari tokens. As long as your model architecture (like XCodec2 + LLM) supports these tokens, it will learn that Devanagari tokens trigger Hindi phonemes, even if the surrounding context is Telugu.

---

### **Phase 3: Validation (The Missing Link)**

You cannot blindly trust Gemini-3-Flash. It *will* hallucinate text for silent audio or interpret background noise as words.

**Solution: The MFA (Montreal Forced Aligner) Filter**
You don't need to train an aligner from scratch. Use **pretrained Indic models** (available via MFA or AI4Bharat) to validate the Gemini transcripts.

1. **Pipeline:**
* `Audio` + `Gemini Native Transcript` → **MFA Aligner**.

2. **Logic:**
* MFA attempts to align the phonemes of the transcript to the audio.
* **Metric:** Check the **Phone Confidence Score** or **Unaligned Word Count**.
* **Pass:** If > 90% of words align with high confidence.
* **Fail:** If MFA cannot align the text (usually means Gemini hallucinated a sentence that wasn't spoken).

3. **Action:** Discard "Fail" segments. Do not try to fix them; you have 120k hours, you can afford to lose 10% dirty data to ensure 90% purity.

---

### **Phase 4: Emotional & Audio Event Strategy**

You asked: *When to fuse emotions? SFT or Post-train?*

**Do it in Pre-training.**
For a "Human-like" model, these events (`[laugh]`, `[breath]`) are not "instructions"—they are part of the vocabulary.

1. **Tokenizer:** Add `[laugh]`, `[breath]`, etc., as **special tokens** in your tokenizer (just like `<EOS>`).
2. **Training:**
* Train the model on the text *containing* these tags: "Hello [breath] how are you? [laugh]".
* The model learns to predict the `[laugh]` token just like it predicts the word "you".
* When the codec decoder sees the audio features associated with that token, it generates the sound.

3. **Inference Control:**
* Because the model learned `[laugh]` as a token, you can now *force* a laugh by inserting that tag in the input text prompt.

**Recommendation:** Do not rely on a separate "Emotion Encoder" (like a Reference Audio style transfer) alone. Integrating discrete event tokens into the text sequence gives you far finer control.

---

### **Summary of Next Steps**

1. **Select Model:** **Gemini 1.5 Pro** (or 2.5/3 Pro) is significantly better at following complex JSON schemas and "verbatim" instructions than Flash. Given you are processing snippets (not long context), the cost might be higher, but for *training data*, quality > quantity. **Test Flash first with the prompt below; if it hallucinates, switch to Pro.**
2. **Deploy Worker:** Create a Cloudflare Worker that sends the audio URL to Gemini with the schema below.
3. **Validate:** Set up a small VPS running **MFA** to spot-check the first 1,000 generated transcripts.

**The Golden Prompt for your Worker:**

```text
Input: Audio File
Task: Transcribe the speech for a Text-to-Speech model training dataset.

Constraints:
1.  VERBATIM ONLY: Do not paraphrase. Keep stutters, repetitions, and grammar errors.
2.  SCRIPT:
    - If the speaker speaks Telugu, use Telugu script.
    - If the speaker switches to English (Code-switching), use English script for those specific words.
    - If the speaker quotes Hindi, use Devanagari script for those words.
3.  EVENTS: Insert tags [laugh], [breath], [cough], [giggle] exactly where they occur.
4.  FORMAT: Return a JSON object.

JSON Schema:
{
  "transcription_native": "The mixed script string (e.g. Telugu + English words)",
  "transcription_romanized": "The romanized version of the entire text",
  "contains_code_mixing": boolean,
  "detected_emotions": ["happy", "neutral", "sad"],
  "audio_quality_score": (1-10 integer, assess background noise/clarity)
}

```

This strategy aligns with your "podcast quality" assumption while building safety rails against LLM hallucinations. Validating with MFA is the industry standard for this exact workflow.


**Evaluating additional docs**

Considering whether to continue with gemini.md or start new files like gpt_research.md, claude.md, and kimi.md by briefly reviewing current content.