Instruction:
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <approval_policy>on-request</approval_policy>
  <sandbox_mode>workspace-write</sandbox_mode>
  <network_access>restricted</network_access>
  <shell>bash</shell>
</environment_context>


<environment_context>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>


# Context from my IDE setup:

## Open tabs:
- Reference1.md: docs/Reference1.md
- Agents.md: Agents.md
- task.md: docs/task.md
- Reference5.md: docs/Reference5.md
- Reference6.md: docs/Reference6.md

## My request for Codex:
Task - [task.md](task.md)
Reference docs by other AI agents - /home/ubuntu/maya-transcribe/docs
Agents - [Agents.md](Agents.md)
secrets - /home/ubuntu/maya-transcribe/.env


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 28
drwxrwxr-x  3 ubuntu ubuntu 4096 Jan 13 03:28 .
drwxr-x--- 43 ubuntu ubuntu 4096 Jan 13 02:33 ..
-rw-rw-r--  1 ubuntu ubuntu 7238 Jan 13 03:28 Agents.md
drwxrwxr-x  2 ubuntu ubuntu 4096 Jan 13 03:12 docs
-rw-rw-r--  1 ubuntu ubuntu 3652 Jan 13 02:35 .env
-rw-rw-r--  1 ubuntu ubuntu 2627 Jan 13 02:35 task.md


Summary: auto


**Searching for AGENTS.md**

I’m noticing that there’s an Agents.md file at the root, but no AGENTS.md file that I can find. The user provided instructions about AGENTS, so it seems like the file might not exist after all. The spec states that repos often contain AGENTS.md files, yet the instructions from the user are what I should consider. I’ll make sure to search for AGENTS.md in the repo to confirm.


find .. -maxdepth 3 -name 'AGENTS.md' -o -name 'AGENT.md' -o -name 'agents.md'


Output:
Exit code: 0
Wall time: 0 seconds
Output:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 84
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 13 03:12 .
drwxrwxr-x 3 ubuntu ubuntu  4096 Jan 13 03:28 ..
-rw-rw-r-- 1 ubuntu ubuntu 11196 Jan 13 03:09 Reference1.md
-rw-rw-r-- 1 ubuntu ubuntu 18053 Jan 13 03:09 Reference2.md
-rw-rw-r-- 1 ubuntu ubuntu  5123 Jan 13 03:09 Reference3.md
-rw-rw-r-- 1 ubuntu ubuntu 13601 Jan 13 03:09 Reference4.md
-rw-rw-r-- 1 ubuntu ubuntu 11999 Jan 13 03:10 Reference5.md
-rw-rw-r-- 1 ubuntu ubuntu  4816 Jan 13 03:10 Reference6.md


**Checking file paths**

Okay, I'm looking at the task file paths. The user mentioned opening "docs/task.md," but the root directory lists "task.md" directly. I wonder if "docs/task.md" actually exists. The open tabs show references, which adds to my confusion. Maybe there's a symlink or a relative path involved? I should check the presence of "docs/task.md" using a command like `ls`. Let’s see what I find out!


Output:
Exit code: 2
Wall time: 0 seconds
Output:
ls: cannot access 'docs/task.md': No such file or directory
-rw-rw-r-- 1 ubuntu ubuntu 2627 Jan 13 02:35 task.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
suppose i have scraped youtube videos and have cleaned with a robust cleaning pipeline and now have single speaker segments of audio files that are suitable for TTS, ASR or speech systems. Now in-order for me to train them, i need to transcribe these audio files. Assuming i have audio files of sizes from 1s atleast to like 30s, 50s and 60s max, how can i accurately create transcriptions for these. compute is not a constraint here. quality is. so, my youtube videos are majorly

Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali , Odia, Marathi, Assamese, English

the unknown ones are not flagged, i might try to get more metadata for them and try flagging them as per the title and description for language. but i was wondering if we have a way to identify the language as well in our pipeline. Now, assuming the situation here, for indian space and languages mentioned above, and most of these are high quality youtube podcasts and or are explanation videos that has minimal to low and are cleaned out of music. Given the situation and since we are anyway generating the transcripts offline, like it doesnt have to be real-time or latency, or quick, given no compute constraints, what's like the best way or the cleverest way possible to generate transcripts to these segments so that i get WER as minimal as possible. i can pass these segments to as many models as possible to compare them all and or get them checked by a superiour LLMs or are there any LLM specialized in indic, where i can maybe run these segments and they create transcripts like A/B testing, what's that cleverest way. when i said i have no constraints, i literally meant we can run any specialized indic llms that can help create transcriptions with prompt guidence like gemma, or anything else, but we need to strategically plan this to avoid hallucinations or creation of content rather transcribing the exiting ones. that doesnt mean we can't use LLM, we need to be clever here. what's that combination we can use like voxtral+whisper large ? or whisper + indic conformers or indicwhisper or what's that best combo we can use to get WER as minimal as possible, we just need to be smart enough and clever here since we are doing this offline and or have no constraints over compute or anything and given we have enough time and everything, we just can't compromise on quality or WER for these transcriptions, what's the go-to strategy that we can adapt to where we just can't afford to make mistakes and have high quality transcriptions to these audio segments so we can train the best in-class indic ASR and TTS models.


**Clarifying task intentions**

The user asked about "task.md," but it’s unclear if they want a written strategy document or have a question instead. I wonder if it’s about implementing code for the pipeline or creating a documentation plan? Since it’s in the repo, it might be for storing answers. There are also reference documents from other AI agents under docs/Reference*.md, which I should review for context. Agents.md could provide insight into personas or instructions, so I’ll look at that too.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Agents.md — Meta-Prompt + Workflow for “Reference Ensemble” Research (ASR Transcription Quality First)

## Role
You are a **coding + research agent** operating inside a repo/workspace that contains:
- `task.md` (the original problem statement + requirements)
- `Reference1.md` … `Reference6.md` (different LLM-proposed approaches + code snippets)
- Any supporting code/config files already present

Your job is to **read everything**, extract the best ideas, **reject weak or risky suggestions**, and synthesize a **single optimal plan** to produce **near-zero-error transcripts** for multi-lingual audio (offline; compute not a constraint; quality is).

---

## Non-Negotiables
1. **No invention.** Only claim something is “recommended” if it is supported by the references or is a clearly-labeled inference.
2. **Quality > Coverage.** If forced to choose, reject questionable segments rather than poison training data.
3. **Audio-grounded verification required.** Any transcript must be **verified against the audio** before being accepted.
4. **Make decisions.** For every major technique, explicitly decide: **Adopt / Adapt / Reject / Park** and justify.
5. **Keep this file updated.** This file (`Agents.md`) is the living artifact: top section = this prompt; below it = your analysis, TODOs, decisions, and final plan.

---

## Output Contract (What you must produce in THIS file)
By the end, `Agents.md` must contain:
- A crisp understanding of `task.md` (constraints, metrics, languages, tooling assumptions)
- A per-reference analysis for Reference1–Reference6:
  - Summary, unique contributions, strengths, failure modes, missing pieces
  - Concrete actionable items + what to ignore
  - Adopt/Adapt/Reject/Park decisions
- A unified **Master Plan** (pipeline + thresholds + verification + human loop)
- An implementation blueprint (data structures, job orchestration, evaluation harness)
- A validation protocol (gold set, WER/CER reporting, active learning loop)

---

## Step 0 — Read Order (do this first)
1. Read `task.md` fully.
2. Skim all `Reference*.md` quickly to build a map of what exists.
3. Then deep-read each reference, one by one, and complete the per-reference template below.

---

## Step 1 — Extract Requirements from `task.md`
Fill this section after reading `task.md`.

### Task Summary (from task.md)
- **Goal:**
- **Languages:**
- **Audio segment length range:**
- **Compute constraints:**
- **Quality target (WER/CER):**
- **Acceptance philosophy (precision-first vs recall-first):**
- **Any required tools/frameworks:**
- **Any banned tools/constraints:**

### Success Metrics
Define what “good”” means in numbers and gates.
- Primary: (WER / CER / SER)
- Secondary: rejection rate, human edit rate, alignment confidence, hallucination rate
- Must-meet thresholds:
  - **Max allowable WER/CER for accepted samples:**
  - **Max disagreement allowed between systems:**
  - **Minimum alignment confidence:**

### Assumptions (explicit)
List assumptions you will rely on (and mark what needs verification).
- A1:
- A2:
- A3:

---

## Step 2 — Create an Evaluation Rubric (used for every idea)
For each technique/tool/model suggested in references, score it:

**Scoring dimensions (0–5):**
- Expected quality gain
- Robustness across Indic languages + code-switching
- Verification friendliness (supports confidence/alignment scoring)
- Implementation complexity
- Failure risk (hallucinations, bias, brittle dependencies)

**Decision labels:**
- **Adopt** = use as-is
- **Adapt** = use but modify (thresholds, different model, narrowed scope)
- **Reject** = do not use
- **Park** = keep as optional experiment

---

## Step 3 — Per-Reference Processing Template (repeat for each)
For each `ReferenceN.md`, fill:

### ReferenceN — One-paragraph summary
- What is the approach in plain terms?

### Key Contributions (unique ideas worth preserving)
- C1:
- C2:
- C3:

### Strengths (why it could work well)
- S1:
- S2:

### Risks / Failure Modes
Be harsh and specific.
- R1:
- R2:

### Missing Pieces / Gaps
- G1:
- G2:

### Actionable Extract (convert into implementable tasks)
Each item must be concrete.
- [ ] A1:
- [ ] A2:

### Decisions
For each major suggestion:
- Suggestion: …
  - Decision: Adopt / Adapt / Reject / Park
  - Why:
  - What to do instead (if rejected):

### Integration Notes
How this reference fits with others.
- Conflicts with:
- Complements:
- Dependencies:

---

## Step 4 — Synthesis After All References
After finishing Reference1–Reference6, produce:

### Master Plan (single best pipeline)
Must include:
1. **Pre-QC** (audio sanity checks, VAD trimming, clipping/SNR checks)
2. **Language ID routing** (include handling for uncertain/mixed)
3. **ASR ensemble** (per language group; include at least 2 diverse systems)
4. **Normalization layer** (unicode, punctuation, numerals, script normalization)
5. **System combination** (e.g., voting/ROVER or other)
6. **Audio-grounded verification gate** (forced alignment / CTC scoring / confidence)
7. **Reject + re-decode logic** (different decoding params / alternative model)
8. **Human-in-the-loop for rejects** (only the tail; scalable workflow)
9. **Storage schema** (store all hypotheses + scores for auditability)

### Implementation Blueprint (engineering plan)
- Orchestration: batching, GPU scheduling, retries
- Data structures:
  - `segment_id`, `language_probs`, `hypotheses[]`, `rover_output`, `alignment_scores`, `decision`, `audit_log`
- Reproducibility: config-driven pipeline, versioned models, deterministic decoding where possible
- Observability: dashboards for rejection reasons, per-language error patterns

### Validation Protocol (prove it works)
- Build a **gold set** (stratified across languages, speakers, domains, lengths)
- Compute WER/CER by language + conditions
- Threshold tuning approach
- Active learning loop: rejects → human fix → retrain / recalibrate

### Final Deliverable
- A short “Do This Next” checklist that an engineer can execute immediately.

---

## Global TODO Checklist (tick as you complete)
- [ ] Read `task.md` and fill **Step 1**
- [ ] Create evaluation rubric (**Step 2**) and decision labels
- [ ] Complete Reference1 analysis
- [ ] Complete Reference2 analysis
- [ ] Complete Reference3 analysis
- [ ] Complete Reference4 analysis
- [ ] Complete Reference5 analysis
- [ ] Complete Reference6 analysis
- [ ] Write **Master Plan**
- [ ] Write **Implementation Blueprint**
- [ ] Write **Validation Protocol**
- [ ] Produce final “Do This Next” execution checklist

---

# Analysis Workspace (Fill below as you go)
> Everything below this line is your working output. Keep it clean and structured.

## Step 1 — Task Understanding (from task.md)
(TODO)

## Step 2 — Evaluation Rubric
(TODO)

---

## Reference1 — Analysis
(TODO: fill using the per-reference template)

---

## Reference2 — Analysis
(TODO)

---

## Reference3 — Analysis
(TODO)

---

## Reference4 — Analysis
(TODO)

---

## Reference5 — Analysis
(TODO)

---

## Reference6 — Analysis
(TODO)

---

## Master Plan (Unified)
(TODO)

## Implementation Blueprint
(TODO)

## Validation Protocol
(TODO)

## Do This Next (Execution Checklist)
(TODO)


**Planning document updates**

I need to update Agents.md as a "living artifact" with analysis and a final plan. This includes a thorough plan with the pipeline. First, we should examine Reference1-Reference6 in the docs. I’ll read them and take a quick look, then get into a deep read. I’m thinking about using `sed` for efficient reading and possibly `multi_tool_use.parallel` to manage context size while ensuring we synthesize the full details later.


**Inspecting reference files**

I need to check the file sizes for the references, and I've noticed they range from about 5k to 18k, which feels manageable. I’ll read each file fully using `sed`, but I’m aware the outputs might be large. To synthesize efficiently, I should keep the context small, aiming for around 80k tokens max. Gradually summarizing each reference will help me update Agents.md with the required per-reference analysis. I think I’ll start with Reference1.


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 220
Output:
## Target: near-zero transcript errors (offline, unlimited compute)

The highest-quality approach is **(1) route by language**, **(2) run a *diverse* ASR ensemble**, **(3) pick/merge outputs via *system combination***, and **(4) independently *verify against the audio* with forced-alignment / score-based gating**. Anything that doesn’t clear the gate goes to **human correction** (small %), not into training.

This beats “one giant model” because you get: **error diversity + consensus + audio-grounded verification** (and you can set the gate as strict as you want).

---

## 1) Spoken Language ID (LID) for routing

Do LID on every segment (even if you also have metadata). Use an audio LID model that covers Indic + unknowns:

* **Meta MMS LID (up to 4017 languages)** is designed for spoken LID at scale. ([Hugging Face][1])
* Optionally ensemble it with **SpeechBrain VoxLingua107 ECAPA LID** (strong on YouTube-like audio). ([Hugging Face][2])

**Practical routing rule:**

* If top-1 language prob ≥ 0.90 and margin(top1-top2) ≥ 0.20 → “confident route”
* Else → “unknown/mixed route” and transcribe with multilingual models + later re-check language from transcript.

This makes the rest of the pipeline much cleaner, especially for Hindi/Punjabi/English code-switching.

---

## 2) ASR ensemble (best-in-class for your language set)

### A) Indic languages: use AI4Bharat as your primary “Indic specialist”

AI4Bharat explicitly maintains state-of-the-art ASR work for India and lists **IndicConformer / IndicWhisper / IndicWav2Vec** as their ASR model line. ([ai4bharat.iitm.ac.in][3])
The **IndicConformer 600M multilingual** model supports **IN-22** and uses a **hybrid CTC + RNNT** design (useful because CTC and RNNT often fail differently). ([Hugging Face][4])

**Recommendation for each Indic segment (Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese):**

* Model A: **AI4Bharat IndicConformer-600M-Multilingual** (decode both CTC and RNNT if you can)
* Model B: **Whisper large-v3** as a strong generalist baseline ([Hugging Face][5])
* Model C (optional but valuable): **Meta MMS ASR** multilingual (1107 languages) for diversity ([Hugging Face][1])

### B) English: add the strongest open speech models

* **Voxtral** (open-weight speech understanding) reports better transcription WER than Whisper large-v3 on their published benchmarks and supports automatic language detection. ([Mistral AI][6])
* **NVIDIA Nemotron Speech ASR** is a separate open model line (strong for streaming/throughput; can still be used offline for English). ([Hugging Face][7])

**Recommendation for English segments:**

* Voxtral + Whisper large-v3 (+ Nemotron Speech as extra diversity)

> Note: Voxtral’s public claims highlight multilingual performance and explicitly mention Hindi among supported major languages; for Tamil/Telugu/etc you should treat Voxtral as “test-and-measure” rather than assumed best. ([Mistral AI][6])

---

## 3) Combine outputs (don’t “pick one”; *vote/merge*)

Once you have multiple hypotheses, do **system combination** instead of choosing a single model output.

Classic method: **ROVER (Recognizer Output Voting Error Reduction)** — combines multiple ASR hypotheses to reduce WER. ([NIST][8])

**What to do in practice**

1. Normalize all hypotheses (Unicode normalization, remove trivial punctuation differences, normalize numerals consistently).
2. Align hypotheses (token/word or character level).
3. Do **weighted voting** (weights can be per-language, per-domain, per-model based on a small dev set).
4. Produce:

   * a final transcript
   * a “disagreement map” (positions where models differed)

This yields a measurable win when recognizers have different failure modes (IndicConformer vs Whisper vs MMS usually do).

---

## 4) Audio-grounded verification gate (how you avoid hallucinations + silent errors)

You want a gate that answers: “Does this text truly match the audio?”

### A) Forced alignment / alignment scoring

* **Montreal Forced Aligner (MFA)** is a gold-standard forced alignment workflow (Kaldi-based). ([montreal-forced-aligner.readthedocs.io][9])
* Research comparing forced alignment approaches shows modern ASR-based aligners can underperform classic forced aligners in alignment accuracy. ([arXiv][10])

**Reality for Indic:** MFA requires lexicons/acoustic models per language; that may be annoying for 12+ languages. A very effective alternative is:

### B) CTC-based “which transcript fits best?” scoring

For each candidate transcript (or ROVER result):

* run a **CTC aligner** (e.g., wav2vec2-style) and compute:

  * mean per-character posterior
  * coverage (% characters aligned confidently)
  * spikes of low confidence

This is the key “clever” move: **use an *independent* model to score fit**, not just trust the generator.

WhisperX-style pipelines are built around ASR + forced alignment; it’s a known pattern to …965 chars truncated…. ([arXiv][13])

**Do this:**

* Always run a VAD pass (or use your existing speech/music cleanup) and **trim leading/trailing silence hard**
* For Whisper-like models, run **beam search with temperature=0** first; only fall back to higher temperature if decoding failed (and if you do, treat as low confidence and require alignment to pass)

---

## 6) How to use LLMs “safely” (no invented words)

Text-only LLMs (Gemma, etc.) should **not** be used to “transcribe” from scratch; they will invent plausible text. The safe uses are:

### A) Constrained post-editing (only when models mostly agree)

Give the LLM:

* the 2–3 hypotheses
* the disagreement spans only
* hard constraints: **must choose among provided options**, or output `<unk>` for that span
  This turns the LLM into a **tie-breaker**, not a generator.

### B) Audio-LLMs that actually listen

If you want “LLM help” that’s still audio-grounded, use models that take audio as input (e.g., Voxtral). ([Mistral AI][6])
Even then, keep the same rule: **it must pass the alignment gate** before entering training.

---

## 7) Human-in-the-loop for the hard 1–5% (still scalable)

No matter how much compute you throw, the tail exists (names, rare words, heavy code-switching, low SNR). The optimal strategy is:

* auto-transcribe + auto-verify
* only humans touch **the rejects**

If you want an OSS stack for this, AI4Bharat’s **Shoonya** is an open-source annotation platform built for data labeling at scale. ([shoonya.ai4bharat.org][14])

---

## 8) The “go-to” stack for your exact languages (recommended)

**Language ID**

* MMS LID-4017 (+ optional SpeechBrain VoxLingua107)

**Transcription ensemble**

* Indic: IndicConformer-600M (CTC + RNNT outputs) ([Hugging Face][4])

  * Whisper large-v3 ([Hugging Face][5])
  * MMS ASR (optional diversity) ([Hugging Face][1])
* English: Voxtral + Whisper large-v3 (+ Nemotron Speech as extra) ([Mistral AI][6])

**Combination**

* ROVER (weighted) ([NIST][8])

**Verification**

* CTC alignment scoring (Indic wav2vec2-family aligner) + strict reject rules
* MFA where you can support it cleanly ([montreal-forced-aligner.readthedocs.io][9])

**LLM**

* Only constrained tie-breaks (choose among options) OR audio-LLM transcription + still gated ([Mistral AI][6])

---

## Minimal “flow diagram” (what you implement)

```text
Audio segment
  -> VAD trim + QC (SNR, clipping, duration)
  -> LID (MMS LID + optional SpeechBrain)  -> language / unknown
  -> ASR ensemble (IndicConformer + Whisper + optional MMS / Voxtral)
  -> Normalize hypotheses
  -> ROVER combine + disagreement map
  -> Alignment scoring (CTC aligner) on (top candidates + ROVER)
  -> PASS?  yes -> final transcript
            no  -> re-decode settings OR send to human (Shoonya)
```

---

## One non-negotiable (if you truly “can’t afford mistakes”)

You must treat transcription as **precision-first**, not recall-first:

* It’s better to keep **70–90%** of segments with **near-perfect transcripts** than keep 100% with hidden label noise.
* The rejects are not wasted: they become your **active learning** pool for targeted human correction and later model improvements.

If you want, I can turn this into: (1) concrete threshold recommendations (per-language), (2) a model-ranking protocol using a 1–2k clip gold set across languages, and (3) the exact data structures for storing hypotheses + align scores so you can debug disagreements quickly.

[1]: https://huggingface.co/docs/transformers/en/model_doc/mms?utm_source=chatgpt.com "MMS"
[2]: https://huggingface.co/speechbrain/lang-id-voxlingua107-ecapa?utm_source=chatgpt.com "speechbrain/lang-id-voxlingua107-ecapa"
[3]: https://ai4bharat.iitm.ac.in/areas/asr "AI4Bharat"
[4]: https://huggingface.co/ai4bharat/indic-conformer-600m-multilingual "ai4bharat/indic-conformer-600m-multilingual · Hugging Face"
[5]: https://huggingface.co/openai/whisper-large-v3?utm_source=chatgpt.com "openai/whisper-large-v3"
[6]: https://mistral.ai/news/voxtral "Voxtral | Mistral AI"
[7]: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b?utm_source=chatgpt.com "nvidia/nemotron-speech-streaming-en-0.6b"
[8]: https://www.nist.gov/publications/post-processing-system-yield-reduced-word-error-rates-recognizer-output-voting-error?utm_source=chatgpt.com "Recognizer Output Voting Error Reduction [ROVER]"
[9]: https://montreal-forced-aligner.readthedocs.io/en/v3.2.3/user_guide/workflows/index.html?utm_source=chatgpt.com "Workflows available - Montreal Forced Aligner - Read the Docs"
[10]: https://arxiv.org/html/2406.19363v1?utm_source=chatgpt.com "A Comparison of Modern ASR Methods for Forced Alignment"
[11]: https://aikosh.indiaai.gov.in/home/models/details/ai4bharat_indicwav2vec_speech_model_for_hindi.html?utm_source=chatgpt.com "AI4Bharat - Hindi IndicWav2Vec Speech Model - AIKosh"
[12]: https://github.com/openai/whisper/blob/main/whisper/transcribe.py?utm_source=chatgpt.com "transcribe.py - openai/whisper"


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 118
Output:
High-Quality Transcription Strategy for Multi‑Lingual Indic Audio
Transcribing Indian-language audio segments with minimal word error rate (WER) requires a smart, multi-step pipeline. Below we outline a comprehensive strategy leveraging the best ASR models, ensembling techniques, and LLM-based error correction, given no constraints on compute or latency. The goal is to achieve near-human transcription quality (targeting ~1–2% WER) for languages including Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, and English.
1. Language Identification and Preparation
Before transcription, identify the language of each audio segment to route it to the optimal model. Many modern ASR models (e.g. Whisper, Voxtral, Gemma) can auto-detect language, but a dedicated LID (Language ID) model or metadata cues (video title/description) can provide extra confidence. Ensuring correct language detection is crucial so that the transcript uses the proper script and vocabulary. All audio should be normalized (e.g. convert to 16 kHz mono FLAC) as needed for the models.
Why: Indian languages can be acoustically similar (e.g. Punjabi vs. Hindi in some dialects), and accurate LID ensures using the right language model and decoding alphabet. Misidentification could degrade transcription quality.
2. Leverage State-of-the-Art ASR Models (No Single-Model Reliance)
With language known, transcribe using the strongest ASR models available. Since quality is paramount and compute is unlimited, we can ensemble multiple SOTA models per segment:
OpenAI Whisper (Large-V2) – A strong multilingual baseline. Whisper large-v2 is known for robust performance across many languages, including Indic ones. However, fine-tuned versions of Whisper on Indian data can drastically outperform the base model. For example, IndicWhisper (Whisper fine-tuned on 12 Indian languages, ~10.7k hours) achieved the lowest WER on 39 of 59 Indian ASR benchmarks, reducing WER by ~4.1 absolute on average
. Similarly, specialized models like Oriserve’s Hinglish Whisper (for Hindi-English code-mixed speech) show ~39% relative WER improvement over original Whisper on benchmarks
 and mitigate hallucinations in transcripts
. Using such fine-tuned models for each language (Hindi, Tamil, etc.) will yield more accurate transcripts than a generic model.
Mistral Voxtral (24B) – A new open-source ASR model that outperforms Whisper large-v3 across multiple languages
. Voxtral is a 24B-parameter “speech understanding” model with a Whisper-based audio encoder and an LLM backbone. It has state-of-the-art transcription accuracy on benchmarks (beating even some proprietary services) and native multilingual support (auto language detection, strong results in widely-used languages like English, Spanish, Hindi, etc.
). Given unlimited compute, running Voxtral on each segment can provide an extremely accurate transcript (often better than Whisper). It also supports long context (up to 30+ minutes) and has built-in understanding capabilities, although here we primarily use its transcription strength.
Google Gemma 3n – Google’s latest open multimodal model (small enough for offline use) with high-quality ASR for 140+ languages (including 6 major Indian languages)
. Gemma 3n is designed for on-device speech recognition and translation with chat-style prompting. We can prompt Gemma with instructions (e.g. “transcribe verbatim, do not add extra text”) to guide it. Its multilingual prowess and ability to follow prompts can help maintain accuracy without hallucination. For supported Indian languages (likely Hindi, Tamil, Telugu, Bengali, Marathi, Malayalam), Gemma 3n offers top-tier open-model performance, rivaling other SOTA models. It’s also optimized to run efficiently (even on a laptop) while delivering real-time or offline transcription.
Other Specialized Models: If available, incorporate any language-specific ASR models with known superior performance. For example, NVIDIA’s Conformer-CTC models for Hindi, or AI4Bharat’s IndicConformer (a 30M-parameter real-time model for 22 Indian languages
). These smaller models may not beat Whisper or Voxtral on accuracy, but if fine-tuned on accents or dialects, they could correct certain mistakes. They are optional given our focus on quality over speed, but can be part of the ensemble for diversity. Also, any big tech ASR APIs (Google Cloud, Azure, etc.) could be used on a sample basis to compare output – though not feasible for full data due to cost, their transcripts could serve as a reference for tricky cases.
Strategy: For each audio segment, run at least two or three different top-tier ASR models in parallel, for example Whisper/IndicWhisper, Voxtral, and Gemma. This yields multiple candidate transcripts for the same speech. Each model has different strengths, so their errors will differ. For instance, Voxtral might handle a Hindi-English mixed sentence differently than IndicWhisper; …7851 chars truncated…nscribers who catch context that AI misses
 – a small manual layer can elevate quality from “very good” to “virtually perfect”.
Feedback into Training: Optionally, use those human-corrected segments to further fine-tune your models or correction LLM. Since we have time and no compute limit, iterative improvement is possible. Over a few rounds, the need for human fixes will diminish as models learn from their mistakes.
6. Putting It All Together (Optimal Pipeline)
Finally, here’s a step-by-step pipeline summarizing the strategy:
Language ID & Segmentation: Confirm each audio segment’s language (Malayalam, Kannada, English, etc.) and ensure it’s in the right format (mono 16 kHz, etc.).
Multi-Model Transcription: Run the segment through multiple ASR models – e.g. Whisper Large (or IndicWhisper fine-tune)
, Voxtral Small/24B
, and possibly Gemma 3n or a specialized model. Allow Whisper to auto-detect language or specify the language to each model if needed (to avoid any confusion for languages like Assamese vs. Bengali script).
Consensus Assembly (ROVER): Align the outputs and perform majority voting or confidence-weighted voting to merge into one transcript. This resolves most obvious errors as demonstrated by ROVER’s improvements in WER
.
LLM/LM Correction Pass: Take the merged transcript and run a correction pass. This could involve:
A sequence-to-sequence correction model (trained on similar data) that outputs a cleaned transcript.
Or an LLM prompt that compares multiple versions (the individual ASR outputs plus the merged one) and fixes any anomalies without adding new content.
Optionally, verify uncommon words against a knowledge base or use an n-gram probability check to ensure the transcript is linguistically plausible
.
Final Review & Edit: If any segments are still suspect (e.g., marked by low confidence or by the LLM saying “unclear”), have a human reviewer listen and correct. This should be a small portion of the data thanks to the rigorous automated steps prior.
Output High-Quality Transcript: The end result for each segment is an extremely accurate transcript, ready for use in training TTS or ASR systems.
By adopting this multi-pronged approach – using the best ASR models (Whisper, Voxtral, Gemma, etc.) in combination, plus smart voting and LLM-guided correction – we maximize transcription quality. We effectively minimize WER by catching different types of errors at each stage: acoustic model errors via ensemble, and language/context errors via LLM. This is the “clever combo” that takes advantage of offline, unlimited compute conditions. In summary, the go-to strategy is: use robust multilingual ASR engines fine-tuned for Indic languages, compare their outputs, and refine with advanced language models and selective human oversight. With this strategy, it’s feasible to approach human-level transcription accuracy (WER on the order of 1–2%) for a wide array of Indian languages in offline processing
. The transcripts produced will be of extremely high quality, providing a solid foundation to train top-notch Indic ASR and TTS models. Sources: High-level ASR model performance and capabilities
, IndicWhisper fine-tuning gains
, Hinglish ASR improvements
, ROVER ensemble effectiveness
, and advanced error-correction techniques
 have all informed this approach. Each component is chosen to ensure no compromise on quality, aligning with the ultimate goal of accurate, reliable transcriptions for Indic audio.
Citations

[2305.15386] Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR

https://arxiv.org/abs/2305.15386

Oriserve/Whisper-Hindi2Hinglish-Prime · Hugging Face

https://huggingface.co/Oriserve/Whisper-Hindi2Hinglish-Prime

Oriserve/Whisper-Hindi2Hinglish-Prime · Hugging Face

https://huggingface.co/Oriserve/Whisper-Hindi2Hinglish-Prime

Voxtral | Mistral AI

https://mistral.ai/news/voxtral

Voxtral | Mistral AI

https://mistral.ai/news/voxtral

Gemma-fied! Putting the power of our most capable family of open models into the hands of every Indian developer

https://blog.google/intl/en-in/company-news/gemma-fied-putting-the-power-of-our-most-capable-family-of-open-models-into-the-hands-of-every-indian-developer/

AI4Bharat Models

https://models.ai4bharat.org/

[1706.07238] Automatic Quality Estimation for ASR System Combination

https://arxiv.org/abs/1706.07238

[1706.07238] Automatic Quality Estimation for ASR System Combination

https://arxiv.org/abs/1706.07238

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

Best option for audio or video transcription now? : r/LocalLLaMA

https://www.reddit.com/r/LocalLLaMA/comments/1oe5igo/best_option_for_audio_or_video_transcription_now/
All Sources


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Since compute is not a constraint and your goal is absolute minimal Word Error Rate (WER) to train SOTA models, you should not rely on a single ASR model. Even the best models (like Whisper V3) hallucinate or mishear specific Indian phonemes.

The "cleverest" strategy is to build a **Consensus & Arbitration Pipeline** (often called a ROVER system in research). You will run multiple SOTA models in parallel and use a "Judge" (LLM + Acoustic Aligner) to pick or merge the best result.

Here is the **Ultimate "Zero-Compromise" Offline Pipeline** for Indian Languages.

### Phase 1: The "Gatekeeper" (Ensemble Language Identification)

Since your files have missing metadata, you cannot trust a single LID (Language ID) model. If you feed Hindi audio to a Bengali model, you will get garbage that ruins your TTS training.

* **Strategy:** Run three distinct LID models on every file.
1. **OpenAI Whisper:** (Built-in language detection).
2. **SpeechBrain:** (Pre-trained `lang-id-voxlingua107-ecapa` classifier).
3. **Meta MMS / NVIDIA NeMo:** (Alternative architecture).

* **The Logic:**
* If all 3 agree  **Pass**.
* If they disagree  **Quarantine** for manual review or discard. *Do not risk polluting your dataset.*

### Phase 2: The "Council of Transcribers"

Run every audio segment through the Top 4 SOTA models. Each has different strengths.

| Model | Role | Why it's needed |
| --- | --- | --- |
| **AI4Bharat IndicWhisper** | **The Specialist** | **Crucial.** Fine-tuned specifically for Indian languages. It often outperforms standard Whisper on Dravidian languages (Malayalam, Tamil, Telugu, Kannada). |
| **OpenAI Whisper (Large-v3)** | **The Generalist** | Excellent general robustness and vocabulary. Acts as the baseline anchor. |
| **NVIDIA Canary / Parakeet** | **The Challenger** | Currently SOTA on the Open ASR Leaderboard. Very strong on multilingual tasks and often beats Whisper on speed/accuracy. |
| **Meta MMS (1B)** | **The Diversity Vote** | Trained on Massively Multilingual Speech. Useful tie-breaker for lower-resource languages like Odia or Assamese. |

### Phase 3: The "Judge" (Consensus & Validation)

Now you have 3-4 slightly different transcripts for the same audio. How do you pick the perfect one without a human?

#### A. The Acoustic Truth Filter (Forced Alignment)

This is the scientifically "clever" part. A transcript might look grammatically correct but be phonetically wrong (hallucination).

* **Tool:** Use **Montreal Forced Aligner (MFA)** or **NVIDIA NeMo Aligner**.
* **Action:** Force-align every transcript back to the audio.
* **Metric:** Calculate the **Acoustic Likelihood Score**.
* **Result:** If Whisper says "apple" but the audio sounds like "ample," the alignment score will drop. You pick the transcript with the highest alignment confidence.

#### B. The Semantic Judge (LLM Arbitration)

Use a high-intelligence LLM (GPT-4o, Claude 3.5 Sonnet, or a fine-tuned Indic LLM like **Sarvam-1**) as a "Reviewer."

* **Prompt:** *"I have three transcripts of the same Hindi audio file. Transcript A says [...], Transcript B says [...]. Which one is the most linguistically probable and preserves the specific Indian entity names best? Output only the best version."*
* **Clever Trick:** Use the **ROVER (Recognizer Output Voting Error Reduction)** approach. If 3 models agree on a word, keep it. If they disagree, let the LLM decide based on context.

### The Workflow Diagram

```mermaid
graph TD
    A[Raw Audio Segment] --> B{LID Ensemble}
    B -- Agreement --> C[Confirmed Language]
    B -- Disagreement --> D[Discard/Manual]

    C --> E[AI4Bharat IndicWhisper]
    C --> F[OpenAI Whisper V3]
    C --> G[NVIDIA Canary]

    E --> H[Transcript 1]
    F --> I[Transcript 2]
    G --> J[Transcript 3]

    H & I & J --> K{Alignment Check}
    K --> L[Score Acoustic Confidence]

    H & I & J & L --> M[LLM Consensus Judge]
    M --> N[Final Gold Transcript]

```

### Summary of Tools to Deploy

1. **Primary Model:** **AI4Bharat/IndicWhisper** (HuggingFace). This is non-negotiable for high-quality Indic TTS data.
2. **Secondary Model:** `faster-whisper` (Large-v3 implementation).
3. **Consensus Tool:** `meetecho/rover` or write a simple Python script using `difflib` to find differences.
4. **Alignment:** `Montreal Forced Aligner` (MFA) - pre-trained models are available for most Indian languages.

### One Final "Pro" Tip

Since you are training **TTS** (Text-to-Speech), you must be aggressive about **Speech Quality**, not just text accuracy.

* Use a model like **WavLM** or **Brouhaha** to score the audio for "Signal-to-Noise Ratio" (SNR) and "Reverberation."
* Even if the transcript is perfect, if the audio has background hiss, **throw it out**. Bad audio destroys TTS prosody faster than bad transcripts.

... [Deep dive into cleaning audio data for TTS](https://www.google.com/search?q=https://www.youtube.com/watch%3Fv%3DyYcZz4j73K8)

**Relevance:** This video explains the practical nuances of preparing dataset specifically for TTS training, focusing on why "clean" audio matters as much as the transcription itself.


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 276
Output:
This is an excellent problem to solve systematically. Given your constraints (offline, unlimited compute, quality-first), let me design a **multi-stage ensemble pipeline with LLM-as-Judge** architecture that minimizes WER while avoiding hallucination risks.

## The Core Strategy: Ensemble + Consensus + Verification

The key insight is: **Use multiple specialized ASR systems → Build consensus → Use LLM as a JUDGE (not generator) → Quality score and flag outliers**

---

## Stage 1: Language Identification (Critical First Step)

Since you have unknown language segments, start here:

| Model | Strength | Notes |
|-------|----------|-------|
| **IndicLID** (AI4Bharat) | Best for Indic languages | 22 Indic languages |
| **Whisper's LID** | Good fallback | Built-in during inference |
| **MMS-LID** (Meta) | 4000+ languages | Very robust |
| **NLLB-LID** | 200+ languages | Good accuracy |

**Strategy:** Run IndicLID + MMS-LID → If they agree, proceed. If disagree, use majority voting with Whisper's detection as tiebreaker.

---

## Stage 2: Multi-Model ASR Ensemble

For each audio segment, run ALL of these (since compute isn't a constraint):

### Tier 1: Indic-Specialized Models (Highest Priority)
| Model | Languages | Why |
|-------|-----------|-----|
| **IndicWhisper** (AI4Bharat) | All your languages | Fine-tuned Whisper on Indic, SOTA for most |
| **IndicConformer** (AI4Bharat) | 9 Indic languages | CTC-based, different architecture = diverse errors |
| **MMS** (Meta) | 1000+ including all Indic | Massive multilingual, different training data |
| **IndicWav2Vec2** | Indic languages | Self-supervised, good for low-resource |

### Tier 2: General Multilingual Models
| Model | Why |
|-------|-----|
| **Whisper Large V3** | Strong baseline, good for code-switching |
| **SeamlessM4T v2** (Meta) | Multimodal, different architecture |
| **Canary-1B** (NVIDIA) | Strong multilingual ASR |
| **USM** (Google, if accessible) | 300+ languages |

### Tier 3: Audio-Native LLMs (Use Carefully)
| Model | Why | Caution |
|-------|-----|---------|
| **Gemini 2.0 Flash** | Native audio input, good Indic | Can hallucinate - use for verification only |
| **Qwen2-Audio** | Direct audio understanding | Same caution |
| **Voxtral** (Mistral) | New, multimodal | Limited Indic evaluation data |

---

## Stage 3: Consensus Building Algorithm

This is where you get clever:

```
For each audio segment:
    1. Collect N transcriptions from N models
    2. Normalize all transcripts (unicode, punctuation, numbers)
    3. Compute pairwise WER/CER between all pairs
    4. Find the "centroid" transcript (lowest average distance to others)
    5. Weight by model confidence scores if available
    6. If high disagreement (variance > threshold) → flag for Stage 4
```

### The ROVER (Recognizer Output Voting Error Reduction) Approach:
- Align all hypotheses using dynamic programming
- At each position, vote across models
- Weight votes by model's known accuracy for that language

---

## Stage 4: LLM-as-Judge (NOT Generator)

This is the critical part where you avoid hallucination:

### ✅ DO: Use LLM to SELECT among candidates
```
You are an expert transcription judge for {language}.

Here are 5 different ASR transcriptions of the same audio:
1. [Transcript A]
2. [Transcript B]
3. [Transcript C]
4. [Transcript D]
5. [Transcript E]

Your task:
- Identify which transcript is most likely CORRECT
- Point out specific errors you notice in others (missing words, wrong words)
- You MUST select from the given options - DO NOT create new text
- If all seem wrong, indicate the most plausible one and flag confidence as LOW

Output:
- Selected: [number]
- Confidence: HIGH/MEDIUM/LOW
- Reasoning: [brief explanation]
```

### ❌ DON'T: Ask LLM to "fix" or "improve" transcriptions
This leads to hallucination - the LLM will generate plausible-sounding but incorrect text.

### Best LLMs for Indic Judgment:
1. **Gemini 2.0 Flash/Pro** - Excellent Indic understanding
2. **GPT-4o** - Good multilingual reasoning
3. **Llama 3.3 70B** - If you want local
4. **IndicBERT/MuRIL** - For perplexity scoring (not generation)

---

## Stage 5: Quality Scoring & Flagging

Compute a composite quality score:

```python
quality_score = (
    0.3 * model_agreement_score +      # How much do models agree?
    0.2 * avg_confidence_score +        # Model confidence scores
    0.2 * llm_judge_confidence +        # LLM's confidence in selection
    0.2 * language_model_perplexity +   # Does it read like real {language}?
    0.1 * audio_quality_score           # SNR, clarity metrics
)

if quality_score < threshold:
    flag_for_manual_review()
```

---

## Recommended Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    AUDIO SEGMENT INPUT                          │
└────…1943 chars truncated…─┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│          STAGE 4: LLM-AS-JUDGE (Disagreement Cases)             │
│                                                                 │
│   Present all candidates → LLM selects best → NO generation     │
│                                                                 │
│   Models: Gemini 2.0 Flash, GPT-4o, or Llama 3.3 70B           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│               STAGE 5: QUALITY SCORING                          │
│                                                                 │
│   Composite score → Flag low confidence → Manual review queue   │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 FINAL TRANSCRIPT + METADATA                     │
│  {transcript, language, confidence, model_agreement, flags}     │
└─────────────────────────────────────────────────────────────────┘
```

---

## Language-Specific Recommendations

| Language | Primary Model | Secondary | Notes |
|----------|--------------|-----------|-------|
| **Hindi** | IndicWhisper | Whisper V3 | Best covered, code-switching common |
| **Tamil** | IndicConformer | IndicWhisper | Strong AI4Bharat support |
| **Telugu** | IndicWhisper | MMS | Good coverage |
| **Kannada** | IndicConformer | MMS | Slightly less data |
| **Malayalam** | IndicWhisper | MMS | Complex morphology |
| **Bengali** | IndicWhisper | IndicConformer | Good coverage |
| **Marathi** | IndicWhisper | MMS | Good coverage |
| **Gujarati** | MMS | IndicWhisper | Less specialized data |
| **Punjabi** | MMS | Whisper V3 | Limited specialized models |
| **Odia** | MMS | IndicWav2Vec | Lower resource |
| **Assamese** | MMS | IndicWhisper | Lower resource |
| **English** | Whisper V3 | WhisperX | Best covered |

---

## Additional Clever Tricks

### 1. **Confidence-Weighted Character Voting**
Instead of word-level ROVER, do character-level for Indic scripts (handles morphology better):
```python
# For each position, vote on character level
# Especially useful for agglutinative languages like Tamil, Malayalam
```

### 2. **Cross-Entropy Filtering with IndicBERT/MuRIL**
```python
# Score each transcript's perplexity
# Transcripts with unnaturally high perplexity likely have errors
perplexity = compute_perplexity(transcript, muril_model)
if perplexity > threshold:
    flag_as_suspicious()
```

### 3. **Forced Alignment Post-Processing**
After consensus, run forced alignment (Montreal Forced Aligner or NeMo) to:
- Verify transcript matches audio duration
- Get word-level timestamps
- Catch gross errors (transcript way too long/short for audio)

### 4. **Code-Switching Detection**
Many Indian podcasts mix English. Use:
```python
# Detect language switches within transcript
# Run appropriate model for each segment
# Common: Hindi-English, Tamil-English, etc.
```

### 5. **Phonetic Similarity Fallback**
When models disagree, check if disagreements are phonetically similar:
```python
# "कर रहा" vs "कर रहे" - minor morphological difference, might be acceptable
# Use Soundex/Metaphone adapted for Indic
```

---

## Implementation Priority

If you want to start quickly and iterate:

**Phase 1 (Quick Win):**
1. IndicWhisper + Whisper V3 + MMS
2. Simple majority voting
3. Confidence thresholding

**Phase 2 (Better Quality):**
4. Add IndicConformer + SeamlessM4T
5. Implement ROVER
6. Add LLM-as-Judge for disagreements

**Phase 3 (Maximum Quality):**
7. Full ensemble with all models
8. Perplexity filtering with MuRIL
9. Forced alignment verification
10. Human-in-the-loop for flagged segments

---

Want me to write out the actual implementation code for any of these stages? I can create a complete Python pipeline with all the model integrations.


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 176
Output:
# Optimal strategy for high-quality Indian language transcription

**The lowest possible WER for Indian language audio comes from a four-stage pipeline: language identification with MMS-LID, multi-model transcription using IndicWhisper as the primary engine, ROVER-based ensemble combination, and confidence-gated LLM correction with hallucination safeguards.** This approach can achieve **10-35% WER** across most Indian languages—with Hindi and Kannada performing best (~10-18% WER) and Malayalam remaining most challenging (~32% WER). Ensemble methods alone reduce errors by **10-15% relative**, while properly constrained LLM post-correction adds another **10-20% relative improvement**.

The critical insight is that no single model dominates across all 12 target languages. IndicWhisper achieves the lowest WER on 39 of 59 Vistaar benchmark tests, but MMS excels on long-tail languages and SeamlessM4T handles noisy audio better. A strategic combination maximizes accuracy where quality is the only priority.

## The recommended four-stage pipeline architecture

The optimal architecture flows through distinct stages: preprocessing with VAD segmentation, language identification, parallel multi-model ASR with ensemble combination, and confidence-gated LLM verification.

**Stage 1: Preprocessing**
- Use **Silero VAD** for voice activity detection (trims ~35% silence, saves compute)
- Segment audio into **30-second chunks** (Whisper's optimal context window)
- Normalize to 16kHz mono audio
- Skip aggressive denoising—it often introduces artifacts that hurt transcription quality

**Stage 2: Language Identification**
- Primary: **MMS-LID** (facebook/mms-lid-256) for spoken audio detection across 4,017 languages
- Backup: **SpeechBrain VoxLingua107** for utterance embeddings
- Confidence threshold logic: >0.9 routes to language-specific model; 0.7-0.9 uses multilingual model; <0.7 flags for ensemble processing

**Stage 3: Multi-Model ASR with Ensemble Combination**
- Run **three models in parallel**: IndicWhisper (language-specific), Whisper large-v3, and MMS-1B
- Combine outputs using **ROVER with confidence-weighted voting** (NIST SCTK toolkit)
- Apply Indic text normalization with **IndicNLP Library** (preserves diacritics that Whisper's default normalizer strips)

**Stage 4: LLM-Based Correction with Hallucination Prevention**
- Only process utterances with sentence confidence <0.95 or any word confidence <0.7
- Use **IndicBART** or **Airavata** for correction with constrained prompts
- Apply edit distance check (reject changes >30% of original) and length ratio check (±15% tolerance)
- If checks fail, revert to original ASR output

## Best ASR models ranked by Indian language performance

**IndicWhisper** (AI4Bharat) represents the current state-of-the-art for most Indian languages. Fine-tuned on 10,700+ hours across 12 languages via the Vistaar-Train dataset, it achieves the lowest WER in 39 of 59 benchmarks with **4.1% average relative WER reduction** over competitors.

| Language | IndicWhisper WER | Notes |
|----------|------------------|-------|
| Hindi | **10.3-15.0%** | Best overall performance |
| Kannada | **13.2-22.2%** | Strong across datasets |
| Marathi | **11.4-22.8%** | Consistent performer |
| Bengali | **16.6-24.7%** | Good baseline |
| Punjabi | **16.9-23.1%** | Reliable results |
| Gujarati | **17.8-23.5%** | Solid performance |
| Tamil | **17.2-29.2%** | Higher variance |
| Telugu | **25.0-33.8%** | Challenging language |
| Odia | **15.0-35.2%** | Wide variance by dataset |
| Malayalam | **21.4-44.5%** | Most difficult—limited training data |

**Meta's Omnilingual ASR** (November 2025, 7B parameters) offers the widest coverage at 1,600+ languages with **78% achieving CER <10%**. It includes long-tail Indian languages like Kui, Chattisgarhi, Maithili, and Bagheli that no other model covers. However, its recency means limited independent validation.

**IndicConformer** (AI4Bharat) provides the widest Indian language coverage with all 22 constitutionally scheduled languages supported. The 600M multilingual model uses a hybrid CTC-RNNT architecture and works efficiently via the NeMo framework.

**SeamlessM4T v2** halves WER relative to Whisper Large-v2 on FLEURS benchmarks and handles background noise exceptionally well. Its limitation is the CC-BY-NC license restricting commercial use.

**Sarvam Saarika** specializes in real-world code-mixed speech—critical for YouTube podcast content where Hindi-English mixing is common. While lacking public benchmarks, it's purpose-built for Indian colloquial speech patterns.

## Ensemble strategies that measurably reduce WER

ROVER (Recognizer Output Voting Error Reduction) provides **10-15% relative WER reduction** when combining diverse models. The algorithm works in two stages: first aligning outputs via dynamic programming into a Word Transition Network, then applying confidence-weighted voting at each branch point.

**Implementation with NIST SCTK:**
`…1993 chars truncated… originals
3. Do NOT add, remove, or restructure content
4. If uncertain, keep original unchanged

Transcript: [ASR_OUTPUT]
Low-confidence words: [WORD_LIST]
Corrected:
```

**Hallucination detection safeguards:**
- **Edit distance ratio <0.3**: Reject corrections changing >30% of characters
- **Length ratio ±15%**: Flag outputs differing significantly in length
- **Chain-of-thought verification**: Require the LLM to explain each change, verify explanations are phonetically plausible

**Confidence-based filtering** dramatically reduces unnecessary LLM calls. Sending only utterances with **sentence confidence <0.95 or word confidence <0.7** to LLM correction reduces processing by ~77% while capturing genuinely problematic segments.

## Language identification for unlabeled segments

**MMS-LID** (facebook/mms-lid-256 or mms-lid-4017) is the clear winner for spoken language identification, covering 4,017 languages including all target Indian languages. It works directly on audio using a Wav2Vec2-based classifier.

**Critical insight**: Whisper's built-in language detection is unreliable for Indian languages, particularly:
- **Hindi-Marathi confusion**: Similar Devanagari phonetics
- **Bengali-Assamese confusion**: Related scripts and sounds
- **Short utterances (<3 seconds)**: Insufficient signal
- **Code-mixed content**: Switches mid-sentence

**Recommended detection flow:**
1. Run MMS-LID on audio segment
2. If confidence >0.9: Route to language-specific IndicWhisper model
3. If confidence 0.7-0.9: Use multilingual IndicConformer
4. If confidence <0.7: Run through multiple models, use ensemble combination

**For code-switching detection**, current state-of-the-art achieves ~32% WER on MUCS Hindi-English benchmarks. The best approach is end-to-end multilingual models rather than cascaded LID→ASR pipelines that fail at switch points.

## Handling edge cases in podcast content

**Code-switching** (Hindi-English, Tamil-English) requires special handling since YouTube podcasts frequently mix languages. **Sarvam ASR-Translate** converts code-mixed Indic speech directly to English text with topical context awareness. Alternatively, train or fine-tune multilingual models on code-switched data from MUCS challenge datasets.

**Accented regional speech** varies significantly—the Svarah benchmark covers 117 speakers from 19 Indian states. For production robustness:
- Use multi-condition training (diverse accents in training data)
- Fine-tune on regional accent samples when available
- The IndicVoices dataset from AI4Bharat provides diverse speaker coverage

**Handling low-quality segments** in cleaned YouTube audio:
- Use SNR estimation to route audio appropriately
- For noisy segments: Prefer noise-robust models (SeamlessM4T excels here)
- Apply light preprocessing rather than aggressive denoising
- Flag segments with audio quality scores below threshold for manual review

## Practical implementation recommendations

**Model hosting stack:**
```
Primary ASR: IndicWhisper (ai4bharat/indicwhisper-{lang})
Secondary ASR: Whisper large-v3 (openai/whisper-large-v3)
Tertiary ASR: MMS-1B (facebook/mms-1b-all)
Language ID: MMS-LID (facebook/mms-lid-256)
Text Normalization: IndicNLP Library
Ensemble: NIST SCTK ROVER
LLM Correction: IndicBART or Airavata
```

**Quality scoring without ground truth:**
- Multi-model agreement score (0-1 based on Jaccard similarity between outputs)
- Language model perplexity (flag outliers >2σ from mean)
- Repetition detection (Whisper hallucination pattern: excessive phrase repetition)
- Audio-text alignment confidence from forced alignment

**Confidence thresholds for routing:**
| Score | Action |
|-------|--------|
| >0.95 | Auto-accept without LLM pass |
| 0.80-0.95 | Standard accept, sample for QA |
| 0.60-0.80 | Send to LLM correction |
| <0.60 | Manual review or re-process with ensemble |

## Conclusion

The optimal strategy combines **IndicWhisper as the primary model** (lowest WER on most benchmarks), **MMS-LID for language identification** (4,000+ language coverage), **ROVER ensemble combination** (10-15% relative improvement), and **confidence-gated LLM correction with IndicBART/Airavata** (additional 10-20% improvement with hallucination safeguards).

For the specific use case of YouTube podcast content with code-switching, **Sarvam Saarika** deserves serious consideration despite lacking public benchmarks—it's purpose-built for real-world Indian speech patterns. The key architectural principle is routing: use language confidence to select between language-specific fine-tuned models (high confidence) and multilingual models (lower confidence or code-mixed content).

With compute constraints removed, the maximum quality approach runs **three parallel ASR models, combines via ROVER, then applies selective LLM correction only where confidence is low**—achieving the best possible accuracy while preventing hallucination through constrained prompting and edit distance verification.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Since you have **no compute constraints** and are operating **offline**, you have the luxury of trading time for accuracy. The "cleverest" way to minimize WER (Word Error Rate) for a training dataset is not to rely on a single model, but to build a **Consensus-Based Mixture-of-Experts (MoE) Pipeline** with a **Discriminative Verification Loop**.

This approach treats transcription as a "voting" problem among diverse architectures, using an Indic-specialized LLM as the judge.

Here is the **Zero-Compromise Indic Transcription Strategy**:

### Phase 1: The "Forensic" Language Identification (LID)

You cannot trust metadata. If you feed Hindi audio to a Marathi model, you get "hallucinations" (garbage text that looks real).

* **The Strategy:** Run a **Triple-Check Ensemble** on every file.
1. **Specialist:** **AI4Bharat IndicLID** (Specifically trained to distinguish 22 Indian languages; handles code-mixing better than Western models).
2. **Generalist:** **Whisper Large-v3** (Internal language detection).
3. **Acoustic:** **SpeechBrain (VoxLingua107)** or **Silero**.

* **The Rule:** Only proceed if **2 out of 3** models agree on the language. If they disagree (e.g., confusion between Bengali/Assamese or Hindi/Urdu), flag the file for manual review. **Do not pollute your training set.**

### Phase 2: The ASR "Council of Experts"

Run every segment through **three** distinct SOTA models. We choose these because they have *different* architectural biases, meaning they rarely make the *same* mistake.

1. **Expert A: OpenAI Whisper Large v3**
* **Role:** The Robust Generalist.
* **Why:** Unbeatable for background noise, music, and "Code-Mixing" (English words mixed with Indic languages).

2. **Expert B: AI4Bharat IndicConformer**
* **Role:** The Native Specialist.
* **Why:** Unlike Whisper (Transformer), this uses a Hybrid CTC/RNN-T architecture. It is the current SOTA for pure Indic phonetics and properly captures Indian proper nouns (names, cities) that Whisper often Anglicizes.

3. **Expert C: NVIDIA Canary-1B** (or Google USM if available)
* **Role:** The Third Opinion.
* **Why:** Canary is a "FastConformer" model that often catches short, rapid speech that Whisper misses.
* *Note on "Voxtral":* There is no dominant model named "Voxtral" currently. You likely meant **Voctro** (TTS) or are thinking of a Mistral-based audio model. I recommend replacing this slot with **NVIDIA Canary** or **SeamlessM4T v2** for the best multimodal performance.

**Crucial Step:** Do not just save the text. Extract the **N-Best Hypotheses** (e.g., top 5 options) from each model.

### Phase 3: The "Clever" Judge (LLM Rescoring)

This is where you beat standard ASR. Instead of asking an LLM to *generate* text (which risks hallucination), use it to *score* the hypotheses.

* **The Model:** **Sarvam-1 (2B)**.
* **Why:** Sarvam-1 is trained on 2 trillion Indic tokens. It has a significantly better internal probability model for Hindi/Tamil/Telugu grammar than GPT-4 or Llama-3.

* **The Method (Perplexity Ranking):**
1. Take the transcripts from Whisper, IndicConformer, and Canary.
2. Feed them into **Sarvam-1** and calculate the **Perplexity (PPL)** score for each sentence.
3. **Select the transcript with the lowest Perplexity.**

* *Logic:* A native speaker (the LLM) will find the correct transcript "more natural" (lower perplexity) than a transcript with phonetic errors.

### Phase 4: The "Hallucination Killer" (Forced Alignment)

Since you are training ASR/TTS models, **alignment is non-negotiable**.

* **Tool:** **Montreal Forced Aligner (MFA)** or **NeMo Forced Aligner**.
* **Action:** Align the final selected text back to the audio.
* **The Filter:**
* If the aligner cannot find a specific word in the audio (low confidence score), **drop that word**.
* If the alignment fails for >20% of the segment, **discard the entire segment**.
* *Result:* This mathematically guarantees that your training data contains no "Whisper Hallucinations" (like "Thank you for watching" on silent audio).

### Summary Pipeline

| Component | Tool/Model | Purpose |
| --- | --- | --- |
| **1. ID** | **IndicLID + Whisper + Silero** | Ensure strict language purity. |
| **2. Listen** | **Whisper V3 + IndicConformer + Canary** | Get 3 diverse "opinions". |
| **3. Judge** | **Sarvam-1 (2B)** | Use PPL scoring to pick the most grammatically valid option. |
| **4. Verify** | **Forced Alignment (MFA)** | Mathematical proof that text matches audio. |

**Why this is the "Best" Strategy:**
You are effectively building a **Generative Adversarial Network (GAN)** logic manually. The ASR models are the "Generators," and Sarvam-1 + Forced Aligner are the "Discriminators." By running this offline with infinite compute, you ensure that only the "truth" survives to your training set.


**Updating a project file**

The user hasn't asked for code implementation directly, but it might be useful to include a code skeleton for the pipeline. Their request mentions a task related to "maya-transcribe," and it seems they want to update the Agents.md file based on the indicated instructions. While we don't necessarily need to write code, I should ensure that Agents.md is updated with relevant details, including a proposed blueprint for a pipeline, data schema, and job orchestration options.


**Navigating the "No invention" rule**

I need to be cautious about the "No invention" rule, which requires me to back recommendations with solid references or clearly state when I'm making inferences. While I can use my knowledge, I have to mark anything that's not in the references as inference. It's important to cross-check claims and justify my decisions. I shouldn't focus on reading the .env file; instead, I’ll clarify key task details like languages, segment lengths, compute resources, and quality targets, even though the WER target isn't explicitly stated.


**Designing a high-confidence segmentation system**

I'm thinking of a precision-first gating approach, where only segments with high confidence are accepted, and rejects are set aside for human review. I need to specify quality thresholds and evaluate metrics, like alignment scores. I'll propose language identification and code-switching detection, along with an ASR ensemble for Indic languages using different models like AI4Bharat IndicWhisper and generalist Whisper. I'll also consider using forced alignment for verification and emphasize the importance of audio-grounded gating while ensuring human involvement.


**Drafting an implementation blueprint**

I'm considering a data schema using JSONL or Parquet, and I need to determine how to orchestrate the pipeline—maybe with tools like Hydra, Prefect, Ray, or Celery. While references mention orchestrations and data structures, I’ll propose a validation protocol that includes building a gold set and computing WER/CER per language using active learning. The repo may lack code, but I can suggest a skeleton Python package. However, I think focusing on updating Agents.md is the priority, especially since there’s no README available.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
./task.md:1:suppose i have scraped youtube videos and have cleaned with a robust cleaning pipeline and now have single speaker segments of audio files that are suitable for TTS, ASR or speech systems. Now in-order for me to train them, i need to transcribe these audio files. Assuming i have audio files of sizes from 1s atleast to like 30s, 50s and 60s max, how can i accurately create transcriptions for these. compute is not a constraint here. quality is. so, my youtube videos are majorly
./task.md:2:
./task.md:3:
./task.md:4:
./task.md:5:
./task.md:6:
./task.md:7:Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali , Odia, Marathi, Assamese, English
./task.md:8:
./task.md:9:the unknown ones are not flagged, i might try to get more metadata for them and try flagging them as per the title and description for language. but i was wondering if we have a way to identify the language as well in our pipeline. Now, assuming the situation here, for indian space and languages mentioned above, and most of these are high quality youtube podcasts and or are explanation videos that has minimal to low and are cleaned out of music. Given the situation and since we are anyway generating the transcripts offline, like it doesnt have to be real-time or latency, or quick, given no compute constraints, what's like the best way or the cleverest way possible to generate transcripts to these segments so that i get WER as minimal as possible. i can pass these segments to as many models as possible to compare them all and or get them checked by a superiour LLMs or are there any LLM specialized in indic, where i can maybe run these segments and they create transcripts like A/B testing, what's that cleverest way. when i said i have no constraints, i literally meant we can run any specialized indic llms that can help create transcriptions with prompt guidence like gemma, or anything else, but we need to strategically plan this to avoid hallucinations or creation of content rather transcribing the exiting ones. that doesnt mean we can't use LLM, we need to be clever here. what's that combination we can use like voxtral+whisper large ? or whisper + indic conformers or indicwhisper or what's that best combo we can use to get WER as minimal as possible, we just need to be smart enough and clever here since we are doing this offline and or have no constraints over compute or anything and given we have enough time and everything, we just can't compromise on quality or WER for these transcriptions, what's the go-to strategy that we can adapt to where we just can't afford to make mistakes and have high quality transcriptions to these audio segments so we can train the best in-class indic ASR and TTS models.
./Agents.md:1:# Agents.md — Meta-Prompt + Workflow for “Reference Ensemble” Research (ASR Transcription Quality First)


**Updating project plans**

I’m focusing on updating the Agents.md as the main deliverable. I see the need to check for any instructions regarding skill usage, but it seems no skills were triggered. My next steps include applying a patch and planning multiple tasks. It feels a bit complex, but I think I can manage it! I’ll create a detailed plan with specific steps to ensure everything is clear and organized. Let's use the update_plan tool for that.


Explanation: Read the task and references, then synthesize a single precision-first transcription pipeline and write it into Agents.md per the repo workflow.


Output: Plan updated


wc -l docs/Reference1.md docs/Reference2.md docs/Reference3.md docs/Reference4.md docs/Reference5.md docs/Reference6.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
  222 docs/Reference1.md
  117 docs/Reference2.md
   93 docs/Reference3.md
  275 docs/Reference4.md
  175 docs/Reference5.md
   82 docs/Reference6.md
  964 total


**Updating Agents.md**

I’ve read most of the files, especially files 1, 4, and 5, and for file 2, I reviewed the lines too. I confirmed that the truncation was internal, so I can confidently proceed. Now, it’s time to update Agents.md with filled sections, but I need to be careful. I’ll keep the top meta prompt unchanged, filling in the necessary content for each step and ensuring that decisions are labeled as Adopt, Adapt, Reject, or Park for clarity.


**Implementing Recommendations**

The phrase "No invention" implies that I should cite references when possible, but Agents.md doesn't necessitate citations. However, I still want to indicate which references support my recommendations, like mentioning "Supported by Ref1/Ref5." I must ensure that I don't claim something unless there's support or it’s clearly labeled as inference. For each recommendation, I’ll note references clearly, avoiding specific metrics unless they’re backed. Also, I need to set quality gating thresholds based on references and propose that we use outputs from both CTC and RNNT from IndicConformer.


**Proposing Alignment and QC Strategies**

I need to suggest forced alignment (MFA) while being aware of lexicon issues in multiple languages. Maybe CTC alignment scoring could help with audio-grounded verification. Also, there’s code-switching mentioned in references, which we can incorporate. I'll propose pre-QC processes like VAD trimming and SNR/clipping, possibly using Silero VAD and WavLM. Even though the user has single speaker segments, running VAD will help trim silence. For language identification, I’ll consider MMS LID and look into additional models like SpeechBrain VoxLingua107. We’ll adopt a solid primary model with a fallback, prioritizing quality over coverage.


**Addressing Segment Handling and LID**

I need to note that segments can be as short as 1 second, which might make LID unreliable. To address this, we could consider running hierarchical LID, aggregating adjacent segments from the same source, or using metadata. Since the user is cleaning segments from YouTube, they may have video-level metadata to help gauge language distribution, which we can use as a prior for segment-level LID. I should label this as an inference. Additionally, I’ll propose storing all hypotheses and scores in a structured implementation blueprint with suggestions for dataset structure and compute scheduling.


**Developing the Master Plan**

I’m considering including "faster-whisper" for efficiency, though it's not strictly necessary. The master plan should have at least two diverse ASR systems for each language group. I’ll consider systems like IndicConformer, Whisper, and MMS, with others like Voxtral for English. I need to be cautious about licensing, especially regarding SeamlessM4T. We will normalize scripts for Indic languages, choosing either native scripts or Romanization for TTS training. For quality scoring, using CTC-based scoring seems essential. Quality targets will aim for low WER, targeting below 5%, and I'll structure the content in Agents.md, ensuring thoroughness without excessive verbosity.


**Structuring Segment Processing**

I’ll integrate multiple models for our approach: running a specialized model like IndicWhisper or IndicConformer, along with a generalist like Whisper large-v3 and a diversity model like MMS. This adds redundancy but increases diversity. I need to ensure the output script is handled correctly, likely using a native script for IndicWhisper. For verification, I'll utilize CTC alignment scoring and choose appropriate models, ensuring independence to avoid self-confirmation issues. I’ll establish verification methods including agreement scores and acoustic alignment. The implementation will include a detailed blueprint and organization of data and processes, ensuring thorough updates to Agents.md.


**Planning Patch for Agents.md**

Before making changes, I'll open Agents.md to review and patch it. The file has placeholders that I'll replace with full content. I want to make sure to keep the meta prompt intact and update the global checklist as I complete tasks. In Step 1, I’ll outline the transcription goals, languages, segment lengths, compute constraints, quality targets, acceptance philosophies, required tools, and banned tools. For each reference, I'll summarize key suggestions and decide what's applicable, deciding whether to adopt, adapt, or reject certain elements.


**Crafting Unified Master Plan**

I’m reviewing Reference 6 for suggestions and planning a unified master pipeline. I'll ensure everything on Language Identification (LID), ASR, and verification is methodically structured.

For LID, I’ll use a triple-check system with agreement from two out of three models as a basis to proceed. The ASR panel will extract hypotheses, while a judgment algorithm will rank options by perplexity. I’ll verify outputs through forced alignment and implement a human review process for any rejected segments. I’ll also outline storage, improvement, and development phases to streamline our workflow.


**Updating Agents.md Patch**

I'm set to update Agents.md by replacing placeholders and adding our filled content. Under "Step 1 — Task Understanding," I'll summarize the goals—offline transcription targeting minimal WER—and the constraints, such as segments being cleaned and of certain lengths. I’ll also emphasize non-negotiables like ensuring audio-grounded verification.

Next, I need to focus on success metrics and assumptions, being mindful of what data to include and ensuring the prompt's integrity remains intact. Finally, I’ll structure everything with proper markdown and bullet points. Let's be thorough yet concise!


**Crafting Implementation Blueprint**

I'm working on the implementation blueprint, which will detail the recommended tech stack and necessary modules. I also need to ensure a clear validation protocol with steps laid out. It's crucial to have a bullet checklist for the "Do This Next" section to keep everything organized. I’ll be updating Agents.md by beginning with "Step 1 — Task Understanding" and making sure to tick items off the global checklist. I want to be thorough yet concise, avoiding unnecessary length in the patch!