Instruction:
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <approval_policy>on-request</approval_policy>
  <sandbox_mode>workspace-write</sandbox_mode>
  <network_access>restricted</network_access>
  <shell>bash</shell>
</environment_context>


<environment_context>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>


# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
Start implementing this plan.

Task - [task.md](task.md)
Reference docs by other AI agents - /home/ubuntu/maya-transcribe/docs
secrets - /home/ubuntu/maya-transcribe/.env
Master plan for you to follow [master.md](master.md)
ignore the agents.md as it was intended for another agent and finished it's task and has created the master plan.

First create a robust version with full performance mode and we can always perform optimisations and see where we can compromise and where we cant. if you want to test your pipleine, few options we have are all secrets are in .env in root. and all metadata for each videoID is in supabase and R2 has cleaned, segmented audio files in the bucket named `1-cleaned-data` sometimes it might have some prefixes as well, so if you couldnt find it, search recursively or any other simpler approach or whatever. when you unzip the tar, you will find a folder named segments which contains all audio files. Architect > implement > test > report WERs & end to end latencies and estimations.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Organizing file tasks**

I'm planning to use `multi_tool_use.parallel` to read relevant files like master.md, task.md, and possibly Prompt.md all at once. It seems efficient to read them concurrently! I should also inspect the documentation and check the repository structure to make sure I'm covering all bases. It feels like a good strategy, but I want to ensure I'm not missing anything important in my approach before moving forward.


rg -n "plan" -S master.md task.md Prompt.md || true


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 80
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 13 04:09 .
drwxr-x--- 43 ubuntu ubuntu  4096 Jan 13 04:01 ..
-rw-rw-r--  1 ubuntu ubuntu 41725 Jan 13 04:08 Agents.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 03:12 docs
-rw-rw-r--  1 ubuntu ubuntu  3652 Jan 13 02:35 .env
-rw-rw-r--  1 ubuntu ubuntu  9299 Jan 13 04:09 master.md
-rw-rw-r--  1 ubuntu ubuntu  2152 Jan 13 04:12 Prompt.md
-rw-rw-r--  1 ubuntu ubuntu  2627 Jan 13 02:35 task.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# master.md — Offline “Quality-First” Transcription Master Plan (Indic + English)

## Objective (from `task.md`)
Generate **near-zero-error** transcripts for cleaned, single-speaker YouTube segments (≈1–60s) across:
Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (+ unknowns).
**Compute/latency are not constraints; quality/WER is.**

This document is the **execution-ready summary**. Full per-reference analysis and detailed decisions live in `Agents.md`.

---

## Core idea
To minimize WER offline, don’t bet on one model. Instead:
1) **Route correctly (spoken LID)**
2) **Generate diverse hypotheses (ASR ensemble + N-best)**
3) **Combine (ROVER/confusion network)**
4) **Prove audio-text match (CTC/forced-alignment verification gate)**
5) **Reject uncertainty + human-correct the tail**

---

## Pipeline block diagram
```text
┌───────────────────────────────┐
│          Audio segment         │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│        Pre-QC + VAD trim       │
│   (format normalize + sanity)  │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│        Spoken LID ensemble     │
│  MMS-LID (+ optional 2nd vote) │
└───────────────┬───────────────┘
                │
      ┌─────────┴──────────┐
      │                    │
      v                    v
┌───────────────────────┐  ┌────────────────────────┐
│ Confident language     │  │ Uncertain / mixed       │
│ route                  │  │ route                   │
└───────────┬───────────┘  └───────────┬────────────┘
            │                          │
            v                          v
┌───────────────────────┐  ┌────────────────────────┐
│ ASR ensemble (per-L)   │  │ ASR ensemble (multi)    │
│ IndicWhisper           │  │ Whisper v3              │
│ + Whisper large-v3     │  │ + IndicConformer        │
│ + IndicConformer       │  │ + MMS (+ Gemma 3n opt.)  │
│ + Gemma 3n(probably)   │  │                        │
└───────────┬───────────┘  └───────────┬────────────┘
            │                          │
            └───────────┬──────────────┘
                        │
                        v
┌───────────────────────────────┐
│     Normalize all hypotheses   │
│ (Unicode + IndicNLP + numbers) │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│  System combination (ROVER)    │
│   + disagreement map (audit)   │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│   Audio-grounded verification  │
│  CTC “fit” scoring (+ aligner) │
└───────┬───────────────┬───────┘
        │ PASS          │ FAIL
        v               v
┌───────────────────┐   ┌──────────────────────────────┐
│ ACCEPT            │   │ Re-decode / add model / retry │
│ transcript+scores │   │ (then repeat from ASR stage)  │
└───────────────────┘   └───────────────┬──────────────┘
                                        │ still FAIL after retries
                                        v
                              ┌──────────────────────────┐
                              │ HUMAN REVIEW (tail only)  │
                              │ correct → ACCEPT          │
                              │ discard → DROP            │
                              └──────────────────────────┘
```

---

## Initial model stack (validate on a gold set; tune per language)
**Spoken LID**
- Primary: **MMS-LID** (spoken LID coverage is the key requirement)
- Optional 2nd opinion: **SpeechBrain VoxLingua107 ECAPA** (and/or Whisper LID as tie-break)

**ASR (Indic)**
- **IndicWhisper** (Indic fine-tuned Whisper; “specialist”)
- **Whisper large-v3** (robust multilingual “generalist”)
- **IndicConformer-600M multilingual** (different architecture; error diversity)
- Optional diversity: **MMS ASR** (especially when others disagree / lower-resource languages)

**ASR (English)**
- **Whisper large-v3** + **Voxtral** (treat Voxtral as “measure on your gold set” for your domains)

**System combination**
- **ROVER / confusion-network voting** (keep the disagreement map)

**Audio-grounded verification**
- Primary: **CTC “fit” scoring** (score candidates against audio; reject low-fit)
- Optional: forced alignment where you can support it cleanly

---

## Why this approach is the best way to minimize WER offline (and avoid hallucinations)
This is the strongest generally applicable strategy under “unlimited compute / offline” because it attacks the dominant error sources:
- **Wrong routing is catastrophic:** Spoken LID prevents wrong-language decodes (script/vocabulary mismatch) that can silently poison training data.
- **Ensembles lower error by diversity:** Different models/architectures make different mistakes; if you can afford it, generating multiple hypotheses increases the odds that at least one is correct.
- **ROVER reduces WER when errors differ:** System combination exploits that diversity instead of “picking a winner” blindly.
- **Audio-grounded verification is the key safety gate:** A transcript can look fluent but still be wrong; independent alignment/CTC scoring forces the text to match the audio and catches hallucination patterns.
- **Precision-first acceptance yields near-zero-error training text:** If a segment can’t be verified confidently, rejecting it (and sending only the tail to humans) minimizes label noise, which matters more than maximizing coverage for training.

Important: “minimal WER” is not a claim about global benchmarks; it is achieved **for your data** by tuning gates/weights on a **human gold set** and only accepting segments that pass strict audio-grounded checks.

---

## Execution plan (deliverables)
1. **Gold set + evaluation harness (Week 1)**
   - Build a stratified gold set (all languages + duration buckets + clean/mild noise + code-switch).
   - Produce human transcripts; compute WER/CER per language; establish baselines for each model.
2. **Baseline pipeline (Week 1–2)**
   - Implement: `VAD/QC → MMS-LID → ASR ensemble → normalization → ROVER → CTC-fit verification → accept/reject`.
   - Store full audit trail (all hypotheses + scores).
3. **Calibration (Week 2)**
   - Tune: LID thresholds, per-language ROVER weights, verification-gate thresholds.
   - Goal: maximize **precision** of accepted set; accept-rate is a secondary knob.
4. **Human reject loop (Week 2–3)**
   - Create a review queue that shows audio + top hypotheses + disagreement spans for fast correction.
   - Feed corrected rejects back into calibration (and optional later fine-tuning).
5. **Scale-out + monitoring (Week 3+)**
   - Batch by language; parallelize per-model decoding; track reject reasons and residual error via sampling.

---

## Output schema (minimum needed for auditability)
For each `segment_id`, store:
- audio QC metrics, LID probs (per LID model)
- ASR hypotheses (per model + decode params + N-best if available)
- normalized hypotheses
- ROVER output + disagreement map
- verification scores (CTC/aligner) + decision (`accept|reject|human`) + reason codes

---

## LLM usage policy (optional; safe mode only)
If you use an LLM at all:
- It must **select among existing hypotheses/spans** only (no free-form “fixing”).
- Its output must still pass the same **audio-grounded verification gate** before acceptance.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
suppose i have scraped youtube videos and have cleaned with a robust cleaning pipeline and now have single speaker segments of audio files that are suitable for TTS, ASR or speech systems. Now in-order for me to train them, i need to transcribe these audio files. Assuming i have audio files of sizes from 1s atleast to like 30s, 50s and 60s max, how can i accurately create transcriptions for these. compute is not a constraint here. quality is. so, my youtube videos are majorly

Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali , Odia, Marathi, Assamese, English

the unknown ones are not flagged, i might try to get more metadata for them and try flagging them as per the title and description for language. but i was wondering if we have a way to identify the language as well in our pipeline. Now, assuming the situation here, for indian space and languages mentioned above, and most of these are high quality youtube podcasts and or are explanation videos that has minimal to low and are cleaned out of music. Given the situation and since we are anyway generating the transcripts offline, like it doesnt have to be real-time or latency, or quick, given no compute constraints, what's like the best way or the cleverest way possible to generate transcripts to these segments so that i get WER as minimal as possible. i can pass these segments to as many models as possible to compare them all and or get them checked by a superiour LLMs or are there any LLM specialized in indic, where i can maybe run these segments and they create transcripts like A/B testing, what's that cleverest way. when i said i have no constraints, i literally meant we can run any specialized indic llms that can help create transcriptions with prompt guidence like gemma, or anything else, but we need to strategically plan this to avoid hallucinations or creation of content rather transcribing the exiting ones. that doesnt mean we can't use LLM, we need to be clever here. what's that combination we can use like voxtral+whisper large ? or whisper + indic conformers or indicwhisper or what's that best combo we can use to get WER as minimal as possible, we just need to be smart enough and clever here since we are doing this offline and or have no constraints over compute or anything and given we have enough time and everything, we just can't compromise on quality or WER for these transcriptions, what's the go-to strategy that we can adapt to where we just can't afford to make mistakes and have high quality transcriptions to these audio segments so we can train the best in-class indic ASR and TTS models.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 84
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 13 03:12 .
drwxrwxr-x 3 ubuntu ubuntu  4096 Jan 13 04:09 ..
-rw-rw-r-- 1 ubuntu ubuntu 11196 Jan 13 03:09 Reference1.md
-rw-rw-r-- 1 ubuntu ubuntu 18053 Jan 13 03:09 Reference2.md
-rw-rw-r-- 1 ubuntu ubuntu  5123 Jan 13 03:09 Reference3.md
-rw-rw-r-- 1 ubuntu ubuntu 13601 Jan 13 03:09 Reference4.md
-rw-rw-r-- 1 ubuntu ubuntu 11999 Jan 13 03:10 Reference5.md
-rw-rw-r-- 1 ubuntu ubuntu  4816 Jan 13 03:10 Reference6.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Prompt.md:31:Do not add explanations notes or any extra English text beyond the labels`
master.md:1:# master.md — Offline “Quality-First” Transcription Master Plan (Indic + English)
master.md:128:## Execution plan (deliverables)
task.md:9:the unknown ones are not flagged, i might try to get more metadata for them and try flagging them as per the title and description for language. but i was wondering if we have a way to identify the language as well in our pipeline. Now, assuming the situation here, for indian space and languages mentioned above, and most of these are high quality youtube podcasts and or are explanation videos that has minimal to low and are cleaned out of music. Given the situation and since we are anyway generating the transcripts offline, like it doesnt have to be real-time or latency, or quick, given no compute constraints, what's like the best way or the cleverest way possible to generate transcripts to these segments so that i get WER as minimal as possible. i can pass these segments to as many models as possible to compare them all and or get them checked by a superiour LLMs or are there any LLM specialized in indic, where i can maybe run these segments and they create transcripts like A/B testing, what's that cleverest way. when i said i have no constraints, i literally meant we can run any specialized indic llms that can help create transcriptions with prompt guidence like gemma, or anything else, but we need to strategically plan this to avoid hallucinations or creation of content rather transcribing the exiting ones. that doesnt mean we can't use LLM, we need to be clever here. what's that combination we can use like voxtral+whisper large ? or whisper + indic conformers or indicwhisper or what's that best combo we can use to get WER as minimal as possible, we just need to be smart enough and clever here since we are doing this offline and or have no constraints over compute or anything and given we have enough time and everything, we just can't compromise on quality or WER for these transcriptions, what's the go-to strategy that we can adapt to where we just can't afford to make mistakes and have high quality transcriptions to these audio segments so we can train the best in-class indic ASR and TTS models.


Summary: auto


**Implementing a robust version**

I’m thinking that master.md serves as a conceptual plan. Now, I need to move towards implementing a more robust version that includes a performance mode. First, I should check the repository content, since it might currently be empty apart from documentation and markdown files. I guess it would be helpful to list the contents recursively to see what's there and what needs to be addressed moving forward. Let's get started on that!


find . -maxdepth 3 -type f -not -path './.git/*' -print


Output:
Exit code: 0
Wall time: 0 seconds
Output:
./task.md
./docs/Reference2.md
./docs/Reference6.md
./docs/Reference4.md
./docs/Reference5.md
./docs/Reference3.md
./docs/Reference1.md
./Agents.md
./master.md
./Prompt.md
./.env


**Creating a codebase**

I haven’t created any code yet, so I’ll need to start building a codebase for the pipeline. I'm thinking Python might be the best option here, but I want to make sure it fits the guidelines. I should open the Prompt.md file to review the guidelines for implementing this pipeline properly. It's important to ensure everything aligns with the expectations before I jump into coding. Let's get to it!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
This is a prompt for any LLM based transcription tasks. use this when using Gemma3n or voxtral or any other LLM based actions. Feel free to adjust this if it doesnt make sense. but keep it on the same tone or restrictive tone to avoid hallucinations or content creation rather transcription.

`prompt`

`Instruction: You are a strict, verbatim transcription engine for Indian languages
Input: Audio file attached
Primary audio language: <language>
Task:
Listen to the audio carefully
Identify the primary spoken language as specified above
Transcribe the speech exactly as spoken and produce  outputs as defined below
Sometimes you won't be clear about complete transcription but you know parts of it. From that you can fix the grammar. That's how you can create the best transcription.
You know the first word and the last word. You can create the best transcription. You know the middle word and the last word. You can create the best transcription.
You need to think in terms of what you know and how you can go to the end with 100% accuracy with predicting what comes here, what would be it, what is the right fit and attaching it all together and doing the combinations

STRICT GLOBAL RULES (apply to all outputs):
Verbatim only Include all repetitions filler words stammers false starts hesitations and colloquial expressions exactly as spoken
No normalization Do not correct grammar do not fix pronunciation do not standardize spellings and do not clean up dialect accent or mixed language usage
No inference Do not add meaning structure or emphasis that is not clearly audible
Script fidelity Follow the rules of each output strictly Do not translate paraphrase or summarize
OUTPUT 1: Proper native transcription
Write everything only in the native script of the primary language
Do not use any punctuation marks at all
Even if English words are spoken write them phonetically using the native script of the primary language
Preserve repetitions fillers and stammers exactly

OUTPUT FORMAT RULES:
Clearly label the four outputs as
Output 1: Proper native transcription
Do not add explanations notes or any extra English text beyond the labels`


**Assessing implementation strategy**

I'm thinking about how the project's focused on LLM transcription. However, we might want to consider integrating with other options like Voxtral or Gemma. The user wants a robust version, prioritizing a full performance mode since we might optimize later. There's this unlimited compute environment to utilize, though our container may lack GPUs. It could be useful to design a modular pipeline with placeholders, especially since the user specified to ignore Agents.md and follow master.md.


**Planning robust implementation**

I need to create a robust version with a performance mode, likely implementing it in Python with a CLI. I should integrate with Supabase and R2, and I want to be careful when inspecting the secrets in .env to avoid revealing anything. In our environment, we can read it for integration. We also need to compute Word Error Rate (WER) and end-to-end latencies, so I’ll set up an evaluation harness to compare with gold transcripts stored in Supabase or locally. I'll check the docs for more details on the models and pipeline.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
## Target: near-zero transcript errors (offline, unlimited compute)

The highest-quality approach is **(1) route by language**, **(2) run a *diverse* ASR ensemble**, **(3) pick/merge outputs via *system combination***, and **(4) independently *verify against the audio* with forced-alignment / score-based gating**. Anything that doesn’t clear the gate goes to **human correction** (small %), not into training.

This beats “one giant model” because you get: **error diversity + consensus + audio-grounded verification** (and you can set the gate as strict as you want).

---

## 1) Spoken Language ID (LID) for routing

Do LID on every segment (even if you also have metadata). Use an audio LID model that covers Indic + unknowns:

* **Meta MMS LID (up to 4017 languages)** is designed for spoken LID at scale. ([Hugging Face][1])
* Optionally ensemble it with **SpeechBrain VoxLingua107 ECAPA LID** (strong on YouTube-like audio). ([Hugging Face][2])

**Practical routing rule:**

* If top-1 language prob ≥ 0.90 and margin(top1-top2) ≥ 0.20 → “confident route”
* Else → “unknown/mixed route” and transcribe with multilingual models + later re-check language from transcript.

This makes the rest of the pipeline much cleaner, especially for Hindi/Punjabi/English code-switching.

---

## 2) ASR ensemble (best-in-class for your language set)

### A) Indic languages: use AI4Bharat as your primary “Indic specialist”

AI4Bharat explicitly maintains state-of-the-art ASR work for India and lists **IndicConformer / IndicWhisper / IndicWav2Vec** as their ASR model line. ([ai4bharat.iitm.ac.in][3])
The **IndicConformer 600M multilingual** model supports **IN-22** and uses a **hybrid CTC + RNNT** design (useful because CTC and RNNT often fail differently). ([Hugging Face][4])

**Recommendation for each Indic segment (Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese):**

* Model A: **AI4Bharat IndicConformer-600M-Multilingual** (decode both CTC and RNNT if you can)
* Model B: **Whisper large-v3** as a strong generalist baseline ([Hugging Face][5])
* Model C (optional but valuable): **Meta MMS ASR** multilingual (1107 languages) for diversity ([Hugging Face][1])

### B) English: add the strongest open speech models

* **Voxtral** (open-weight speech understanding) reports better transcription WER than Whisper large-v3 on their published benchmarks and supports automatic language detection. ([Mistral AI][6])
* **NVIDIA Nemotron Speech ASR** is a separate open model line (strong for streaming/throughput; can still be used offline for English). ([Hugging Face][7])

**Recommendation for English segments:**

* Voxtral + Whisper large-v3 (+ Nemotron Speech as extra diversity)

> Note: Voxtral’s public claims highlight multilingual performance and explicitly mention Hindi among supported major languages; for Tamil/Telugu/etc you should treat Voxtral as “test-and-measure” rather than assumed best. ([Mistral AI][6])

---

## 3) Combine outputs (don’t “pick one”; *vote/merge*)

Once you have multiple hypotheses, do **system combination** instead of choosing a single model output.

Classic method: **ROVER (Recognizer Output Voting Error Reduction)** — combines multiple ASR hypotheses to reduce WER. ([NIST][8])

**What to do in practice**

1. Normalize all hypotheses (Unicode normalization, remove trivial punctuation differences, normalize numerals consistently).
2. Align hypotheses (token/word or character level).
3. Do **weighted voting** (weights can be per-language, per-domain, per-model based on a small dev set).
4. Produce:

   * a final transcript
   * a “disagreement map” (positions where models differed)

This yields a measurable win when recognizers have different failure modes (IndicConformer vs Whisper vs MMS usually do).

---

## 4) Audio-grounded verification gate (how you avoid hallucinations + silent errors)

You want a gate that answers: “Does this text truly match the audio?”

### A) Forced alignment / alignment scoring

* **Montreal Forced Aligner (MFA)** is a gold-standard forced alignment workflow (Kaldi-based). ([montreal-forced-aligner.readthedocs.io][9])
* Research comparing forced alignment approaches shows modern ASR-based aligners can underperform classic forced aligners in alignment accuracy. ([arXiv][10])

**Reality for Indic:** MFA requires lexicons/acoustic models per language; that may be annoying for 12+ languages. A very effective alternative is:

### B) CTC-based “which transcript fits best?” scoring

For each candidate transcript (or ROVER result):

* run a **CTC aligner** (e.g., wav2vec2-style) and compute:

  * mean per-character posterior
  * coverage (% characters aligned confidently)
  * spikes of low confidence

This is the key “clever” move: **use an *independent* model to score fit**, not just trust the generator.

WhisperX-style pipelines are built around ASR + forced alignment; it’s a known pattern to use wav2vec2-family aligners for word/segment timing. ([arXiv][10])
(You can use Indic-focused wav2vec2 models as aligners too; AI4Bharat has IndicWav2Vec resources/models for Indian languages. ([aikosh.indiaai.gov.in][11]))

**Gate example (strict):**

* Accept only if:

  * LID confident
  * ROVER disagreement rate < 2–3%
  * CTC alignment score above threshold and no long low-confidence spans
    Else: send to re-decode (different settings) or human.

This is how you get “can’t afford mistakes” quality: **you reject aggressively**.

---

## 5) Hallucination defense (especially for short clips + silence)

Even with cleaned segments, you still want automatic guards:

* Whisper’s own transcription code supports fallback logic using thresholds like `compression_ratio_threshold`, `logprob_threshold`, and `no_speech_threshold`. ([GitHub][12])
* There’s also active research specifically on reducing Whisper hallucinations on non-speech (e.g., Calm-Whisper). ([arXiv][13])

**Do this:**

* Always run a VAD pass (or use your existing speech/music cleanup) and **trim leading/trailing silence hard**
* For Whisper-like models, run **beam search with temperature=0** first; only fall back to higher temperature if decoding failed (and if you do, treat as low confidence and require alignment to pass)

---

## 6) How to use LLMs “safely” (no invented words)

Text-only LLMs (Gemma, etc.) should **not** be used to “transcribe” from scratch; they will invent plausible text. The safe uses are:

### A) Constrained post-editing (only when models mostly agree)

Give the LLM:

* the 2–3 hypotheses
* the disagreement spans only
* hard constraints: **must choose among provided options**, or output `<unk>` for that span
  This turns the LLM into a **tie-breaker**, not a generator.

### B) Audio-LLMs that actually listen

If you want “LLM help” that’s still audio-grounded, use models that take audio as input (e.g., Voxtral). ([Mistral AI][6])
Even then, keep the same rule: **it must pass the alignment gate** before entering training.

---

## 7) Human-in-the-loop for the hard 1–5% (still scalable)

No matter how much compute you throw, the tail exists (names, rare words, heavy code-switching, low SNR). The optimal strategy is:

* auto-transcribe + auto-verify
* only humans touch **the rejects**

If you want an OSS stack for this, AI4Bharat’s **Shoonya** is an open-source annotation platform built for data labeling at scale. ([shoonya.ai4bharat.org][14])

---

## 8) The “go-to” stack for your exact languages (recommended)

**Language ID**

* MMS LID-4017 (+ optional SpeechBrain VoxLingua107)

**Transcription ensemble**

* Indic: IndicConformer-600M (CTC + RNNT outputs) ([Hugging Face][4])

  * Whisper large-v3 ([Hugging Face][5])
  * MMS ASR (optional diversity) ([Hugging Face][1])
* English: Voxtral + Whisper large-v3 (+ Nemotron Speech as extra) ([Mistral AI][6])

**Combination**

* ROVER (weighted) ([NIST][8])

**Verification**

* CTC alignment scoring (Indic wav2vec2-family aligner) + strict reject rules
* MFA where you can support it cleanly ([montreal-forced-aligner.readthedocs.io][9])

**LLM**

* Only constrained tie-breaks (choose among options) OR audio-LLM transcription + still gated ([Mistral AI][6])

---

## Minimal “flow diagram” (what you implement)

```text
Audio segment
  -> VAD trim + QC (SNR, clipping, duration)
  -> LID (MMS LID + optional SpeechBrain)  -> language / unknown
  -> ASR ensemble (IndicConformer + Whisper + optional MMS / Voxtral)
  -> Normalize hypotheses
  -> ROVER combine + disagreement map
  -> Alignment scoring (CTC aligner) on (top candidates + ROVER)
  -> PASS?  yes -> final transcript
            no  -> re-decode settings OR send to human (Shoonya)
```

---

## One non-negotiable (if you truly “can’t afford mistakes”)


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 118
Output:
High-Quality Transcription Strategy for Multi‑Lingual Indic Audio
Transcribing Indian-language audio segments with minimal word error rate (WER) requires a smart, multi-step pipeline. Below we outline a comprehensive strategy leveraging the best ASR models, ensembling techniques, and LLM-based error correction, given no constraints on compute or latency. The goal is to achieve near-human transcription quality (targeting ~1–2% WER) for languages including Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, and English.
1. Language Identification and Preparation
Before transcription, identify the language of each audio segment to route it to the optimal model. Many modern ASR models (e.g. Whisper, Voxtral, Gemma) can auto-detect language, but a dedicated LID (Language ID) model or metadata cues (video title/description) can provide extra confidence. Ensuring correct language detection is crucial so that the transcript uses the proper script and vocabulary. All audio should be normalized (e.g. convert to 16 kHz mono FLAC) as needed for the models.
Why: Indian languages can be acoustically similar (e.g. Punjabi vs. Hindi in some dialects), and accurate LID ensures using the right language model and decoding alphabet. Misidentification could degrade transcription quality.
2. Leverage State-of-the-Art ASR Models (No Single-Model Reliance)
With language known, transcribe using the strongest ASR models available. Since quality is paramount and compute is unlimited, we can ensemble multiple SOTA models per segment:
OpenAI Whisper (Large-V2) – A strong multilingual baseline. Whisper large-v2 is known for robust performance across many languages, including Indic ones. However, fine-tuned versions of Whisper on Indian data can drastically outperform the base model. For example, IndicWhisper (Whisper fine-tuned on 12 Indian languages, ~10.7k hours) achieved the lowest WER on 39 of 59 Indian ASR benchmarks, reducing WER by ~4.1 absolute on average
. Similarly, specialized models like Oriserve’s Hinglish Whisper (for Hindi-English code-mixed speech) show ~39% relative WER improvement over original Whisper on benchmarks
 and mitigate hallucinations in transcripts
. Using such fine-tuned models for each language (Hindi, Tamil, etc.) will yield more accurate transcripts than a generic model.
Mistral Voxtral (24B) – A new open-source ASR model that outperforms Whisper large-v3 across multiple languages
. Voxtral is a 24B-parameter “speech understanding” model with a Whisper-based audio encoder and an LLM backbone. It has state-of-the-art transcription accuracy on benchmarks (beating even some proprietary services) and native multilingual support (auto language detection, strong results in widely-used languages like English, Spanish, Hindi, etc.
). Given unlimited compute, running Voxtral on each segment can provide an extremely accurate transcript (often better than Whisper). It also supports long context (up to 30+ minutes) and has built-in understanding capabilities, although here we primarily use its transcription strength.
Google Gemma 3n – Google’s latest open multimodal model (small enough for offline use) with high-quality ASR for 140+ languages (including 6 major Indian languages)
. Gemma 3n is designed for on-device speech recognition and translation with chat-style prompting. We can prompt Gemma with instructions (e.g. “transcribe verbatim, do not add extra text”) to guide it. Its multilingual prowess and ability to follow prompts can help maintain accuracy without hallucination. For supported Indian languages (likely Hindi, Tamil, Telugu, Bengali, Marathi, Malayalam), Gemma 3n offers top-tier open-model performance, rivaling other SOTA models. It’s also optimized to run efficiently (even on a laptop) while delivering real-time or offline transcription.
Other Specialized Models: If available, incorporate any language-specific ASR models with known superior performance. For example, NVIDIA’s Conformer-CTC models for Hindi, or AI4Bharat’s IndicConformer (a 30M-parameter real-time model for 22 Indian languages
). These smaller models may not beat Whisper or Voxtral on accuracy, but if fine-tuned on accents or dialects, they could correct certain mistakes. They are optional given our focus on quality over speed, but can be part of the ensemble for diversity. Also, any big tech ASR APIs (Google Cloud, Azure, etc.) could be used on a sample basis to compare output – though not feasible for full data due to cost, their transcripts could serve as a reference for tricky cases.
Strategy: For each audio segment, run at least two or three different top-tier ASR models in parallel, for example Whisper/IndicWhisper, Voxtral, and Gemma. This yields multiple candidate transcripts for the same speech. Each model has different strengths, so their errors will differ. For instance, Voxtral might handle a Hindi-English mixed sentence differently than IndicWhisper; …7851 chars truncated…nscribers who catch context that AI misses
 – a small manual layer can elevate quality from “very good” to “virtually perfect”.
Feedback into Training: Optionally, use those human-corrected segments to further fine-tune your models or correction LLM. Since we have time and no compute limit, iterative improvement is possible. Over a few rounds, the need for human fixes will diminish as models learn from their mistakes.
6. Putting It All Together (Optimal Pipeline)
Finally, here’s a step-by-step pipeline summarizing the strategy:
Language ID & Segmentation: Confirm each audio segment’s language (Malayalam, Kannada, English, etc.) and ensure it’s in the right format (mono 16 kHz, etc.).
Multi-Model Transcription: Run the segment through multiple ASR models – e.g. Whisper Large (or IndicWhisper fine-tune)
, Voxtral Small/24B
, and possibly Gemma 3n or a specialized model. Allow Whisper to auto-detect language or specify the language to each model if needed (to avoid any confusion for languages like Assamese vs. Bengali script).
Consensus Assembly (ROVER): Align the outputs and perform majority voting or confidence-weighted voting to merge into one transcript. This resolves most obvious errors as demonstrated by ROVER’s improvements in WER
.
LLM/LM Correction Pass: Take the merged transcript and run a correction pass. This could involve:
A sequence-to-sequence correction model (trained on similar data) that outputs a cleaned transcript.
Or an LLM prompt that compares multiple versions (the individual ASR outputs plus the merged one) and fixes any anomalies without adding new content.
Optionally, verify uncommon words against a knowledge base or use an n-gram probability check to ensure the transcript is linguistically plausible
.
Final Review & Edit: If any segments are still suspect (e.g., marked by low confidence or by the LLM saying “unclear”), have a human reviewer listen and correct. This should be a small portion of the data thanks to the rigorous automated steps prior.
Output High-Quality Transcript: The end result for each segment is an extremely accurate transcript, ready for use in training TTS or ASR systems.
By adopting this multi-pronged approach – using the best ASR models (Whisper, Voxtral, Gemma, etc.) in combination, plus smart voting and LLM-guided correction – we maximize transcription quality. We effectively minimize WER by catching different types of errors at each stage: acoustic model errors via ensemble, and language/context errors via LLM. This is the “clever combo” that takes advantage of offline, unlimited compute conditions. In summary, the go-to strategy is: use robust multilingual ASR engines fine-tuned for Indic languages, compare their outputs, and refine with advanced language models and selective human oversight. With this strategy, it’s feasible to approach human-level transcription accuracy (WER on the order of 1–2%) for a wide array of Indian languages in offline processing
. The transcripts produced will be of extremely high quality, providing a solid foundation to train top-notch Indic ASR and TTS models. Sources: High-level ASR model performance and capabilities
, IndicWhisper fine-tuning gains
, Hinglish ASR improvements
, ROVER ensemble effectiveness
, and advanced error-correction techniques
 have all informed this approach. Each component is chosen to ensure no compromise on quality, aligning with the ultimate goal of accurate, reliable transcriptions for Indic audio.
Citations

[2305.15386] Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR

https://arxiv.org/abs/2305.15386

Oriserve/Whisper-Hindi2Hinglish-Prime · Hugging Face

https://huggingface.co/Oriserve/Whisper-Hindi2Hinglish-Prime

Oriserve/Whisper-Hindi2Hinglish-Prime · Hugging Face

https://huggingface.co/Oriserve/Whisper-Hindi2Hinglish-Prime

Voxtral | Mistral AI

https://mistral.ai/news/voxtral

Voxtral | Mistral AI

https://mistral.ai/news/voxtral

Gemma-fied! Putting the power of our most capable family of open models into the hands of every Indian developer

https://blog.google/intl/en-in/company-news/gemma-fied-putting-the-power-of-our-most-capable-family-of-open-models-into-the-hands-of-every-indian-developer/

AI4Bharat Models

https://models.ai4bharat.org/

[1706.07238] Automatic Quality Estimation for ASR System Combination

https://arxiv.org/abs/1706.07238

[1706.07238] Automatic Quality Estimation for ASR System Combination

https://arxiv.org/abs/1706.07238

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

ASR Error Correction: Methods & Advances

https://www.emergentmind.com/topics/asr-error-correction-aec

Best option for audio or video transcription now? : r/LocalLLaMA

https://www.reddit.com/r/LocalLLaMA/comments/1oe5igo/best_option_for_audio_or_video_transcription_now/
All Sources


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Since compute is not a constraint and your goal is absolute minimal Word Error Rate (WER) to train SOTA models, you should not rely on a single ASR model. Even the best models (like Whisper V3) hallucinate or mishear specific Indian phonemes.

The "cleverest" strategy is to build a **Consensus & Arbitration Pipeline** (often called a ROVER system in research). You will run multiple SOTA models in parallel and use a "Judge" (LLM + Acoustic Aligner) to pick or merge the best result.

Here is the **Ultimate "Zero-Compromise" Offline Pipeline** for Indian Languages.

### Phase 1: The "Gatekeeper" (Ensemble Language Identification)

Since your files have missing metadata, you cannot trust a single LID (Language ID) model. If you feed Hindi audio to a Bengali model, you will get garbage that ruins your TTS training.

* **Strategy:** Run three distinct LID models on every file.
1. **OpenAI Whisper:** (Built-in language detection).
2. **SpeechBrain:** (Pre-trained `lang-id-voxlingua107-ecapa` classifier).
3. **Meta MMS / NVIDIA NeMo:** (Alternative architecture).

* **The Logic:**
* If all 3 agree  **Pass**.
* If they disagree  **Quarantine** for manual review or discard. *Do not risk polluting your dataset.*

### Phase 2: The "Council of Transcribers"

Run every audio segment through the Top 4 SOTA models. Each has different strengths.

| Model | Role | Why it's needed |
| --- | --- | --- |
| **AI4Bharat IndicWhisper** | **The Specialist** | **Crucial.** Fine-tuned specifically for Indian languages. It often outperforms standard Whisper on Dravidian languages (Malayalam, Tamil, Telugu, Kannada). |
| **OpenAI Whisper (Large-v3)** | **The Generalist** | Excellent general robustness and vocabulary. Acts as the baseline anchor. |
| **NVIDIA Canary / Parakeet** | **The Challenger** | Currently SOTA on the Open ASR Leaderboard. Very strong on multilingual tasks and often beats Whisper on speed/accuracy. |
| **Meta MMS (1B)** | **The Diversity Vote** | Trained on Massively Multilingual Speech. Useful tie-breaker for lower-resource languages like Odia or Assamese. |

### Phase 3: The "Judge" (Consensus & Validation)

Now you have 3-4 slightly different transcripts for the same audio. How do you pick the perfect one without a human?

#### A. The Acoustic Truth Filter (Forced Alignment)

This is the scientifically "clever" part. A transcript might look grammatically correct but be phonetically wrong (hallucination).

* **Tool:** Use **Montreal Forced Aligner (MFA)** or **NVIDIA NeMo Aligner**.
* **Action:** Force-align every transcript back to the audio.
* **Metric:** Calculate the **Acoustic Likelihood Score**.
* **Result:** If Whisper says "apple" but the audio sounds like "ample," the alignment score will drop. You pick the transcript with the highest alignment confidence.

#### B. The Semantic Judge (LLM Arbitration)

Use a high-intelligence LLM (GPT-4o, Claude 3.5 Sonnet, or a fine-tuned Indic LLM like **Sarvam-1**) as a "Reviewer."

* **Prompt:** *"I have three transcripts of the same Hindi audio file. Transcript A says [...], Transcript B says [...]. Which one is the most linguistically probable and preserves the specific Indian entity names best? Output only the best version."*
* **Clever Trick:** Use the **ROVER (Recognizer Output Voting Error Reduction)** approach. If 3 models agree on a word, keep it. If they disagree, let the LLM decide based on context.

### The Workflow Diagram

```mermaid
graph TD
    A[Raw Audio Segment] --> B{LID Ensemble}
    B -- Agreement --> C[Confirmed Language]
    B -- Disagreement --> D[Discard/Manual]

    C --> E[AI4Bharat IndicWhisper]
    C --> F[OpenAI Whisper V3]
    C --> G[NVIDIA Canary]

    E --> H[Transcript 1]
    F --> I[Transcript 2]
    G --> J[Transcript 3]

    H & I & J --> K{Alignment Check}
    K --> L[Score Acoustic Confidence]

    H & I & J & L --> M[LLM Consensus Judge]
    M --> N[Final Gold Transcript]

```

### Summary of Tools to Deploy

1. **Primary Model:** **AI4Bharat/IndicWhisper** (HuggingFace). This is non-negotiable for high-quality Indic TTS data.
2. **Secondary Model:** `faster-whisper` (Large-v3 implementation).
3. **Consensus Tool:** `meetecho/rover` or write a simple Python script using `difflib` to find differences.
4. **Alignment:** `Montreal Forced Aligner` (MFA) - pre-trained models are available for most Indian languages.

### One Final "Pro" Tip

Since you are training **TTS** (Text-to-Speech), you must be aggressive about **Speech Quality**, not just text accuracy.

* Use a model like **WavLM** or **Brouhaha** to score the audio for "Signal-to-Noise Ratio" (SNR) and "Reverberation."
* Even if the transcript is perfect, if the audio has background hiss, **throw it out**. Bad audio destroys TTS prosody faster than bad transcripts.

... [Deep dive into cleaning audio data for TTS](https://www.google.com/search?q=https://www.youtube.com/watch%3Fv%3DyYcZz4j73K8)

**Relevance:** This video explains the practical nuances of preparing dataset specifically for TTS training, focusing on why "clean" audio matters as much as the transcription itself.


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 200
Output:
This is an excellent problem to solve systematically. Given your constraints (offline, unlimited compute, quality-first), let me design a **multi-stage ensemble pipeline with LLM-as-Judge** architecture that minimizes WER while avoiding hallucination risks.

## The Core Strategy: Ensemble + Consensus + Verification

The key insight is: **Use multiple specialized ASR systems → Build consensus → Use LLM as a JUDGE (not generator) → Quality score and flag outliers**

---

## Stage 1: Language Identification (Critical First Step)

Since you have unknown language segments, start here:

| Model | Strength | Notes |
|-------|----------|-------|
| **IndicLID** (AI4Bharat) | Best for Indic languages | 22 Indic languages |
| **Whisper's LID** | Good fallback | Built-in during inference |
| **MMS-LID** (Meta) | 4000+ languages | Very robust |
| **NLLB-LID** | 200+ languages | Good accuracy |

**Strategy:** Run IndicLID + MMS-LID → If they agree, proceed. If disagree, use majority voting with Whisper's detection as tiebreaker.

---

## Stage 2: Multi-Model ASR Ensemble

For each audio segment, run ALL of these (since compute isn't a constraint):

### Tier 1: Indic-Specialized Models (Highest Priority)
| Model | Languages | Why |
|-------|-----------|-----|
| **IndicWhisper** (AI4Bharat) | All your languages | Fine-tuned Whisper on Indic, SOTA for most |
| **IndicConformer** (AI4Bharat) | 9 Indic languages | CTC-based, different architecture = diverse errors |
| **MMS** (Meta) | 1000+ including all Indic | Massive multilingual, different training data |
| **IndicWav2Vec2** | Indic languages | Self-supervised, good for low-resource |

### Tier 2: General Multilingual Models
| Model | Why |
|-------|-----|
| **Whisper Large V3** | Strong baseline, good for code-switching |
| **SeamlessM4T v2** (Meta) | Multimodal, different architecture |
| **Canary-1B** (NVIDIA) | Strong multilingual ASR |
| **USM** (Google, if accessible) | 300+ languages |

### Tier 3: Audio-Native LLMs (Use Carefully)
| Model | Why | Caution |
|-------|-----|---------|
| **Gemini 2.0 Flash** | Native audio input, good Indic | Can hallucinate - use for verification only |
| **Qwen2-Audio** | Direct audio understanding | Same caution |
| **Voxtral** (Mistral) | New, multimodal | Limited Indic evaluation data |

---

## Stage 3: Consensus Building Algorithm

This is where you get clever:

```
For each audio segment:
    1. Collect N transcriptions from N models
    2. Normalize all transcripts (unicode, punctuation, numbers)
    3. Compute pairwise WER/CER between all pairs
    4. Find the "centroid" transcript (lowest average distance to others)
    5. Weight by model confidence scores if available
    6. If high disagreement (variance > threshold) → flag for Stage 4
```

### The ROVER (Recognizer Output Voting Error Reduction) Approach:
- Align all hypotheses using dynamic programming
- At each position, vote across models
- Weight votes by model's known accuracy for that language

---

## Stage 4: LLM-as-Judge (NOT Generator)

This is the critical part where you avoid hallucination:

### ✅ DO: Use LLM to SELECT among candidates
```
You are an expert transcription judge for {language}.

Here are 5 different ASR transcriptions of the same audio:
1. [Transcript A]
2. [Transcript B]
3. [Transcript C]
4. [Transcript D]
5. [Transcript E]

Your task:
- Identify which transcript is most likely CORRECT
- Point out specific errors you notice in others (missing words, wrong words)
- You MUST select from the given options - DO NOT create new text
- If all seem wrong, indicate the most plausible one and flag confidence as LOW

Output:
- Selected: [number]
- Confidence: HIGH/MEDIUM/LOW
- Reasoning: [brief explanation]
```

### ❌ DON'T: Ask LLM to "fix" or "improve" transcriptions
This leads to hallucination - the LLM will generate plausible-sounding but incorrect text.

### Best LLMs for Indic Judgment:
1. **Gemini 2.0 Flash/Pro** - Excellent Indic understanding
2. **GPT-4o** - Good multilingual reasoning
3. **Llama 3.3 70B** - If you want local
4. **IndicBERT/MuRIL** - For perplexity scoring (not generation)

---

## Stage 5: Quality Scoring & Flagging

Compute a composite quality score:

```python
quality_score = (
    0.3 * model_agreement_score +      # How much do models agree?
    0.2 * avg_confidence_score +        # Model confidence scores
    0.2 * llm_judge_confidence +        # LLM's confidence in selection
    0.2 * language_model_perplexity +   # Does it read like real {language}?
    0.1 * audio_quality_score           # SNR, clarity metrics
)

if quality_score < threshold:
    flag_for_manual_review()
```

---

## Recommended Pipeline Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                    AUDIO SEGMENT INPUT                          │
└────…527 chars truncated…───┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 STAGE 2: MULTI-MODEL ASR                        │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐    │
│  │  IndicWhisper  │  │IndicConformer  │  │      MMS       │    │
│  └────────────────┘  └────────────────┘  └────────────────┘    │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐    │
│  │ Whisper L-V3   │  │  SeamlessM4T   │  │  IndicWav2Vec  │    │
│  └────────────────┘  └────────────────┘  └────────────────┘    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│            STAGE 3: CONSENSUS BUILDING (ROVER)                  │
│                                                                 │
│    Align → Vote → Weight by confidence → Select centroid        │
│                                                                 │
│    High Agreement? ──YES──→ Accept transcript                   │
│          │                                                      │
│          NO                                                     │
│          ▼                                                      │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│          STAGE 4: LLM-AS-JUDGE (Disagreement Cases)             │
│                                                                 │
│   Present all candidates → LLM selects best → NO generation     │
│                                                                 │
│   Models: Gemini 2.0 Flash, GPT-4o, or Llama 3.3 70B           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│               STAGE 5: QUALITY SCORING                          │
│                                                                 │
│   Composite score → Flag low confidence → Manual review queue   │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                 FINAL TRANSCRIPT + METADATA                     │
│  {transcript, language, confidence, model_agreement, flags}     │
└─────────────────────────────────────────────────────────────────┘
```

---

## Language-Specific Recommendations

| Language | Primary Model | Secondary | Notes |
|----------|--------------|-----------|-------|
| **Hindi** | IndicWhisper | Whisper V3 | Best covered, code-switching common |
| **Tamil** | IndicConformer | IndicWhisper | Strong AI4Bharat support |
| **Telugu** | IndicWhisper | MMS | Good coverage |


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 176
Output:
# Optimal strategy for high-quality Indian language transcription

**The lowest possible WER for Indian language audio comes from a four-stage pipeline: language identification with MMS-LID, multi-model transcription using IndicWhisper as the primary engine, ROVER-based ensemble combination, and confidence-gated LLM correction with hallucination safeguards.** This approach can achieve **10-35% WER** across most Indian languages—with Hindi and Kannada performing best (~10-18% WER) and Malayalam remaining most challenging (~32% WER). Ensemble methods alone reduce errors by **10-15% relative**, while properly constrained LLM post-correction adds another **10-20% relative improvement**.

The critical insight is that no single model dominates across all 12 target languages. IndicWhisper achieves the lowest WER on 39 of 59 Vistaar benchmark tests, but MMS excels on long-tail languages and SeamlessM4T handles noisy audio better. A strategic combination maximizes accuracy where quality is the only priority.

## The recommended four-stage pipeline architecture

The optimal architecture flows through distinct stages: preprocessing with VAD segmentation, language identification, parallel multi-model ASR with ensemble combination, and confidence-gated LLM verification.

**Stage 1: Preprocessing**
- Use **Silero VAD** for voice activity detection (trims ~35% silence, saves compute)
- Segment audio into **30-second chunks** (Whisper's optimal context window)
- Normalize to 16kHz mono audio
- Skip aggressive denoising—it often introduces artifacts that hurt transcription quality

**Stage 2: Language Identification**
- Primary: **MMS-LID** (facebook/mms-lid-256) for spoken audio detection across 4,017 languages
- Backup: **SpeechBrain VoxLingua107** for utterance embeddings
- Confidence threshold logic: >0.9 routes to language-specific model; 0.7-0.9 uses multilingual model; <0.7 flags for ensemble processing

**Stage 3: Multi-Model ASR with Ensemble Combination**
- Run **three models in parallel**: IndicWhisper (language-specific), Whisper large-v3, and MMS-1B
- Combine outputs using **ROVER with confidence-weighted voting** (NIST SCTK toolkit)
- Apply Indic text normalization with **IndicNLP Library** (preserves diacritics that Whisper's default normalizer strips)

**Stage 4: LLM-Based Correction with Hallucination Prevention**
- Only process utterances with sentence confidence <0.95 or any word confidence <0.7
- Use **IndicBART** or **Airavata** for correction with constrained prompts
- Apply edit distance check (reject changes >30% of original) and length ratio check (±15% tolerance)
- If checks fail, revert to original ASR output

## Best ASR models ranked by Indian language performance

**IndicWhisper** (AI4Bharat) represents the current state-of-the-art for most Indian languages. Fine-tuned on 10,700+ hours across 12 languages via the Vistaar-Train dataset, it achieves the lowest WER in 39 of 59 benchmarks with **4.1% average relative WER reduction** over competitors.

| Language | IndicWhisper WER | Notes |
|----------|------------------|-------|
| Hindi | **10.3-15.0%** | Best overall performance |
| Kannada | **13.2-22.2%** | Strong across datasets |
| Marathi | **11.4-22.8%** | Consistent performer |
| Bengali | **16.6-24.7%** | Good baseline |
| Punjabi | **16.9-23.1%** | Reliable results |
| Gujarati | **17.8-23.5%** | Solid performance |
| Tamil | **17.2-29.2%** | Higher variance |
| Telugu | **25.0-33.8%** | Challenging language |
| Odia | **15.0-35.2%** | Wide variance by dataset |
| Malayalam | **21.4-44.5%** | Most difficult—limited training data |

**Meta's Omnilingual ASR** (November 2025, 7B parameters) offers the widest coverage at 1,600+ languages with **78% achieving CER <10%**. It includes long-tail Indian languages like Kui, Chattisgarhi, Maithili, and Bagheli that no other model covers. However, its recency means limited independent validation.

**IndicConformer** (AI4Bharat) provides the widest Indian language coverage with all 22 constitutionally scheduled languages supported. The 600M multilingual model uses a hybrid CTC-RNNT architecture and works efficiently via the NeMo framework.

**SeamlessM4T v2** halves WER relative to Whisper Large-v2 on FLEURS benchmarks and handles background noise exceptionally well. Its limitation is the CC-BY-NC license restricting commercial use.

**Sarvam Saarika** specializes in real-world code-mixed speech—critical for YouTube podcast content where Hindi-English mixing is common. While lacking public benchmarks, it's purpose-built for Indian colloquial speech patterns.

## Ensemble strategies that measurably reduce WER

ROVER (Recognizer Output Voting Error Reduction) provides **10-15% relative WER reduction** when combining diverse models. The algorithm works in two stages: first aligning outputs via dynamic programming into a Word Transition Network, then applying confidence-weighted voting at each branch point.

**Implementation with NIST SCTK:**
`…1993 chars truncated… originals
3. Do NOT add, remove, or restructure content
4. If uncertain, keep original unchanged

Transcript: [ASR_OUTPUT]
Low-confidence words: [WORD_LIST]
Corrected:
```

**Hallucination detection safeguards:**
- **Edit distance ratio <0.3**: Reject corrections changing >30% of characters
- **Length ratio ±15%**: Flag outputs differing significantly in length
- **Chain-of-thought verification**: Require the LLM to explain each change, verify explanations are phonetically plausible

**Confidence-based filtering** dramatically reduces unnecessary LLM calls. Sending only utterances with **sentence confidence <0.95 or word confidence <0.7** to LLM correction reduces processing by ~77% while capturing genuinely problematic segments.

## Language identification for unlabeled segments

**MMS-LID** (facebook/mms-lid-256 or mms-lid-4017) is the clear winner for spoken language identification, covering 4,017 languages including all target Indian languages. It works directly on audio using a Wav2Vec2-based classifier.

**Critical insight**: Whisper's built-in language detection is unreliable for Indian languages, particularly:
- **Hindi-Marathi confusion**: Similar Devanagari phonetics
- **Bengali-Assamese confusion**: Related scripts and sounds
- **Short utterances (<3 seconds)**: Insufficient signal
- **Code-mixed content**: Switches mid-sentence

**Recommended detection flow:**
1. Run MMS-LID on audio segment
2. If confidence >0.9: Route to language-specific IndicWhisper model
3. If confidence 0.7-0.9: Use multilingual IndicConformer
4. If confidence <0.7: Run through multiple models, use ensemble combination

**For code-switching detection**, current state-of-the-art achieves ~32% WER on MUCS Hindi-English benchmarks. The best approach is end-to-end multilingual models rather than cascaded LID→ASR pipelines that fail at switch points.

## Handling edge cases in podcast content

**Code-switching** (Hindi-English, Tamil-English) requires special handling since YouTube podcasts frequently mix languages. **Sarvam ASR-Translate** converts code-mixed Indic speech directly to English text with topical context awareness. Alternatively, train or fine-tune multilingual models on code-switched data from MUCS challenge datasets.

**Accented regional speech** varies significantly—the Svarah benchmark covers 117 speakers from 19 Indian states. For production robustness:
- Use multi-condition training (diverse accents in training data)
- Fine-tune on regional accent samples when available
- The IndicVoices dataset from AI4Bharat provides diverse speaker coverage

**Handling low-quality segments** in cleaned YouTube audio:
- Use SNR estimation to route audio appropriately
- For noisy segments: Prefer noise-robust models (SeamlessM4T excels here)
- Apply light preprocessing rather than aggressive denoising
- Flag segments with audio quality scores below threshold for manual review

## Practical implementation recommendations

**Model hosting stack:**
```
Primary ASR: IndicWhisper (ai4bharat/indicwhisper-{lang})
Secondary ASR: Whisper large-v3 (openai/whisper-large-v3)
Tertiary ASR: MMS-1B (facebook/mms-1b-all)
Language ID: MMS-LID (facebook/mms-lid-256)
Text Normalization: IndicNLP Library
Ensemble: NIST SCTK ROVER
LLM Correction: IndicBART or Airavata
```

**Quality scoring without ground truth:**
- Multi-model agreement score (0-1 based on Jaccard similarity between outputs)
- Language model perplexity (flag outliers >2σ from mean)
- Repetition detection (Whisper hallucination pattern: excessive phrase repetition)
- Audio-text alignment confidence from forced alignment

**Confidence thresholds for routing:**
| Score | Action |
|-------|--------|
| >0.95 | Auto-accept without LLM pass |
| 0.80-0.95 | Standard accept, sample for QA |
| 0.60-0.80 | Send to LLM correction |
| <0.60 | Manual review or re-process with ensemble |

## Conclusion

The optimal strategy combines **IndicWhisper as the primary model** (lowest WER on most benchmarks), **MMS-LID for language identification** (4,000+ language coverage), **ROVER ensemble combination** (10-15% relative improvement), and **confidence-gated LLM correction with IndicBART/Airavata** (additional 10-20% improvement with hallucination safeguards).

For the specific use case of YouTube podcast content with code-switching, **Sarvam Saarika** deserves serious consideration despite lacking public benchmarks—it's purpose-built for real-world Indian speech patterns. The key architectural principle is routing: use language confidence to select between language-specific fine-tuned models (high confidence) and multilingual models (lower confidence or code-mixed content).

With compute constraints removed, the maximum quality approach runs **three parallel ASR models, combines via ROVER, then applies selective LLM correction only where confidence is low**—achieving the best possible accuracy while preventing hallucination through constrained prompting and edit distance verification.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Since you have **no compute constraints** and are operating **offline**, you have the luxury of trading time for accuracy. The "cleverest" way to minimize WER (Word Error Rate) for a training dataset is not to rely on a single model, but to build a **Consensus-Based Mixture-of-Experts (MoE) Pipeline** with a **Discriminative Verification Loop**.

This approach treats transcription as a "voting" problem among diverse architectures, using an Indic-specialized LLM as the judge.

Here is the **Zero-Compromise Indic Transcription Strategy**:

### Phase 1: The "Forensic" Language Identification (LID)

You cannot trust metadata. If you feed Hindi audio to a Marathi model, you get "hallucinations" (garbage text that looks real).

* **The Strategy:** Run a **Triple-Check Ensemble** on every file.
1. **Specialist:** **AI4Bharat IndicLID** (Specifically trained to distinguish 22 Indian languages; handles code-mixing better than Western models).
2. **Generalist:** **Whisper Large-v3** (Internal language detection).
3. **Acoustic:** **SpeechBrain (VoxLingua107)** or **Silero**.

* **The Rule:** Only proceed if **2 out of 3** models agree on the language. If they disagree (e.g., confusion between Bengali/Assamese or Hindi/Urdu), flag the file for manual review. **Do not pollute your training set.**

### Phase 2: The ASR "Council of Experts"

Run every segment through **three** distinct SOTA models. We choose these because they have *different* architectural biases, meaning they rarely make the *same* mistake.

1. **Expert A: OpenAI Whisper Large v3**
* **Role:** The Robust Generalist.
* **Why:** Unbeatable for background noise, music, and "Code-Mixing" (English words mixed with Indic languages).

2. **Expert B: AI4Bharat IndicConformer**
* **Role:** The Native Specialist.
* **Why:** Unlike Whisper (Transformer), this uses a Hybrid CTC/RNN-T architecture. It is the current SOTA for pure Indic phonetics and properly captures Indian proper nouns (names, cities) that Whisper often Anglicizes.

3. **Expert C: NVIDIA Canary-1B** (or Google USM if available)
* **Role:** The Third Opinion.
* **Why:** Canary is a "FastConformer" model that often catches short, rapid speech that Whisper misses.
* *Note on "Voxtral":* There is no dominant model named "Voxtral" currently. You likely meant **Voctro** (TTS) or are thinking of a Mistral-based audio model. I recommend replacing this slot with **NVIDIA Canary** or **SeamlessM4T v2** for the best multimodal performance.

**Crucial Step:** Do not just save the text. Extract the **N-Best Hypotheses** (e.g., top 5 options) from each model.

### Phase 3: The "Clever" Judge (LLM Rescoring)

This is where you beat standard ASR. Instead of asking an LLM to *generate* text (which risks hallucination), use it to *score* the hypotheses.

* **The Model:** **Sarvam-1 (2B)**.
* **Why:** Sarvam-1 is trained on 2 trillion Indic tokens. It has a significantly better internal probability model for Hindi/Tamil/Telugu grammar than GPT-4 or Llama-3.

* **The Method (Perplexity Ranking):**
1. Take the transcripts from Whisper, IndicConformer, and Canary.
2. Feed them into **Sarvam-1** and calculate the **Perplexity (PPL)** score for each sentence.
3. **Select the transcript with the lowest Perplexity.**

* *Logic:* A native speaker (the LLM) will find the correct transcript "more natural" (lower perplexity) than a transcript with phonetic errors.

### Phase 4: The "Hallucination Killer" (Forced Alignment)

Since you are training ASR/TTS models, **alignment is non-negotiable**.

* **Tool:** **Montreal Forced Aligner (MFA)** or **NeMo Forced Aligner**.
* **Action:** Align the final selected text back to the audio.
* **The Filter:**
* If the aligner cannot find a specific word in the audio (low confidence score), **drop that word**.
* If the alignment fails for >20% of the segment, **discard the entire segment**.
* *Result:* This mathematically guarantees that your training data contains no "Whisper Hallucinations" (like "Thank you for watching" on silent audio).

### Summary Pipeline

| Component | Tool/Model | Purpose |
| --- | --- | --- |
| **1. ID** | **IndicLID + Whisper + Silero** | Ensure strict language purity. |
| **2. Listen** | **Whisper V3 + IndicConformer + Canary** | Get 3 diverse "opinions". |
| **3. Judge** | **Sarvam-1 (2B)** | Use PPL scoring to pick the most grammatically valid option. |
| **4. Verify** | **Forced Alignment (MFA)** | Mathematical proof that text matches audio. |

**Why this is the "Best" Strategy:**
You are effectively building a **Generative Adversarial Network (GAN)** logic manually. The ASR models are the "Generators," and Sarvam-1 + Forced Aligner are the "Discriminators." By running this offline with infinite compute, you ensure that only the "truth" survives to your training set.


**Considering audio download implementation**

I need to implement a pipeline to download audio segments from a videoID stored in a Supabase record with an R2 path. Since I'm unsure about the database schema, it seems I should inspect the Supabase database. I'll connect using credentials from the .env file and implement the Supabase client for querying. I should avoid running on user data, but I can attempt a small sample test. I’ll also check the .env for variable names without exposing any secrets.


**Considering secret handling**

I need to ensure we don’t inadvertently leak secrets in the final message. Treating secrets as confidential is essential. Even though the user might see terminal output, it’s still their environment and their secrets, so it's best practice to avoid printing them. I can inspect the .env file while masking values. Using a command like `awk -F= '{print $1}' .env` will help me display variable names only, keeping it safe. Let’s proceed with that.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
ACCOUNT_ID
ACCOUNT_ID
API_TOKEN
DATABASE
DATABASE_URL
docker_image
docker_pat
HF_TOKEN
HF_TOKEN
HOST
MAX_CONSECUTIVE_EMPTY
MAX_CONSECUTIVE_EMPTY
MAX_UTILIZATION
NGROK
ORCHESTRATOR_URL
PORT
PROJECT_ID
R2_ACCESS_KEY_ID
R2_ACCESS_KEY_ID
R2_BUCKET
R2_BUCKET
R2_ENDPOINT_URL
R2_ENDPOINT_URL
R2_PREFIX
R2_SECRET_ACCESS_KEY
R2_SECRET_ACCESS_KEY
RUNPOD_KEY
S3_API
SUPABASE_ADMIN
SUPABASE_ANON
URL
USER
VAST_AI
WORKER_PROVIDER
WORKER_REGION


**Evaluating configuration and implementation**

I’m noticing there might be duplicates due to multiple sections in my setup. I have my R2 credentials and endpoint, as well as Supabase keys like SUPABASE_ADMIN and SUPABASE_ANON. I need to inspect the .env file carefully, ensuring I mask values. I should implement a config loader with python-dotenv to read this. Also, I’m debating whether to use AWS S3 or Cloudflare R2 for file access. Following that, I need to think about handling tar archives, downloading, and ensuring safe extraction. I'll also set up some audio quality checks.


**Planning audio processing implementation**

I see that Silero VAD uses Torch, which I plan to use for models anyway, but I could keep it optional. The user seems to want robust performance, so I'll lean toward implementing with the best components, even if it's heavy. We’ll include dependencies, but they should remain optional.

I also need to implement language identification using MMS-LID and will opt for the facebook/mms-lid-256 model as it’s smaller. I’ll use the transformers pipeline for audio classification, mapping predicted labels to language codes based on ISO 639-3. For ASR, I’ll explore multiple models, including Whisper large-v3, using various pipelines.


**Evaluating ASR model choices**

For offline processing, `faster-whisper` could be a good choice regarding speed, and it might offer similar quality. It seems like the easiest option since it's run on CPUs, even though it's heavy. I need to consider using both `faster-whisper` (CTranslate2) and `openai-whisper`.

There’s also the `ai4bharat/indicwhisper`, possibly requiring Hugging Face, and using `transformers` for speech-to-text seems feasible. It appears that `faster-whisper` can download and convert models from Hugging Face automatically. As for custom fine-tuned models, that may not be straightforward, especially with `IndicConformer` built on NeMo, which could be heavier to work with.


**Implementing ASR architecture**

I need to start implementing the architecture with a plugin interface for models. The initial setup might include integrating the whisper large-v3 with faster-whisper and potentially `openai/whisper-large-v3`, while also considering HF-based models for language identification. The user seems to want a robust version, so I should aim to integrate at least two ASR models right now.

I'm thinking of using `openai/whisper-large-v3` alongside `facebook/seamless-m4t-v2-large`, though I'm unsure about the licensing. There's also the option of `nvidia/parakeet`. For combining outputs, I could implement an ROVER-like consensus method or roll out a simplified version in Python using dynamic programming for alignment. Tools like `jiwer` could help with the alignment task.


**Designing the robust pipeline**

I’m working on an initial pipeline that will use char-level or word-level alignment and voting for verification. Integrating audio-grounded verification through CTC-based scoring needs an aligner model like wav2vec2. I can consider using the `ctc-segmentation` library or `torchaudio`, which has forced alignment capabilities.

Furthermore, I should utilize Whisper’s metrics like average logprob and no-speech probability for quality assessment and implement a consensus disagreement method using pairwise WER between model outputs. The user has outlined a detailed plan, including creating a Python package `maya_transcribe`, setting up configuration, and providing a CLI. I’ll need to ensure the structure allows for a seamless future integration of models while focusing on building the verification pipeline. This is definitely a substantial project but seems manageable with a clear plan. I should also check the relevant documentation and ensure I follow the user's instructions carefully.


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Considering database structure**

I'm thinking there might be a `videos` table that includes `video_id`, `language`, and maybe `r2_key`, plus a `segments` table with `segment_id` and `transcript`. I can query this with a Supabase admin key using the Supabase Python library. Without a schema, I wonder if a generic metadata fetcher can work — but I doubt searching across tables is possible. The user asked me to "Start implementing this plan," so I'll focus on the code architecture for a pipeline that runs on local audio segments downloaded from R2.


**Planning the `maya_transcribe` package**

I'm planning to create a `maya_transcribe` Python package that includes a CLI command called `maya-transcribe`. I'll have several commands: one for `download`, which fetches video tar files from R2 by video_id, another for `transcribe`, processing audio segments into JSON outputs, and an `eval` command to compute WER given references.

I’m also thinking about an end-to-end `pipeline` that includes downloading, extracting, transcribing, saving, and optionally uploading to Supabase. I need to ensure we have configuration options for R2 and Supabase settings, using a `.env` file for defaults. The user wants a robust pipeline with a performance mode, so I’ll be implementing the full pipeline, including ensemble and gating features.


**Implementing ASR and LID**

I’m thinking we need to implement at least LID with MMS-LID, and for ASR, we can use Whisper large-v3 (faster-whisper). Maybe we’ll also consider another model like `seamless-m4t`, although that one is quite large, or possibly `openai/whisper-large-v3` alongside `ai4bharat/indicwhisper`.

I should aim for at least two ASR backends in a robust version, utilizing a mix of faster-whisper and a transformers pipeline for HF models. For IndicWhisper, I’ll need to check for a fine-tuned model on HF, but I'd like to avoid massive downloads, implementing it so that downloads happen during execution. For testing, I can run it with a smaller model like `openai/whisper-tiny` for quicker results.


**Configuring performance mode**

The user desires full performance, so I’m thinking we can implement a "performance mode." This would mean using high-quality models by default, but allowing for a `--fast` option to specify smaller models for testing. I’ll set up a configuration option with `quality_mode` or `performance_mode`, ensuring that "full performance mode" prioritizes quality over speed.

I want to handle language-specific decoding for Whisper—specifying language and initial prompts—using the LID output to set the language. Also, I’ll incorporate Unicode normalization and possibly a normalizer for Indic languages, though I need to be cautious about making it too heavy. Regex and `unicodedata` might be useful here.


**Implementing text normalization and ROVER**

I should implement a `text_normalize(text, lang_code)` function that handles Unicode normalization, whitespace stripping, punctuation removal, digit standardization, and more. For ROVER, I’m thinking of creating a simplified version in Python that aligns multiple word sequences using dynamic programming to generate a confusion network (WCN) through progressive alignments.

I’ll store results like agreement scores and quality metrics in JSON records. Since the user wants end-to-end latency measurements, I'll use `time.perf_counter()` for tracking. For parallelization, I’ll possibly use multiprocessing with a worker pool for efficient processing, but need to be cautious about memory management. Finally, I need to package everything with the correct dependencies and choose between `pyproject.toml` or `requirements.txt`.


**Planning dependencies and audio processing**

I'm thinking of implementing MMS-LID, which will require Torch and Transformers. I'll add these as extra dependencies in an optional group, but make sure to include them in the robust version. For audio quality control (QC) and voice activity detection (VAD), I might use `webrtcvad`, which is lightweight. However, for a more robust solution, I'll consider `silero-vad`, since it also requires Torch.

For offline processing, I could implement simple energy-based trimming as an optional feature. Additionally, I might integrate `ffmpeg` for audio conversion and will need to check if it's installed by running `ffmpeg -version`.


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)


Summary: auto


**Setting up the environment**

Okay, I've got ffmpeg installed, which is great! Now, I need to handle tar extraction using Python's tarfile. Starting from scratch, I might set up a Python environment, but that’s not necessary for writing code. For testing, I may need to install packages using pip, but I can create a requirements file first. Since we're in an environment with full access, I can start examining the .env variables, including docker_image for orchestration, and ensure the design supports local or worker runs.


**Designing for performance**

The user mentioned creating a robust version with full performance mode first, which makes sense! We’ll focus on correctness and modular design, optimizing later as needed. They outlined a process: architect, implement, test, and report WERs and end-to-end latencies. I should deliver working code and run the pipeline on sample data to output metrics. If we don’t have audio data locally, we can download it from R2. For testing, we need a video ID and can access objects in the R2 bucket using boto3, requiring the .env credentials.


**Preparing to list bucket keys**

I need to be cautious not to output any secrets while still being able to list object keys, which is fine. I'll write a quick Python snippet to read the .env file and connect to R2 to list the bucket keys. I plan to do this in the shell using `python - <<'PY' ...`. First, I’ll check if boto3 is installed, as it might not be yet. Let’s make sure everything is set up correctly!


python -c "import boto3, dotenv; print('ok')" 2>/dev/null || echo 'missing'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
missing


**Planning project structure**

I’m noticing that I need to set up some dependencies like boto3 or python-dotenv. So, I'm thinking about creating both a `pyproject.toml` and a `requirements.txt` for installing them. Given that the user hasn't asked for packaging yet, I still want to include these for good practice.

I think I'll include a `Makefile` and a `.gitignore` too. Since the current repo only has docs and the .env file, I’ll create a clear plan first. Let's prepare the project structure with a `maya_transcribe/` package and necessary documentation.


Explanation: Implement the master pipeline end-to-end: data fetch (R2/Supabase), preprocessing + LID, ASR ensemble + combination + gating, and an eval/latency harness so we can run a first real sample.


Output: Plan updated


**Configuring env variables**

I need to load the configuration from the .env file for my project. I'll implement a `config.py` in the `maya_transcribe` module to read these environment variables, and use `python-dotenv` to load the .env automatically if it's available. Since the user has a .env with secrets, I’ll ensure to include a `load_dotenv` function. Also, I’m considering adding logging through `structlog`, but I might opt for something simpler. First, I should check the Python version.


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Python 3.10.12


**Planning project dependencies**

I'm setting up a project in Python3 and need to create a code structure. First, I'll make a `pyproject.toml` file for dependencies. I need to decide which ones are required and which are optional. For the core, I plan to include packages like `python-dotenv`, `boto3`, and others. For optional ASR, I might want to add `faster-whisper` and similar packages. It's tricky since heavier dependencies could be overwhelming, but I want to ease the installation for the user while ensuring robustness.


**Implementing installation options**

The user requested a robust version, so I'm thinking it makes sense to include heavier dependencies as optional extras and provide clear installation instructions like `pip install -e .[full]`. I could create extras groups like `r2` for boto3, and `asr` for packages like faster-whisper, torch, and transformers. Although a `requirements.txt` isn't strictly necessary since I have `pyproject.toml`, I might include it for convenience as the user may be running inside a container. And, it seems useful to create a `README.md` with instructions, even if the user didn't ask!


**Designing project architecture**

I'm planning to create a folder for common functions called `maya_transcribe/utils`. Before applying any patches, I need to design the architecture. I'll define several data classes: `SegmentJob` for audio paths and metadata, `LIDResult` for language predictions, `ASRResult` for model info and text, and `PipelineResult` for decisions. I think using Pydantic for JSON serialization is robust, even if it feels a bit excessive. While I'd like to keep it simple, I’ll choose to work with Pydantic v2 to ensure validation, considering it requires Python 3.8 or later. I'll implement the models in `maya_transcribe/schemas.py` and handle JSONL reading/writing in `maya_transcribe/io.py`. Plus, I’ll work on the CLI using Typer too!


**Setting up the project structure**

I'm working on the code for the CLI using Typer. The initial command will handle video downloads, and I’ll need to set up the `maya_transcribe/r2.py` for the R2 client. I might also create a `maya_transcribe/supabase.py` for retrieving metadata, but right now, I just want to scaffold the project. My focus for step one is to create a basic structure with placeholders in the code. I'll establish the necessary directories, including `maya_transcribe/` with various files like `__init__.py`, `cli.py`, and others. Since there aren't any code style guidelines available, I'll proceed to implement a patch to create these files and organize the project neatly.


Input:
*** Begin Patch
*** Add File: pyproject.toml
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "maya-transcribe"
+version = "0.1.0"
+description = "Quality-first offline transcription pipeline (Indic + English)"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+  "boto3>=1.34.0",
+  "jiwer>=3.0.4",
+  "numpy>=1.24.0",
+  "python-dotenv>=1.0.1",
+  "rich>=13.7.1",
+  "typer>=0.12.3",
+]
+
+[project.optional-dependencies]
+supabase = ["supabase>=2.6.0"]
+asr = [
+  "faster-whisper>=1.0.3",
+  "soundfile>=0.12.1",
+]
+torch = [
+  "torch>=2.2.0",
+  "torchaudio>=2.2.0",
+  "transformers>=4.41.0",
+]
+full = [
+  "supabase>=2.6.0",
+  "faster-whisper>=1.0.3",
+  "soundfile>=0.12.1",
+  "torch>=2.2.0",
+  "torchaudio>=2.2.0",
+  "transformers>=4.41.0",
+]
+
+[project.scripts]
+maya-transcribe = "maya_transcribe.cli:app"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["maya_transcribe*"]
+
*** Add File: README.md
+# maya-transcribe
+
+Offline, quality-first transcription pipeline for cleaned single-speaker audio segments (Indic + English).
+
+## Install
+
+```bash
+python3 -m pip install -U pip
+python3 -m pip install -e '.[full]'
+```
+
+If you only need R2 download + evaluation (no ASR), install the base deps:
+
+```bash
+python3 -m pip install -e .
+```
+
+## Configure
+
+Create / update `.env` in the repo root. The CLI auto-loads it.
+
+Minimum for R2:
+- `R2_ENDPOINT_URL`
+- `R2_ACCESS_KEY_ID`
+- `R2_SECRET_ACCESS_KEY`
+- `R2_BUCKET` (defaults to `1-cleaned-data` if unset)
+- `R2_PREFIX` (optional)
+
+## Usage
+
+```bash
+maya-transcribe --help
+maya-transcribe download --video-id <VIDEO_ID> --out data/
+```
+
+More pipeline commands will land in subsequent steps (`transcribe`, `run`, `eval`).
+
*** Add File: .gitignore
+.DS_Store
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.venv/
+venv/
+dist/
+build/
+*.egg-info/
+
+data/
+outputs/
+cache/
+
*** Add File: maya_transcribe/__init__.py
+__all__ = ["__version__"]
+
+__version__ = "0.1.0"
+
*** Add File: maya_transcribe/cli.py
+from __future__ import annotations
+
+from pathlib import Path
+
+import typer
+
+from maya_transcribe.config import Settings
+from maya_transcribe.r2 import R2Client
+from maya_transcribe.tar import safe_extract_tar
+
+app = typer.Typer(add_completion=False, no_args_is_help=True)
+
+
+@app.command()
+def download(
+    video_id: str = typer.Option(..., help="YouTube video id (used to locate the tar in R2)."),
+    out: Path = typer.Option(Path("data"), help="Output directory."),
+    prefix: str | None = typer.Option(None, help="Optional R2 prefix override."),
+    dry_run: bool = typer.Option(False, help="Only show matched key(s); do not download."),
+) -> None:
+    """
+    Download the cleaned/segmented tar for a video from R2 and extract it.
+    """
+    settings = Settings.load()
+    r2 = R2Client.from_settings(settings)
+
+    out.mkdir(parents=True, exist_ok=True)
+    r2_prefix = prefix if prefix is not None else settings.r2_prefix
+
+    matches = r2.find_keys_containing(video_id=video_id, prefix=r2_prefix)
+    if not matches:
+        raise typer.Exit(code=2)
+
+    if dry_run:
+        for key in matches:
+            typer.echo(key)
+        return
+
+    if len(matches) > 1:
+        typer.echo("Multiple matches found; downloading the shortest key (most specific).")
+
+    key = sorted(matches, key=len)[0]
+    tar_path = r2.download_to(key=key, out_dir=out / video_id)
+    extract_dir = out / video_id
+    safe_extract_tar(tar_path, extract_dir=extract_dir)
+    typer.echo(f"Downloaded: {tar_path}")
+    typer.echo(f"Extracted to: {extract_dir}")
+
+
+if __name__ == "__main__":
+    app()
+
*** Add File: maya_transcribe/config.py
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Final
+
+from dotenv import load_dotenv
+
+
+@dataclass(frozen=True)
+class Settings:
+    r2_endpoint_url: str
+    r2_access_key_id: str
+    r2_secret_access_key: str
+    r2_bucket: str = "1-cleaned-data"
+    r2_prefix: str | None = None
+
+    @staticmethod
+    def load(env_path: str | Path = ".env") -> "Settings":
+        load_dotenv(dotenv_path=env_path, override=False)
+
+        import os
+
+        def required(name: str) -> str:
+            value = os.getenv(name)
+            if not value:
+                raise RuntimeError(f"Missing required env var: {name}")
+            return value
+
+        endpoint_url: Final[str] = required("R2_ENDPOINT_URL")
+        access_key_id: Final[str] = required("R2_ACCESS_KEY_ID")
+        secret_access_key: Final[str] = required("R2_SECRET_ACCESS_KEY")
+        bucket: Final[str] = os.getenv("R2_BUCKET") or "1-cleaned-data"
+        prefix = os.getenv("R2_PREFIX") or None
+
+        return Settings(
+            r2_endpoint_url=endpoint_url,
+            r2_access_key_id=access_key_id,
+            r2_secret_access_key=secret_access_key,
+            r2_bucket=bucket,
+            r2_prefix=prefix,
+        )
+
*** Add File: maya_transcribe/r2.py
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+from maya_transcribe.config import Settings
+
+
+@dataclass(frozen=True)
+class R2Client:
+    endpoint_url: str
+    access_key_id: str
+    secret_access_key: str
+    bucket: str
+
+    @staticmethod
+    def from_settings(settings: Settings) -> "R2Client":
+        return R2Client(
+            endpoint_url=settings.r2_endpoint_url,
+            access_key_id=settings.r2_access_key_id,
+            secret_access_key=settings.r2_secret_access_key,
+            bucket=settings.r2_bucket,
+        )
+
+    def _client(self):
+        import boto3
+
+        return boto3.client(
+            "s3",
+            endpoint_url=self.endpoint_url,
+            aws_access_key_id=self.access_key_id,
+            aws_secret_access_key=self.secret_access_key,
+        )
+
+    def find_keys_containing(self, video_id: str, prefix: str | None = None, limit: int = 5000) -> list[str]:
+        s3 = self._client()
+        continuation_token: str | None = None
+        matches: list[str] = []
+
+        while True:
+            kwargs = {"Bucket": self.bucket, "MaxKeys": 1000}
+            if prefix:
+                kwargs["Prefix"] = prefix
+            if continuation_token:
+                kwargs["ContinuationToken"] = continuation_token
+
+            resp = s3.list_objects_v2(**kwargs)
+            for obj in resp.get("Contents", []):
+                key = obj.get("Key")
+                if not key:
+                    continue
+                if video_id in key:
+                    matches.append(key)
+                    if len(matches) >= limit:
+                        return matches
+
+            if not resp.get("IsTruncated"):
+                break
+            continuation_token = resp.get("NextContinuationToken")
+            if not continuation_token:
+                break
+
+        return matches
+
+    def download_to(self, key: str, out_dir: Path) -> Path:
+        out_dir.mkdir(parents=True, exist_ok=True)
+        target = out_dir / Path(key).name
+
+        s3 = self._client()
+        s3.download_file(self.bucket, key, str(target))
+        return target
+
*** Add File: maya_transcribe/tar.py
+from __future__ import annotations
+
+import os
+import tarfile
+from pathlib import Path
+
+
+def safe_extract_tar(tar_path: Path, extract_dir: Path) -> None:
+    extract_dir.mkdir(parents=True, exist_ok=True)
+
+    with tarfile.open(tar_path, "r:*") as tf:
+        members = tf.getmembers()
+        for m in members:
+            member_path = extract_dir / m.name
+            if not _is_within_directory(extract_dir, member_path):
+                raise RuntimeError(f"Refusing to extract outside target dir: {m.name}")
+        tf.extractall(path=extract_dir)
+
+
+def _is_within_directory(directory: Path, target: Path) -> bool:
+    abs_directory = Path(os.path.abspath(directory))
+    abs_target = Path(os.path.abspath(target))
+    try:
+        abs_target.relative_to(abs_directory)
+        return True
+    except ValueError:
+        return False
+
*** End Patch


Output: {"output":"Success. Updated the following files:\nA pyproject.toml\nA README.md\nA .gitignore\nA maya_transcribe/__init__.py\nA maya_transcribe/cli.py\nA maya_transcribe/config.py\nA maya_transcribe/r2.py\nA maya_transcribe/tar.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}