Instruction:
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `never`: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.</permissions instructions>


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <shell>bash</shell>
</environment_context>


plan was [master.md](master.md)

Full Benchmark Analysis Summary

  Run Comparison
  ┌─────────────┬────────────────────┬──────────────────┬───────────────────┐
  │   Metric    │ Baseline (FW only) │ + Vaani (strict) │ + Vaani (relaxed) │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ WER (final) │ 30.6%              │ 23.7%            │ 23.7%             │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Accept rate │ 0/200              │ 11/200           │ 81/200            │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Human rate  │ 200/200            │ 189/200          │ 119/200           │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Oracle best │ 29.7%              │ 9.8%             │ 9.8%              │
  └─────────────┴────────────────────┴──────────────────┴───────────────────┘
  Decision Quality with Relaxed Threshold
  ┌──────────┬───────┬──────────┬───────────────┐
  │ Decision │ Count │ WER Mean │ Disagree Rate │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Accept   │ 81    │ 18.6%    │ 8.5%          │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Human    │ 119   │ 27.1%    │ 13.2%         │
  └──────────┴───────┴──────────┴───────────────┘
  WER by Disagreement Rate Bucket

  - 0-5%: n=35, WER=6.0% (excellent)
  - 5-10%: n=47, WER=19.6%
  - 10-15%: n=68, WER=24.6%
  - 15-25%: n=44, WER=36.9%
  - 25-100%: n=6, WER=50.9%

  Remaining Human Decision Reasons

  - lid_uncertain: 63 (LID confused between Hindi/Urdu/related languages)
  - asr_disagreement: 50 (still above 15% threshold)
  - ctc_ambiguous: 49 (CTC margin too low)

  LID Analysis

  - 82% correctly identified as Hindi
  - 31.5% flagged uncertain due to low margin (Hindi vs Urdu/Gujarati/Bhojpuri)
  - This is expected: MMS-LID struggles with closely related Indic languages

  Final Selection Distribution

  - ROVER: 64.5% (129/200)
  - Vaani Hindi: 34.0% (68/200)
  - FW large-v3: 1.5% (3/200)

  Key Insights

  1. Oracle potential is real: 9.8% WER is achievable with perfect selection
  2. Current selection gap: 13.9pp left on the table due to suboptimal hypothesis selection
  3. Gating works directionally: Accept segments (18.6% WER) are better than Human segments (27.1% WER)
  4. Disagreement correlates with quality: Strong correlation between disagreement rate and WER
  5. LID is the new bottleneck: 31.5% flagged uncertain due to Hindi/Urdu confusion

  Recommendations

  1. Consider LID-Urdu equivalence for Hindi: Treat Hindi and Urdu as equivalent for routing since they share script (when using Devanagari transcription)
  2. Better CTC selection: Current CTC selection isn't capturing the oracle improvement - the selection_gap of 13.9pp suggests the CTC model isn't reliably picking the best hypothesis
  3. Two-stage approach: Accept high-confidence segments (low disagreement), use LLM-based selection for ambiguous cases

✻ Sautéed for 32m 59s

● Background command "Run benchmark with relaxed disagreement threshold" completed (exit code 0).

● I've already analyzed those results above. The benchmark with relaxed threshold (15% disagreement rate) completed successfully:

  Key result: Accept rate improved from 11 → 81 segments (40.5%) while maintaining the same 23.7% WER.

  The analysis shows the pipeline is working - accepted segments have lower WER (18.6%) than segments routed to human review (27.1%). The main remaining bottlenecks are:
  - LID uncertainty (Hindi/Urdu confusion) - 63 segments
  - Disagreement still above threshold - 50 segments
  - CTC ambiguity - 49 segments

  The oracle potential (9.8% WER) is significantly better than what we're achieving (23.7%), indicating room for improved hypothesis selection.

But these WER scores are not what i'm looking for. i need <5% WER scores for all languages consistently. For that to happen, i can run any LLMs like Gemma3n, voxtral.

Can we try these LLMs for audio transcription. can you test and give the performance scores or WER scores when compared to our recent ones above.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning repo inspection**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Agents.md
data
docs
master.md
maya_transcribe
maya_transcribe.egg-info
outputs
Prompt.md
pyproject.toml
README.md
task.md


Summary: auto


**Noticing filename case difference**

The instructions or documentation to add should be in Agents.md, not AGENTS.md, indicating a case-sensitive filename detail to watch when updating.


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 731
Output:
# Agents.md — Meta-Prompt + Workflow for “Reference Ensemble” Research (ASR Transcription Quality First)

## Role
You are a **coding + research agent** operating inside a repo/workspace that contains:
- `task.md` (the original problem statement + requirements)
- `Reference1.md` … `Reference6.md` (different LLM-proposed approaches + code snippets)
- Any supporting code/config files already present

Your job is to **read everything**, extract the best ideas, **reject weak or risky suggestions**, and synthesize a **single optimal plan** to produce **near-zero-error transcripts** for multi-lingual audio (offline; compute not a constraint; quality is).

---

## Non-Negotiables
1. **No invention.** Only claim something is “recommended” if it is supported by the references or is a clearly-labeled inference.
2. **Quality > Coverage.** If forced to choose, reject questionable segments rather than poison training data.
3. **Audio-grounded verification required.** Any transcript must be **verified against the audio** before being accepted.
4. **Make decisions.** For every major technique, explicitly decide: **Adopt / Adapt / Reject / Park** and justify.
5. **Keep this file updated.** This file (`Agents.md`) is the living artifact: top section = this prompt; below it = your analysis, TODOs, decisions, and final plan.

---

## Output Contract (What you must produce in THIS file)
By the end, `Agents.md` must contain:
- A crisp understanding of `task.md` (constraints, metrics, languages, tooling assumptions)
- A per-reference analysis for Reference1–Reference6:
  - Summary, unique contributions, strengths, failure modes, missing pieces
  - Concrete actionable items + what to ignore
  - Adopt/Adapt/Reject/Park decisions
- A unified **Master Plan** (pipeline + thresholds + verification + human loop)
- An implementation blueprint (data structures, job orchestration, evaluation harness)
- A validation protocol (gold set, WER/CER reporting, active learning loop)

---

## Step 0 — Read Order (do this first)
1. Read `task.md` fully.
2. Skim all `Reference*.md` quickly to build a map of what exists.
3. Then deep-read each reference, one by one, and complete the per-reference template below.

---

## Step 1 — Extract Requirements from `task.md`
Fill this section after reading `task.md`.

### Task Summary (from task.md)
- **Goal:**
- **Languages:**
- **Audio segment length range:**
- **Compute constraints:**
- **Quality target (WER/CER):**
- **Acceptance philosophy (precision-first vs recall-first):**
- **Any required tools/frameworks:**
- **Any banned tools/constraints:**

### Success Metrics
Define what “good”” means in numbers and gates.
- Primary: (WER / CER / SER)
- Secondary: rejection rate, human edit rate, alignment confidence, hallucination rate
- Must-meet thresholds:
  - **Max allowable WER/CER for accepted samples:**
  - **Max disagreement allowed between systems:**
  - **Minimum alignment confidence:**

### Assumptions (explicit)
List assumptions you will rely on (and mark what needs verification).
- A1:
- A2:
- A3:

---

## Step 2 — Create an Evaluation Rubric (used for every idea)
For each technique/tool/model suggested in references, score it:

**Scoring dimensions (0–5):**
- Expected quality gain
- Robustness across Indic languages + code-switching
- Verification friendliness (supports confidence/alignment scoring)
- Implementation complexity
- Failure risk (hallucinations, bias, brittle dependencies)

**Decision labels:**
- **Adopt** = use as-is
- **Adapt** = use but modify (thresholds, different model, narrowed scope)
- **Reject** = do not use
- **Park** = keep as optional experiment

---

## Step 3 — Per-Reference Processing Template (repeat for each)
For each `ReferenceN.md`, fill:

### ReferenceN — One-paragraph summary
- What is the approach in plain terms?

### Key Contributions (unique ideas worth preserving)
- C1:
- C2:
- C3:

### Strengths (why it could work well)
- S1:
- S2:

### Risks / Failure Modes
Be harsh and specific.
- R1:
- R2:

### Missing Pieces / Gaps
- G1:
- G2:

### Actionable Extract (convert into implementable tasks)
Each item must be concrete.
- [ ] A1:
- [ ] A2:

### Decisions
For each major suggestion:
- Suggestion: …
  - Decision: Adopt / Adapt / Reject / Park
  - Why:
  - What to do instead (if rejected):

### Integration Notes
How this reference fits with others.
- Conflicts with:
- Complements:
- Dependencies:

---

## Step 4 — Synthesis After All References
After finishing Reference1–Reference6, produce:

### Master Plan (single best pipeline)
Must include:
1. **Pre-QC** (audio sanity checks, VAD trimming, clipping/SNR checks)
2. **Language ID routing** (include handling for uncertain/mixed)
3. **ASR ensemble** (per language group; include at least 2 diverse systems)
4. **Normalization layer** (unicode, punctuation, numerals, script normalization)
5. **System combination** (e.g., voting/ROVER or other)
6. **Audio-grounded verification gate** (forced alignment / CTC scoring / confidence)
7. **Reject + re-decode logic** (different decoding params / alternative model)
8. **Human-in-the-loop for rejects** (only the tail; scalable workflow)
9. **Storage schema** (store all hypotheses + scores for auditability)

### Implementation Blueprint (engineering plan)
- Orchestration: batching, GPU scheduling, retries
- Data structures:
  - `segment_id`, `language_probs`, `hypotheses[]`, `rover_output`, `alignment_scores`, `decision`, `audit_log`
- Reproducibility: config-driven pipeline, versioned models, deterministic decoding where possible
- Observability: dashboards for rejection reasons, per-language error patterns

### Validation Protocol (prove it works)
- Build a **gold set** (stratified across languages, speakers, domains, lengths)
- Compute WER/CER by language + conditions
- Threshold tuning approach
- Active learning loop: rejects → human fix → retrain / recalibrate

### Final Deliverable
- A short “Do This Next” checklist that an engineer can execute immediately.

---

## Global TODO Checklist (tick as you complete)
- [x] Read `task.md` and fill **Step 1**
- [x] Create evaluation rubric (**Step 2**) and decision labels
- [x] Complete Reference1 analysis
- [x] Complete Reference2 analysis
- [x] Complete Reference3 analysis
- [x] Complete Reference4 analysis
- [x] Complete Reference5 analysis
- [x] Complete Reference6 analysis
- [x] Write **Master Plan**
- [x] Write **Implementation Blueprint**
- [x] Write **Validation Protocol**
- [x] Produce final “Do This Next” execution checklist

---

# Analysis Workspace (Fill below as you go)
> Everything below this line is your working output. Keep it clean and structured.

## Step 1 — Task Understanding (from task.md)
### Task Summary (from task.md)
- **Goal:** Produce highly accurate, offline transcriptions for cleaned, single-speaker audio segments so they are safe for training ASR/TTS.
- **Languages:** Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (plus “unknown/unflagged” segments).
- **Audio segment length range:** ~1s up to ~60s max (typical ~30–50s).
- **Compute constraints:** None (can run multiple large models, offline, no latency requirements).
- **Quality target (WER/CER):** “As low as possible” / “can’t afford mistakes” → treat as **precision-first** (accept fewer, near-perfect segments rather than keeping everything with hidden label noise).
- **Acceptance philosophy (precision-first vs recall-first):** **Precision-first**; strict rejection + targeted human correction for the hard tail.
- **Any required tools/frameworks:** Not specified; implied needs: spoken language ID, multi-model ASR, system-combination, and audio-grounded verification (forced alignment / CTC scoring).
- **Any banned tools/constraints:** Not explicit, but the user explicitly worries about **hallucinations** → avoid any step that can invent content (especially text-only LLM “transcription”).

### Success Metrics
Define “good” in numbers and gates (must be tuned on a gold set; initial values are starting heuristics, not guarantees).
- **Primary:** WER by language (space-delimited) + CER by language (robust for Indic scripts), reported on a stratified gold set.
- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).
- **Must-meet thresholds (calibrate):**
  - **Max allowable WER/CER for accepted samples (measured on QA sample):** set a target like “≤2–5% WER” (inference; depends on domain + language) and tune gates until the accepted pool meets it.
  - **Max disagreement allowed between systems:** require strong agreement among at least 2 diverse recognizers (ROVER consensus score / pairwise CER threshold; tune).
  - **Minimum alignment confidence:** require high audio-text fit via alignment/CTC scoring (tune per language + duration).

### Assumptions (explicit)
- **A1:** Segments are already single-speaker, mostly speech-only (music largely removed) and broadly “podcast/explainer” quality.
- **A2:** We can run and store outputs from multiple ASR systems per segment (large GPU RAM / multi-GPU available).
- **A3:** We can afford a human correction workflow for a small reject tail (e.g., 1–10% depending on strictness).

## Step 2 — Evaluation Rubric
Use this rubric to judge every technique/model (0–5 each):
- **Expected quality gain** (WER/CER reduction on gold set)
- **Robustness across Indic + code-switching** (Hindi-English, etc.)
- **Verification friendliness** (exposes confidence/logprobs; supports alignment/scoring)
- **Implementation complexity** (dependencies, infra, licensing)
- **Failure risk** (hallucinations, brittle LID, script errors, silent label noise)

**Decision labels**
- **Adopt:** Use as-is in the master pipeline.
- **Adapt:** Use, but with constraints/thresholds or narrower scope.
- **Reject:** Do not use (too risky / not verifiable / not supported).
- **Park:** Optional experiment after the core pipeline is working.

---

## Reference1 — Analysis
### Reference1 — One-paragraph summary
Proposes a best-practice offline pipeline: spoken language ID → diverse ASR ensemble per language → system combination (ROVER) → independent audio-grounded verification (forced alignment / CTC scoring) → strict rejection + small human correction tail; LLMs only as constrained tie-breakers.

### Key Contributions (unique ideas worth preserving)
- C1: **MMS LID (+ VoxLingua107)** with a clear confidence/margin routing rule.
- C2: **Explicit system-combination (ROVER)** + “disagreement map” as a debug artifact.
- C3: **Independent verification gate** via forced alignment or **CTC “fit” scoring** (don’t trust the generating model).

### Strengths (why it could work well)
- S1: Strong alignment with “quality > coverage” and “audio-grounded verification required.”
- S2: Uses **error diversity** (Indic specialist vs Whisper generalist vs MMS) to make ensemble voting useful.

### Risks / Failure Modes
- R1: MFA forced alignment across 12+ languages may be operationally painful (lexicons/acoustic models per language).
- R2: Voxtral claims for non-Hindi Indic languages are explicitly “test-and-measure”; avoid assuming it’s best everywhere.

### Missing Pieces / Gaps
- G1: No concrete thresholding protocol (needs gold-set tuning plan).
- G2: No implementation details for storing hypotheses/scores and orchestration (but it hints at them).

### Actionable Extract (convert into implementable tasks)
- [ ] Implement LID with MMS-LID (+ optional VoxLingua107) and a “confident vs unknown/mixed” routing rule.
- [ ] Run a 3-model ASR ensemble per segment (Indic specialist + Whisper + diversity model).
- [ ] Normalize transcripts and combine via ROVER (or equivalent confusion-network voting).
- [ ] Add an audio-grounded verification gate (CTC scoring first; MFA where feasible).
- [ ] Add a strict reject → human correction loop (e.g., Shoonya).

### Decisions
- Suggestion: MMS LID (+ VoxLingua107) and strict routing thresholds
  - Decision: **Adopt (with tuning)**
  - Why: Spoken LID routing is foundational; thresholds are sensible starting points but must be calibrated.
- Suggestion: IndicConformer + Whisper large-v3 (+ MMS ASR) ensemble
  - Decision: **Adopt / Adapt**
  - Why: Core diversity is strong; we’ll also incorporate **IndicWhisper** (from other refs) as an additional Indic specialist.
- Suggestion: ROVER system combination
  - Decision: **Adopt**
  - Why: Clear, well-known method to reduce WER when recognizers have diverse errors.
- Suggestion: Verification via CTC scoring / forced alignment
  - Decision: **Adopt**
  - Why: Directly enforces “audio-grounded verification required”; critical to control hallucinations.
- Suggestion: Text-only LLMs for transcription
  - Decision: **Reject**
  - Why: Not audio-grounded; high hallucination risk. Use only constrained tie-breaks.

### Integration Notes
- Complements: Ref5’s concrete 4-stage pipeline; Ref4’s LLM-as-judge constraints; Ref3/6’s emphasis on strict gating.
- Potential conflict: Ref6’s incorrect dismissal of Voxtral; resolve by “measure, don’t assume.”

---

## Reference2 — Analysis
### Reference2 — One-paragraph summary
Advocates multi-model ensembling (Whisper/IndicWhisper, Voxtral, Gemma, etc.), ROVER-style consensus, plus an LLM-based correction stage and selective human review to reach near-human WER.

### Key Contributions (unique ideas worth preserving)
- C1: Emphasizes **language-specific fine-tunes** (IndicWhisper; Hinglish variants) outperforming generic Whisper for Indic.
- C2: Reinforces **ROVER + multi-candidate comparison** as the “clever” offline strategy.
- C3: Highlights the importance of a **human layer for the tail** and iterative improvement.

### Strengths (why it could work well)
- S1: Correctly treats “no compute constraints” as permission to ensemble broadly.
- S2: Explicitly calls out hallucination risk and the need for constraints when using LLMs.

### Risks / Failure Modes
- R1: Promotes “LLM correction” somewhat broadly; unconstrained “fixing” can invent content.
- R2: Mentions several models (e.g., Gemma multimodal ASR claims) without a verification/gating specification; risk of adopting untested components.
- R3: Contains truncated/incomplete sections (“…chars truncated…”), so details may be missing.

### Missing Pieces / Gaps
- G1: No audio-grounded verification gate definition (forced alignment/CTC scoring is not central here).
- G2: No concrete data schema + orchestration plan.

### Actionable Extract (convert into implementable tasks)
- [ ] Use IndicWhisper (language-specific) as a primary Indic recognizer in the ensemble.
- [ ] Treat Voxtral/Gemma/etc as **“test-and-measure”** optional candidates; evaluate on a gold set before relying on them.
- [ ] If using an LLM, constrain it to selecting among provided hypotheses/spans (no free generation).

### Decisions
- Suggestion: IndicWhisper as a key model
  - Decision: **Adopt**
  - Why: Multiple references converge on “IndicWhisper fine-tune beats generic Whisper for Indic.”
- Suggestion: Broad LLM post-editing to “improve” transcripts
  - Decision: **Adapt → constrain heavily**
  - Why: Useful only as a tie-breaker among hypotheses; otherwise hallucination risk.
- Suggestion: Gemma as a prompted transcriber
  - Decision: **Park**
  - Why: Not supported elsewhere; treat as optional experiment after core pipeline is validated.

### Integration Notes
- Complements: Ref1’s ensemble + ROVER framing.
- Conflicts with: Precision-first “no invention” policy if LLM correction is unconstrained.

---

## Reference3 — Analysis
### Reference3 — One-paragraph summary
Proposes a “Consensus & Arbitration” pipeline: triple-ensemble LID; a “council” of 3–4 ASR models (IndicWhisper, Whisper, Canary/Parakeet, MMS); forced alignment to pick the most acoustically faithful transcript; optional LLM arbitration; plus audio quality scoring for TTS readiness.

### Key Contributions (unique ideas worth preserving)
- C1: **Hardline LID gating**: disagreement → quarantine/discard to prevent dataset poisoning.
- C2: **Acoustic truth filter** framing: choose among candidates using alignment likelihood.
- C3: Adds **audio quality scoring** (beyond transcription accuracy) for TTS training.

### Strengths (why it could work well)
- S1: Strong alignment with “quality > coverage” and strict acceptance.
- S2: Separation between generation (ASR models) and validation (alignment) is correct.

### Risks / Failure Modes
- R1: Claims about specific leaderboards/SOTA status (Canary/Parakeet) aren’t supported within the references.
- R2: “LLM selects best transcript” is risky unless constrained; may introduce invented tokens.
- R3: External “deep dive” link is not actionable for this repo plan.

### Missing Pieces / Gaps
- G1: No concrete combination method (ROVER specifics) beyond mentioning it.
- G2: No language/script normalization plan.

### Actionable Extract (convert into implementable tasks)
- [ ] Add strict LID gating logic (agreement required; otherwise flag/reject).
- [ ] Implement forced alignment / alignment scoring to pick among hypotheses.
- [ ] Add audio-quality scoring (SNR/clipping/reverb proxies) to protect TTS quality.

### Decisions
- Suggestion: 3-model LID ensemble and quarantine on disagreement
  - Decision: **Adapt**
  - Why: Good for precision; but make thresholds configurable so we don’t over-reject short segments; include an “unknown/mixed” route.
- Suggestion: Add NVIDIA Canary/Parakeet as a challenger ASR
  - Decision: **Park**
  - Why: Could add diversity, but requires validation + integration work; not required for a strong first version.
- Suggestion: LLM arbitration
  - Decision: **Adapt → constrained tie-break only**
  - Why: Must not generate new text; must be downstream of audio-grounded scoring.
- Suggestion: Audio quality scoring for TTS
  - Decision: **Adopt**
  - Why: Even perfect transcripts don’t rescue low-SNR/reverberant speech for TTS.

### Integration Notes
- Complements: Ref1’s verification gate; Ref5’s preprocessing + confidence filtering.

---

## Reference4 — Analysis
### Reference4 — One-paragraph summary
Defines a multi-stage “ensemble + consensus + verification” architecture with an explicit LLM-as-judge prompt (selection-only), disagreement detection, and a composite quality score that combines agreement, confidence, LM plausibility, and audio quality.

### Key Contributions (unique ideas worth preserving)
- C1: **Centroid transcript** idea (lowest average distance to others) as a simple “consensus” heuristic.
- C2: **LLM-as-judge prompt template** that forbids generation and supports a LOW-confidence flag.
- C3: Suggests **character-level voting** and **forced alignment post-processing** for verification.

### Strengths (why it could work well)
- S1: Explicitly frames LLM as a judge, not a generator (hallucination-aware).
- S2: Provides practical engineering logic for disagreement routing and composite scoring.

### Risks / Failure Modes
- R1: Includes many models, including proprietary ones (Gemini/USM), which may not be feasible or consistent with “offline” assumptions.
- R2: Perplexity/LM-based plausibility can be misleading for rare names, code-switching, and domain terms unless treated as a weak signal.
- R3: Mentions NLLB-LID (primarily text LID) in an audio LID context; likely not directly applicable.

### Missing Pieces / Gaps
- G1: No specific, audio-grounded verification metric definition (alignment/CTC score thresholds).
- G2: No plan for storing model outputs and auditability beyond a sketch.

### Actionable Extract (convert into implementable tasks)
- [ ] Imple…432 tokens truncated…ength checks to prevent “over-correction.”
- C3: **IndicNLP normalization** callout (preserving diacritics / script correctness).

### Strengths (why it could work well)
- S1: Practical and actionable; aligns with precision-first philosophy.
- S2: Builds in hallucination safeguards (only touch low-confidence cases; reject large edits).

### Risks / Failure Modes
- R1: Some performance numbers and claims are presented without provenance inside the repo (treat as directional, not guarantees).
- R2: LLM/correction stage still needs an audio-grounded verification gate; text-only checks aren’t enough to “prove” correctness.
- R3: Some model mentions appear speculative/unclear (e.g., “Omnilingual ASR 2025”); validate before adoption.

### Missing Pieces / Gaps
- G1: Verification method is listed but not fully specified (how to compute/threshold alignment confidence).
- G2: Orchestration and storage schema details aren’t spelled out.

### Actionable Extract (convert into implementable tasks)
- [ ] Implement preprocessing: 16kHz mono, VAD trimming, basic audio QC.
- [ ] Implement LID: MMS-LID + thresholds.
- [ ] Implement ASR ensemble: IndicWhisper + Whisper large-v3 + MMS.
- [ ] Implement ROVER combination and store full hypothesis set.
- [ ] Add constrained correction only for low-confidence cases; enforce edit distance + length ratio constraints.

### Decisions
- Suggestion: Silero VAD + basic audio normalization
  - Decision: **Adopt**
  - Why: Cheap win; improves both LID and ASR stability.
- Suggestion: IndicWhisper + Whisper + MMS as “core ensemble”
  - Decision: **Adopt**
  - Why: Strong diversity and consistent with other refs.
- Suggestion: “LLM-based correction” stage
  - Decision: **Adapt**
  - Why: Only as constrained tie-break among options and always downstream of audio-grounded scoring.
- Suggestion: IndicNLP normalization
  - Decision: **Adopt**
  - Why: Normalization is required for system combination and consistent training text.

### Integration Notes
- Complements: Ref1’s verification gate; Ref4’s LLM judge prompt discipline.

---

## Reference6 — Analysis
### Reference6 — One-paragraph summary
Proposes a consensus MoE pipeline: triple LID check, 3-ASR “council,” extract N-best hypotheses, use an Indic LLM (Sarvam-1) to pick the lowest-perplexity option, and then enforce correctness with forced alignment (discard segments with poor alignment).

### Key Contributions (unique ideas worth preserving)
- C1: Emphasizes **N-best hypotheses**, not just 1-best transcripts (useful for later rescoring/ROVER).
- C2: Reinforces **forced alignment as a “hallucination killer”** and strict discard logic.

### Strengths (why it could work well)
- S1: Clear insistence on verification and aggressive filtering.
- S2: N-best extraction is a concrete engineering improvement for offline pipelines.

### Risks / Failure Modes
- R1: Uses **LLM perplexity as the main selector**, which is not audio-grounded and can prefer fluent-but-wrong text (names, code-switching, domain terms).
- R2: Contains an incorrect/contradictory statement about Voxtral’s existence vs Ref1/2 (treat as unreliable).
- R3: “Drop misaligned words” can silently delete content; for training data, it may be safer to reject the segment than to produce partial/edited transcripts unless you can verify the edit is correct.

### Missing Pieces / Gaps
- G1: No system combination method (ROVER/confusion network) despite multiple ASR sources.
- G2: No script normalization plan.

### Actionable Extract (convert into implementable tasks)
- [ ] Extract N-best hypotheses (top-k beams) where supported (Whisper/Conformer/MMS).
- [ ] Use forced alignment/CTC scoring as a hard gate; discard segments that fail.

### Decisions
- Suggestion: N-best extraction
  - Decision: **Adopt**
  - Why: Enables better combination/verification and reduces reliance on any single decode.
- Suggestion: Perplexity-based selection with Sarvam-1
  - Decision: **Park**
  - Why: Could be an auxiliary feature, but not acceptable as a primary correctness signal without audio grounding.
- Suggestion: Aggressive forced alignment discard thresholds
  - Decision: **Adapt**
  - Why: Keep the “discard on poor fit” principle, but tune thresholds on a gold set and prefer “reject segment” over “silently drop words.”

### Integration Notes
- Complements: Ref1/5’s ensemble + verification pipeline.
- Conflicts with: Ref1’s caution that text-only LLMs shouldn’t be used as generators/selectors without audio gating.

---

## Master Plan (Unified)
Goal: **near-zero-error accepted transcripts** via (1) diverse generation and (2) independent, audio-grounded verification; everything uncertain is rejected and routed to human correction.

### 1) Pre-QC (audio sanity checks)
- Enforce canonical format: 16kHz, mono, PCM/FLAC/WAV (adapt as needed).
- Run VAD trim (e.g., Silero VAD) to hard-trim leading/trailing silence and remove non-speech tails.
- Compute audio QC metrics and reject early if bad (precision-first): clipping %, extreme loudness, very low speech ratio, suspiciously short effective speech, etc. (Inference: exact metrics/thresholds must be tuned.)

### 2) Language ID routing (handling unknown/mixed)
- Run spoken LID on every segment:
  - Primary: **MMS-LID** (Refs 1/5).
  - Optional second opinion: **SpeechBrain VoxLingua107 ECAPA** (Refs 1/5) and/or Whisper LID (Refs 3/6) as a tie-breaker.
- Routing (starting heuristic from Ref1; tune on gold set):
  - If top-1 prob ≥ 0.90 and margin(top1-top2) ≥ 0.20 → **confident route**
  - Else → **unknown/mixed route** (process with multilingual ensemble and stricter verification).

### 3) ASR ensemble (at least 2–3 diverse systems)
**Core idea (Refs 1/5):** combine an Indic specialist + a strong multilingual generalist + a diversity model.
- **Indic segments (the 11 Indic languages):**
  - **IndicWhisper** (AI4Bharat fine-tuned Whisper) (Refs 3/5, also implied in Ref2/4).
  - **Whisper large-v3** as generalist anchor (Refs 1/3/5).
  - **IndicConformer-600M multilingual** (hybrid CTC/RNNT; diversity in failure modes) (Refs 1/4/6).
  - Optional diversity: **MMS ASR** (Refs 1/3/5) especially for lower-resource (Odia/Assamese) or when others disagree.
- **English segments:**
  - **Whisper large-v3** + **Voxtral** (Refs 1/2) (and optionally a third English ASR for diversity if needed).
- **Unknown/mixed segments:**
  - Run a multilingual-heavy set: Whisper large-v3 + IndicConformer multilingual + MMS (+ Voxtral if available) and later decide based on verification.

Decoding policy (Ref1):
- Start with deterministic/high-precision decoding (beam search, temperature=0 for Whisper-like models).
- If decoding fails/degenerates, re-decode with alternate params; treat fallbacks as lower-confidence and require stricter verification.
- Where supported, store **N-best hypotheses** (Ref6) not only 1-best.

### 4) Normalization layer (mandatory before combination/eval)
- Unicode normalize; normalize punctuation/numerals consistently; standardize whitespace.
- Use **IndicNLP** (Ref5) or equivalent for Indic script normalization so system combination isn’t dominated by trivial differences.

### 5) System combination (don’t just pick one)
- Use **ROVER / confusion-network voting** (Refs 1/4/5) to produce:
  - `combined_transcript`
  - a per-token **consensus/conflict map**
- Consider character-level voting for Indic scripts as an adaptation (Ref4) if word tokenization is unreliable.

### 6) Audio-grounded verification gate (non-negotiable)
Use at least one **independent** audio-grounded check before accepting a transcript.
- **Primary gate:** **CTC-based “fit” scoring** (Ref1) on:
  - top candidates (best per-model 1-best and/or N-best)
  - the ROVER-combined transcript
  - Choose the best-scoring candidate and compute pass/fail metrics (coverage, mean posterior, confidence dips).
- **Secondary/optional gate:** forced alignment (MFA / NeMo aligner) (Refs 1/3/6) where language resources support it cleanly.
- **Reject policy:** if alignment/CTC fit is poor, **reject the whole segment** rather than “editing” words out (precision-first; avoid silent corruption).

### 7) LLM usage (only if constrained, never as free generator)
- Only use an LLM as a **selection/tie-breaker among existing hypotheses/spans** (Refs 1/4/5).
- The LLM output is still provisional until it passes the same audio-grounded verification gate.
- Prefer audio-native models (e.g., Voxtral) over text-only models when possible (Ref1), but still enforce verification.

### 8) Reject + re-decode logic
- If a segment fails the gate:
  - Try an alternate decode (different beam/LM, add a diversity model like MMS).
  - If still failing → send to human queue (don’t “force accept”).

### 9) Human-in-the-loop for rejects
- Only humans touch the reject tail (Refs 1/2/5).
- Use an annotation UI that shows: audio + top hypotheses + disagreement highlights (Ref1) and lets the annotator pick/edit.
- If you want OSS tooling: **Shoonya** (AI4Bharat) is suggested in Ref1.

### 10) Storage schema (auditability)
Store everything needed to debug and to re-run deterministically:
- `segment_id`, source metadata, duration, audio QC metrics
- `language_probs[]` (per LID model)
- `hypotheses[]` (model, decode params, 1-best + N-best, confidences/logprobs if available)
- `normalized_hypotheses[]`
- `rover_output` + `disagreement_map`
- `verification_scores[]` (CTC/aligner metrics)
- `decision` (accept/reject/human) + `reason_codes` + timestamps

### Block Diagram (pipeline)
```text
┌───────────────────────────────┐
│          Audio segment         │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│        Pre-QC + VAD trim       │
│   (format normalize + sanity)  │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│        Spoken LID ensemble     │
│  MMS-LID (+ optional 2nd vote) │
└───────────────┬───────────────┘
                │
      ┌─────────┴──────────┐
      │                    │
      v                    v
┌───────────────────────┐  ┌────────────────────────┐
│ Confident language     │  │ Uncertain / mixed       │
│ route                  │  │ route                   │
└───────────┬───────────┘  └───────────┬────────────┘
            │                          │
            v                          v
┌───────────────────────┐  ┌────────────────────────┐
│ ASR ensemble (per-L)   │  │ ASR ensemble (multi)    │
│ IndicWhisper           │  │ Whisper v3              │
│ + Whisper large-v3     │  │ + IndicConformer        │
│ + IndicConformer       │  │ + MMS (+ Voxtral opt.)  │
│ (+ MMS optional)       │  │                        │
└───────────┬───────────┘  └───────────┬────────────┘
            │                          │
            └───────────┬──────────────┘
                        │
                        v
┌───────────────────────────────┐
│     Normalize all hypotheses   │
│ (Unicode + IndicNLP + numbers) │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│  System combination (ROVER)    │
│   + disagreement map (audit)   │
└───────────────┬───────────────┘
                │
                v
┌───────────────────────────────┐
│   Audio-grounded verification  │
│  CTC “fit” scoring (+ aligner) │
└───────┬───────────────┬───────┘
        │ PASS          │ FAIL
        v               v
┌───────────────────┐   ┌──────────────────────────────┐
│ ACCEPT            │   │ Re-decode / add model / retry │
│ transcript+scores │   │ (then repeat from ASR stage)  │
└───────────────────┘   └───────────────┬──────────────┘
                                        │ still FAIL after retries
                                        v
                              ┌──────────────────────────┐
                              │ HUMAN REVIEW (tail only)  │
                              │ correct → ACCEPT          │
                              │ discard → DROP            │
                              └──────────────────────────┘
```

### Why this plan drives WER down (offline, no compute constraints)
- **Stops routing errors early:** Spoken LID prevents “wrong-language/wrong-script” decodes that create catastrophic transcripts (Refs 1/5).
- **Exploits error diversity:** Different ASR architectures fail differently; running multiple strong but diverse recognizers increases the chance the correct word sequence exists among hypotheses (Refs 1/3/5/6).
- **Combines, not guesses:** ROVER/confusion-network voting reduces WER when recognizers disagree in different places; consensus plus a disagreement map makes errors visible and auditable (Refs 1/4/5).
- **Verifies against the audio:** Independent CTC/forced-alignment scoring rejects fluent-but-wrong or hallucinated text (the main failure mode you can’t tolerate for training) (Refs 1/3/6).
- **Precision-first acceptance:** When you “can’t afford mistakes,” the optimal strategy is to reject uncertainty and only keep segments that clear strict audio-grounded gates; the tail goes to humans, yielding a near-zero-error accepted set (Refs 1/2/5).

### Execution Plan (phased)
1. **Gold set + harness:** Build a stratified human-transcribed gold set; wire WER/CER reporting by language and duration bucket.
2. **Baseline pipeline:** Implement `VAD/QC → MMS-LID → (IndicWhisper + Whisper v3 + IndicConformer [+ MMS]) → normalize → ROVER → CTC-fit gate → accept/reject`.
3. **Calibrate:** Tune LID thresholds, ROVER weights, and verification gates on the gold set to maximize precision of accepted segments.
4. **Human loop:** Add a review queue/UI for rejects that shows audio + top hypotheses + disagreement spans; feed corrections back into threshold tuning (and optional fine-tunes later).
5. **Scale + monitor:** Batch by language, parallelize per-model inference, and track reject reasons and residual error via spot-check QA sampling.

## Implementation Blueprint
### Engineering scope (minimal first, then scale)
**Phase A (strong baseline):** LID + 3-model ASR ensemble + normalization + ROVER + CTC scoring gate + reject queue.
**Phase B (quality hardening):** add N-best storage, better disagreement analytics, forced alignment where feasible, and a constrained LLM tie-breaker only for narrow cases.
**Phase C (operations):** human review UI integration, dashboards, and active learning loop.

### Orchestration
- Batch segments by language route to maximize GPU throughput (Inference).
- Run models in parallel across available GPUs; treat each model decode as an independent job with retries.
- Make everything config-driven (`models`, `decode_params`, `thresholds`, `weights`), and record config snapshots with outputs.

### Data structures (suggested)
- `Segment`: `{segment_id, source_id, path, duration_s, qc_metrics}`
- `LidResult`: `{model, probs: {lang: p}, top1, top2, margin}`
- `Hypothesis`: `{model, lang, text, nbest[], token_logprobs?, avg_logprob?, decode_params}`
- `CombineResult`: `{method: rover, text, consensus_score, disagreement_spans[]}`
- `VerifyResult`: `{method: ctc_fit|forced_align, score, coverage, pass, diagnostics}`
- `Decision`: `{status: accept|reject|human, reason_codes[], chosen_text}`

### Model integration notes (practical)
- Prefer models that can expose confidence/logprob signals; if not, lean more on independent CTC scoring + agreement.
- Keep model versions pinned (Hugging Face commit hash / NeMo checkpoint version) for reproducibility.
- Persist raw hypotheses + scores; never overwrite outputs in-place without versioning.

### System combination (ROVER)
- Use an existing implementation/toolkit where possible (Ref1/5 mention NIST SCTK; Ref3 suggests a simple script).
- Maintain per-language model weights learned on a dev set (Inference; implied by Ref1/4/5).

### Observability
- Aggregate metrics by language, duration bucket, and source video.
- Track top reject reasons (low LID confidence, high disagreement, low alignment score, audio QC failure).
- Sample a fixed % of “accepted” for spot-check human QA to estimate residual label noise (Inference).

## Validation Protocol
### Build a gold set (required before tuning gates)
- Stratified sample across:
  - all languages
  - duration buckets (1–3s, 3–10s, 10–30s, 30–60s)
  - audio conditions (clean vs mildly noisy)
  - code-switching presence (at least Hindi-English; if present in your data)
- Human-transcribe this gold set with strict guidelines (verbatim, include disfluencies if desired for training consistency).

### Evaluate components and the full pipeline
- Score each base ASR model (WER/CER) by language and condition.
- Score ensemble variants:
  - best-single-model
  - ROVER combination (with/without MMS/IndicConformer)
  - verification gate impact (precision/recall tradeoff vs gold)

### Tune thresholds and weights
- Tune:
  - LID “confident route” thresholds
  - model weights for ROVER per language
  - verification gate thresholds (CTC fit / alignment pass)
- Objective: maximize **precision** of accepted set subject to a feasible accept rate (precision-first).

### Active learning loop
- Collect rejects + hard disagreements as an “error discovery” pool.
- Human-correct a subset, then:
  - recalibrate thresholds/weights
  - optionally fine-tune the best-performing model(s) for your domain (Inference; consistent with Ref2’s iterative improvement idea)

## Do This Next (Execution Checklist)
1. Create a stratified gold set (per language + durations) and produce human transcripts.
2. Stand up the baseline pipeline: `MMS-LID → (IndicWhisper + Whisper large-v3 + IndicConformer [+ MMS]) → normalize → ROVER → CTC fit gate → accept/reject`.
3. Store *all* hypotheses + scores in an auditable schema (JSONL/Parquet) and build simple per-language dashboards.
4. Tune routing/verification thresholds on the gold set until the accepted pool hits your precision target.
5. Add a human review workflow for rejects (UI + queue); feed corrected tail back into evaluation and (optional) fine-tuning.


**Planning inspection for audio transcription**

I'm preparing to explore the repo for scripts, models, and benchmarks related to audio transcription, focusing on README and task files to understand the pipeline and target WER.


rg -n "voxtral|gemma|llm|mms|whisper|vaani" -S .


Output:
Exit code: 0
Wall time: 0.2 seconds
Total output lines: 9657
Output:
./maya_transcribe.egg-info/PKG-INFO:16:Requires-Dist: faster-whisper>=1.0.3; extra == "asr"
./maya_transcribe.egg-info/PKG-INFO:24:Requires-Dist: faster-whisper>=1.0.3; extra == "full"
./pyproject.toml:29:  "faster-whisper>=1.0.3",
./pyproject.toml:47:  "faster-whisper>=1.0.3",
./maya_transcribe.egg-info/requires.txt:9:faster-whisper>=1.0.3
./maya_transcribe.egg-info/requires.txt:14:faster-whisper>=1.0.3
./task.md:9:the unknown ones are not flagged, i might try to get more metadata for them and try flagging them as per the title and description for language. but i was wondering if we have a way to identify the language as well in our pipeline. Now, assuming the situation here, for indian space and languages mentioned above, and most of these are high quality youtube podcasts and or are explanation videos that has minimal to low and are cleaned out of music. Given the situation and since we are anyway generating the transcripts offline, like it doesnt have to be real-time or latency, or quick, given no compute constraints, what's like the best way or the cleverest way possible to generate transcripts to these segments so that i get WER as minimal as possible. i can pass these segments to as many models as possible to compare them all and or get them checked by a superiour LLMs or are there any LLM specialized in indic, where i can maybe run these segments and they create transcripts like A/B testing, what's that cleverest way. when i said i have no constraints, i literally meant we can run any specialized indic llms that can help create transcriptions with prompt guidence like gemma, or anything else, but we need to strategically plan this to avoid hallucinations or creation of content rather transcribing the exiting ones. that doesnt mean we can't use LLM, we need to be clever here. what's that combination we can use like voxtral+whisper large ? or whisper + indic conformers or indicwhisper or what's that best combo we can use to get WER as minimal as possible, we just need to be smart enough and clever here since we are doing this offline and or have no constraints over compute or anything and given we have enough time and everything, we just can't compromise on quality or WER for these transcriptions, what's the go-to strategy that we can adapt to where we just can't afford to make mistakes and have high quality transcriptions to these audio segments so we can train the best in-class indic ASR and TTS models.
./README.md:34:maya-transcribe run --segments-dir data/<VIDEO_ID>/**/segments --out outputs/<VIDEO_ID> --whisper-model large-v3
./README.md:44:- Add additional Whisper-family checkpoints: `--transformers-whisper-model <hf_model_id>` (repeatable)
./outputs/test_run5/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391afc89b", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run5/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "src_ffprobe": {"path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "duration_sec": 12.11625, "sample_rate": 48000, "channels": 1}, "ffprobe": {"path": "outputs/test_run5/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "eng", "prob": 0.999092698097229}, {"code": "spa", "prob": 0.00018186250235885382}, {"code": "haw", "prob": 6.795905937906355e-05}, {"code": "lat", "prob": 6.20875071035698e-05}, {"code": "war", "prob": 3.527590160956606e-05}]}, "route": {"language_hint": "eng"}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "ensemble": {"unique_models": ["faster-whisper:tiny"], "unique_backends": ["faster-whisper"], "warnings": ["low_model_diversity"]}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "ctc": null, "final": {"text": "intricate web of", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 1.432625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "eng", "top1_prob": 0.999092698097229, "margin": 0.9989108355948702, "confident": true}, "no_speech_prob_max": 0.1619873046875, "heuristics_asr_key": "faster-whisper:tiny|det", "avg_logprob_mean": -0.5895182490348816, "compression_ratio_mean": 0.6666666666666666, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.19274936662986875, "lid_sec": 0.4290622090920806, "asr_sec": 0.784179609734565, "normalize_sec": 5.9102196246385574e-05, "rover_sec": 9.4579067081213e-05, "ctc_sec": 2.398155629634857e-07, "verify_sec": 7.834797725081444e-05, "total_sec": 1.406317532993853}}
./docs/Reference1.md:13:* **Meta MMS LID (up to 4017 languages)** is designed for spoken LID at scale. ([Hugging Face][1])
./docs/Reference1.md:29:AI4Bharat explicitly maintains state-of-the-art ASR work for India and lists **IndicConformer / IndicWhisper / IndicWav2Vec** as their ASR model line. ([ai4bharat.iitm.ac.in][3])
./docs/Reference1.md:35:* Model B: **Whisper large-v3** as a strong generalist baseline ([Hugging Face][5])
./docs/Reference1.md:36:* Model C (optional but valuable): **Meta MMS ASR** multilingual (1107 languages) for diversity ([Hugging Face][1])
./docs/Reference1.md:40:* **Voxtral** (open-weight speech understanding) reports better transcription WER than Whisper large-v3 on their published benchmarks and supports automatic language detection. ([Mistral AI][6])
./docs/Reference1.md:45:* Voxtral + Whisper large-v3 (+ Nemotron Speech as extra diversity)
./docs/Reference1.md:47:> Note: Voxtral’s public claims highlight multilingual performance and explicitly mention Hindi among supported major languages; for Tamil/Telugu/etc you should treat Voxtral as “test-and-measure” rather than assumed best. ([Mistral AI][6])
./docs/Reference1.md:67:This yields a measurable win when recognizers have different failure modes (IndicConformer vs Whisper vs MMS usually do).
./docs/Reference1.md:94:WhisperX-style pipelines are built around ASR + forced alignment; it’s a known pattern to use wav2vec2-family aligners for word/segment timing. ([arXiv][10])
./docs/Reference1.md:114:* Whisper’s own transcription code supports fallback logic using thresholds like `compression_ratio_threshold`, `logprob_threshold`, and `no_speech_threshold`. ([GitHub][12])
./docs/Reference1.md:115:* There’s also active research specifically on reducing Whisper hallucinations on non-speech (e.g., Calm-Whisper). ([arXiv][13])
./docs/Reference1.md:120:* For Whisper-like models, run **beam search with temperature=0** first; only fall back to higher temperature if decoding failed (and if you do, treat as low confidence and require alignment to pass)
./docs/Reference1.md:124:## 6) How to use LLMs “safely” (no invented words)
./docs/Reference1.md:126:Text-only LLMs (Gemma, etc.) should **not** be used to “transcribe” from scratch; they will invent plausible text. The safe uses are:
./docs/Reference1.md:130:Give the LLM:
./docs/Reference1.md:135:  This turns the LLM into a **tie-breaker**, not a generator.
./docs/Reference1.md:137:### B) Audio-LLMs that actually listen
./docs/Reference1.md:139:If you want “LLM help” that’s still audio-grounded, use models that take audio as input (e.g., Voxtral). ([Mistral AI][6])
./docs/Reference1.md:159:* MMS LID-4017 (+ optional SpeechBrain VoxLingua107)
./docs/Reference1.md:165:  * Whisper large-v3 ([Hugging Face][5])
./docs/Reference1.md:166:  * MMS ASR (optional diversity) ([Hugging Face][1])
./docs/Reference1.md:167:* English: Voxtral + Whisper large-v3 (+ Nemotron Speech as extra) ([Mistral AI][6])
./docs/Reference1.md:178:**LLM**
./docs/Reference1.md:180:* Only constrained tie-breaks (choose among options) OR audio-LLM transcription + still gated ([Mistral AI][6])
./docs/Reference1.md:189:  -> LID (MMS LID + optional SpeechBrain)  -> language / unknown
./docs/Reference1.md:190:  -> ASR ensemble (IndicConformer + Whisper + optional MMS / Voxtral)
./docs/Reference1.md:209:[1]: https://huggingface.co/docs/transformers/en/model_doc/mms?utm_source=chatgpt.com "MMS"
./docs/Reference1.md:213:[5]: https://huggingface.co/openai/whisper-large-v3?utm_source=chatgpt.com "openai/whisper-large-v3"
./docs/Reference1.md:214:[6]: https://mistral.ai/news/voxtral "Voxtral | Mistral AI"
./docs/Reference1.md:220:[12]: https://github.com/openai/whisper/blob/main/whisper/transcribe.py?utm_source=chatgpt.com "transcribe.py - openai/whisper"
./docs/Reference1.md:221:[13]: https://arxiv.org/abs/2505.12969?utm_source=chatgpt.com "Calm-Whisper: Reduce Whisper Hallucination On Non-Speech By Calming Crazy Heads Down"
./maya_transcribe/asr.py:9:from maya_transcribe.language import ISO3_TO_WHISPER_LANG, to_whisper_lang
./maya_transcribe/asr.py:12:_WHISPER_TO_ISO3: dict[str, str] = {v: k for k, v in ISO3_TO_WHISPER_LANG.items()}
./maya_transcribe/asr.py:35:class FasterWhisperASR:
./maya_transcribe/asr.py:43:            from faster_whisper import WhisperModel
./maya_transcribe/asr.py:46:                "faster-whisper is not installed. Install with: pip install -e '.[asr]'"
./maya_transcribe/asr.py:63:        self._model = WhisperModel(model_size_or_path, device=device, compute_type=compute_type)
./maya_transcribe/asr.py:67:        return f"faster-whisper:{self.model_size_or_path}"
./maya_transcribe/asr.py:78:        lang = to_whisper_lang(language)
./maya_transcribe/asr.py:79:        # faster-whisper's Assamese token ("as") tends to produce Devanagari output,
./maya_transcribe/asr.py:84:            # faster-whisper supports a subset of Whisper's language tokens.
./maya_transcribe/asr.py:87:                from faster_whisper.tokenizer import _LANGUAGE_CODES  # type: ignore
./maya_transcribe/asr.py:135:class TransformersWhisperASR:
./maya_transcribe/asr.py:137:    Whisper-family ASR using HuggingFace Transformers (useful for fine-tuned checkpoints like IndicWhisper).
./maya_transcribe/asr.py:151:                "TransformersWhisperASR requires the optional dependencies: torch + transformers. "
./maya_transcribe/asr.py:232:        whisper_lang = to_whisper_lang(language)
./maya_transcribe/asr.py:233:        if whisper_lang:
./maya_transcribe/asr.py:237:                    forced_decoder_ids = get_ids(language=whisper_lang, task="transcribe")
./maya_transcribe/asr.py:254:                    # Whisper's newer generate() prefers forced ids via generation_config.
./maya_transcribe/asr.py:278:            language=whisper_lang or language,
./maya_transcribe/asr.py:290:class TransformersWhisperASRRouter:
./maya_transcribe/asr.py:292:    Lazily load and cache Whisper-family HF checkpoints by language.
./maya_transcribe/asr.py:310:        self._cache: OrderedDict[str, TransformersWhisperASR] = OrderedDict()
./maya_transcribe/asr.py:333:            backend = TransformersWhisperASR(model_id=model_id, device=self.device, dtype=self.dtype)
./maya_transcribe/asr.py:367:        return _WHISPER_TO_ISO3.get(code, None)
./maya_transcribe/asr.py:371:class MMSASR:
./maya_transcribe/asr.py:373:    MMS ASR backend using CTC decoding (facebook/mms-1b-all).
./maya_transcribe/asr.py:374:    Provides architectural diversity from Whisper's encoder-decoder approach.
./maya_transcribe/asr.py:379:        model_id: str = "facebook/mms-1b-all",
./maya_transcribe/asr.py:387:                "MMSASR requires torch + transformers. Install with: pip install -e '.[torch]'"
./maya_transcribe/asr.py:400:        # MMS models require set_target_lang for language-specific vocab.
./maya_transcribe/asr.py:405:        return f"mms-ctc:{self.model_id.split('/')[-1]}"
./maya_transcribe/asr.py:434:        """Set target language for MMS multilingual tokenizer."""
./maya_transcribe/asr.py:478:            raise RuntimeError("MMS processor did not return input_values or input_features")
./maya_transcribe/asr.py:497:class MMSASRRouter:
./maya_transcribe/asr.py:499:    Router for MMS ASR with language-specific model loading.
./maya_transcribe/asr.py:500:    MMS-1B-all supports 1000+ languages with a single model but benefits from
./maya_transcribe/asr.py:507:        model_id: str = "facebook/mms-1b-all",
./maya_transcribe/asr.py:512:        self._backend: MMSASR | None = None
./maya_transcribe/asr.py:521:            self._backend = MMSASR(model_id=self.model_id, device=self.device)
./maya_transcribe/asr.py:528:    Provides Conformer architecture diversity from Whisper's encoder-decoder.
./docs/Reference3.md:1:Since compute is not a constraint and your goal is absolute minimal Word Error Rate (WER) to train SOTA models, you should not rely on a single ASR model. Even the best models (like Whisper V3) hallucinate or mishear specific Indian phonemes.
./docs/Reference3.md:3:The "cleverest" strategy is to build a **Consensus & Arbitration Pipeline** (often called a ROVER system in research). You will run multiple SOTA models in parallel and use a "Judge" (LLM + Acoustic Aligner) to pick or merge the best result.
./docs/Reference3.md:12:1. **OpenAI Whisper:** (Built-in language detection).
./docs/Reference3.md:14:3. **Meta MMS / NVIDIA NeMo:** (Alternative architecture).
./docs/Reference3.md:29:| **AI4Bharat IndicWhisper** | **The Specialist** | **Crucial.** Fine-tuned specifically for Indian languages. It often outperforms standard Whisper on Dravidian languages (Malayalam, Tamil, Telugu, Kannada). |
./docs/Reference3.md:30:| **OpenAI Whisper (Large-v3)** | **The Generalist** | Excellent general robustness and vocabulary. Acts as the baseline anchor. |
./docs/Reference3.md:31:| **NVIDIA Canary / Parakeet** | **The Challenger** | Currently SOTA on the Open ASR Leaderboard. Very strong on multilingual tasks and often beats Whisper on speed/accuracy. |
./docs/Reference3.md:32:| **Meta MMS (1B)** | **The Diversity Vote** | Trained on Massively Multilingual Speech. Useful tie-breaker for lower-resource languages like Odia or Assamese. |
./docs/Reference3.md:45:* **Result:** If Whisper says "apple" but the audio sounds like "ample," the alignment score will drop. You pick the transcript with the highest alignment confidence.
./docs/Reference3.md:47:#### B. The Semantic Judge (LLM Arbitration)
./docs/Reference3.md:49:Use a high-intelligence LLM (GPT-4o, Claude 3.5 Sonnet, or a fine-tuned Indic LLM like **Sarvam-1**) as a "Reviewer."
./docs/Reference3.md:52:* **Clever Trick:** Use the **ROVER (Recognizer Output Voting Error Reduction)** approach. If 3 models agree on a word, keep it. If they disagree, let the LLM decide based on context.
./docs/Reference3.md:62:    C --> E[AI4Bharat IndicWhisper]
./docs/Reference3.md:63:    C --> F[OpenAI Whisper V3]
./docs/Reference3.md:73:    H & I & J & L --> M[LLM Consensus Judge]
./docs/Reference3.md:80:1. **Primary Model:** **AI4Bharat/IndicWhisper** (HuggingFace). This is non-negotiable for high-quality Indic TTS data.
./docs/Reference3.md:81:2. **Secondary Model:** `faster-whisper` (Large-v3 implementation).
./outputs/test_run_ctc3/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391afc89b", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run_ctc3/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "src_ffprobe": {"path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "duration_sec": 12.11625, "sample_rate": 48000, "channels": 1}, "ffprobe": {"path": "outputs/test_run_ctc3/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "eng", "prob": 0.999092698097229}, {"code": "spa", "prob": 0.00018186250235885382}, {"code": "haw", "prob": 6.795905937906355e-05}, {"code": "lat", "prob": 6.20875071035698e-05}, {"code": "war", "prob": 3.527590160956606e-05}]}, "route": {"language_hint": "eng"}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "ctc": {"model_id": "facebook/wav2vec2-base-960h", "best_key": "rover", "best_loss": 1.2419719696044922, "second_best_key": null, "second_best_loss": null, "margin": null, "scores": {"rover": {"loss": 1.2419719696044922, "model_id": "facebook/wav2vec2-base-960h", "unk_ratio": 0.0, "target_len": 16, "token_score_mean": -0.4490797834514524, "token_score_min": -6.753193378448486, "token_coverage": 0.9375, "token_logp_threshold": -3.0, "span_count": 16, "span_mismatch": false}, "faster-whisper:tiny|det": {"loss": 1.2419719696044922, "model_id": "facebook/wav2vec2-base-960h", "unk_ratio": 0.0, "target_len": 16, "token_score_mean": -0.4490797834514524, "token_score_min": -6.753193378448486, "token_coverage": 0.9375, "token_logp_threshold": -3.0, "span_count": 16, "span_mismatch": false}, "faster-whisper:tiny|samp_t0.2": {"loss": 1.2419719696044922, "model_id": "facebook/wav2vec2-base-960h", "unk_ratio": 0.0, "target_len": 16, "token_score_mean": -0.4490797834514524, "token_score_min": -6.753193378448486, "token_coverage": 0.9375, "token_logp_threshold": -3.0, "span_count": 16, "span_mismatch": false}}, "errors": {}}, "final": {"text": "intricate web of", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 1.43262…12586191 tokens truncated…r:large-v3|det": {"model": "faster-whisper:large-v3", "text": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 4.0200000000000005, "text": " అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "avg_logprob": -0.0656866766512394, "no_speech_prob": 0.017791748046875, "compression_ratio": 1.528301886792453}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 4.0200000000000005, "text": " అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "avg_logprob": -0.06581274982084308, "no_speech_prob": 0.017791748046875, "compression_ratio": 1.528301886792453}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "transformers:Vignesh-M/Indic-whisper": {"model": "transformers:Vignesh-M/Indic-whisper", "text": "அந்து வெள்ளை கொத்த சினிமாலுக்கு சந்தகம் செய்யச்செட்டப்படு ஆமாம் அப்ளிகேஷன்ஸ்", "language": "te", "language_probability": null, "segments": [], "meta": {"device": "cuda", "dtype": "float16", "max_new_tokens": 256, "num_beams": 5}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.0656866766512394, "no_speech_prob_mean": 0.017791748046875, "compression_ratio_mean": 1.528301886792453}, "faster-whisper:large-v3|samp_t0.2": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.06581274982084308, "no_speech_prob_mean": 0.017791748046875, "compression_ratio_mean": 1.528301886792453}, "transformers:Vignesh-M/Indic-whisper": {"language": "te", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "transformers:Vignesh-M/Indic-whisper"], "unique_backends": ["faster-whisper", "transformers"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "faster-whisper:large-v3|samp_t0.2": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "transformers:Vignesh-M/Indic-whisper": "அந்து வெள்ளை கொத்த சினிமாலுக்கு சந்தகம் செய்யச்செட்டப்படு ஆமாம் அப்ளிகேஷன்ஸ்"}, "rover": {"text": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "skeleton_key": "faster-whisper:large-v3|det", "disagreement_rate": 0.4666666666666667, "disagreements": [{"kind": "token", "index": 0, "chosen": "అందువల్లే", "options": {"வெள்ளை": 1.0, "అందువల్లే": 2.0}}, {"kind": "token", "index": 1, "chosen": "కొత్త", "options": {"கொத்த": 1.0, "కొత్త": 2.0}}, {"kind": "token", "index": 2, "chosen": "సినిమాలకు", "options": {"சினிமாலுக்கு": 1.0, "సినిమాలకు": 2.0}}, {"kind": "token", "index": 3, "chosen": "సంతకం", "options": {"சந்தகம்": 1.0, "సంతకం": 2.0}}, {"kind": "token", "index": 4, "chosen": "చేసేటప్పుడు", "options": {"செய்யச்செட்டப்படு": 1.0, "చేసేటప్పుడు": 2.0}}, {"kind": "token", "index": 5, "chosen": "ఆమా", "options": {"ஆமாம்": 1.0, "ఆమా": 2.0}}, {"kind": "token", "index": 6, "chosen": "అప్లికేశన్", "options": {"அப்ளிகேஷன்ஸ்": 1.0, "అప్లికేశన్": 2.0}}]}, "ctc": null, "final": {"text": "అందువల్లే కొత్త సినిమాలకు సంతకం చేసేటప్పుడు ఆమా అప్లికేశన్", "source": "rover"}, "verification": {"decision": "human", "reasons": ["asr_disagreement"], "scores": {"duration_sec": 4.05775, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.4666666666666667, "lid": {"available": true, "top1": "tel", "top1_prob": 0.9998016953468323, "margin": 0.9997586071840487, "confident": true}, "no_speech_prob_max": 0.017791748046875, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.0656866766512394, "compression_ratio_mean": 1.528301886792453, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.212910998146981, "lid_sec": 0.10561589058488607, "asr_sec": 4.72756619611755, "normalize_sec": 0.00010815495625138283, "rover_sec": 0.00033154571428894997, "ctc_sec": 4.507601261138916e-07, "verify_sec": 5.9962738305330276e-05, "total_sec": 5.046679684892297}}
./outputs/bench/maya_hf_v2/indic_whisper/results.jsonl:2399:{"schema_version": 1, "segment_id": "Kathbath__telugu__valid__973__844424932555056-170-f.m4a-0e2c014af5aa4277", "base_id": "Kathbath__telugu__valid__973__844424932555056-170-f.m4a", "relpath": "tel/Kathbath__telugu__valid__973__844424932555056-170-f.m4a.wav", "uid": "0e2c014af5aa4277", "src_path": "data/bench/maya_hf_v2/audio/tel/Kathbath__telugu__valid__973__844424932555056-170-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_v2/indic_whisper/audio/Kathbath__telugu__valid__973__844424932555056-170-f.m4a-0e2c014af5aa4277.wav", "src_ffprobe": {"path": "data/bench/maya_hf_v2/audio/tel/Kathbath__telugu__valid__973__844424932555056-170-f.m4a.wav", "duration_sec": 7.616125, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_v2/indic_whisper/audio/Kathbath__telugu__valid__973__844424932555056-170-f.m4a-0e2c014af5aa4277.wav", "duration_sec": 3.992938, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 3.9929375, "rms_dbfs": -23.1921424265539, "peak_dbfs": -3.4846621363161603, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "tel", "prob": 0.9998433589935303}, {"code": "sco", "prob": 2.752190084720496e-05}, {"code": "tam", "prob": 2.085634696413763e-05}, {"code": "mal", "prob": 1.418612191628199e-05}, {"code": "kan", "prob": 1.1597052434808575e-05}]}, "route": {"language_hint": "tel", "language_hint_source": "lid"}, "asr": {"faster-whisper:large-v3|det": {"model": "faster-whisper:large-v3", "text": "తమ్మని తాము కాపాడు కోవల అంటే వారిని కట్టడి చేయాలనే", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 3.96, "text": " తమ్మని తాము కాపాడు కోవల అంటే వారిని కట్టడి చేయాలనే", "avg_logprob": -0.12153672680412371, "no_speech_prob": 0.2108154296875, "compression_ratio": 1.4623655913978495}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "తమ్మని తాము కాపాడుకోవల అంటే వారిని కట్టడి చేయాలనే", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 3.96, "text": " తమ్మని తాము కాపాడుకోవల అంటే వారిని కట్టడి చేయాలనే", "avg_logprob": -0.12426874677961071, "no_speech_prob": 0.2108154296875, "compression_ratio": 1.4361702127659575}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "transformers:Vignesh-M/Indic-whisper": {"model": "transformers:Vignesh-M/Indic-whisper", "text": "தமனி தாமு காப்பாடு காவலாட்ட வர்ணனை கட்டடிச் செய்யலானே", "language": "te", "language_probability": null, "segments": [], "meta": {"device": "cuda", "dtype": "float16", "max_new_tokens": 256, "num_beams": 5}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.12153672680412371, "no_speech_prob_mean": 0.2108154296875, "compression_ratio_mean": 1.4623655913978495}, "faster-whisper:large-v3|samp_t0.2": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.12426874677961071, "no_speech_prob_mean": 0.2108154296875, "compression_ratio_mean": 1.4361702127659575}, "transformers:Vignesh-M/Indic-whisper": {"language": "te", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "transformers:Vignesh-M/Indic-whisper"], "unique_backends": ["faster-whisper", "transformers"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "తమ్మని తాము కాపాడు కోవల అంటే వారిని కట్టడి చేయాలనే", "faster-whisper:large-v3|samp_t0.2": "తమ్మని తాము కాపాడుకోవల అంటే వారిని కట్టడి చేయాలనే", "transformers:Vignesh-M/Indic-whisper": "தமனி தாமு காப்பாடு காவலாட்ட வர்ணனை கட்டடிச் செய்யலானே"}, "rover": {"text": "తమ్మని తాము కాపాడుకోవల అంటే వారిని కట్టడి చేయాలనే", "skeleton_key": "faster-whisper:large-v3|samp_t0.2", "disagreement_rate": 0.4666666666666667, "disagreements": [{"kind": "token", "index": 0, "chosen": "తమ్మని", "options": {"தமனி": 1.0, "తమ్మని": 2.0}}, {"kind": "token", "index": 1, "chosen": "తాము", "options": {"தாமு": 1.0, "తాము": 2.0}}, {"kind": "token", "index": 2, "chosen": "కాపాడుకోవల", "options": {"காப்பாடு": 1.0, "కాపాడుకోవల": 1.0, "కోవల": 1.0}}, {"kind": "token", "index": 3, "chosen": "అంటే", "options": {"காவலாட்ட": 1.0, "అంటే": 2.0}}, {"kind": "token", "index": 4, "chosen": "వారిని", "options": {"வர்ணனை": 1.0, "వారిని": 2.0}}, {"kind": "token", "index": 5, "chosen": "కట్టడి", "options": {"கட்டடிச்": 1.0, "కట్టడి": 2.0}}, {"kind": "token", "index": 6, "chosen": "చేయాలనే", "options": {"செய்யலானே": 1.0, "చేయాలనే": 2.0}}]}, "ctc": null, "final": {"text": "తమ్మని తాము కాపాడుకోవల అంటే వారిని కట్టడి చేయాలనే", "source": "rover"}, "verification": {"decision": "human", "reasons": ["asr_disagreement"], "scores": {"duration_sec": 3.9929375, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.4666666666666667, "lid": {"available": true, "top1": "tel", "top1_prob": 0.9998433589935303, "margin": 0.9998158370926831, "confident": true}, "no_speech_prob_max": 0.2108154296875, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.12153672680412371, "compression_ratio_mean": 1.4623655913978495, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.2113340748474002, "lid_sec": 0.1047569359652698, "asr_sec": 3.77521511586383, "normalize_sec": 8.649472147226334e-05, "rover_sec": 0.0003453129902482033, "ctc_sec": 6.211921572685242e-07, "verify_sec": 6.524287164211273e-05, "total_sec": 4.091890691779554}}
./outputs/bench/maya_hf_v2/indic_whisper/results.jsonl:2400:{"schema_version": 1, "segment_id": "Kathbath__telugu__valid__990__844424932556595-170-f.m4a-b6ace4bc1e880b2e", "base_id": "Kathbath__telugu__valid__990__844424932556595-170-f.m4a", "relpath": "tel/Kathbath__telugu__valid__990__844424932556595-170-f.m4a.wav", "uid": "b6ace4bc1e880b2e", "src_path": "data/bench/maya_hf_v2/audio/tel/Kathbath__telugu__valid__990__844424932556595-170-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_v2/indic_whisper/audio/Kathbath__telugu__valid__990__844424932556595-170-f.m4a-b6ace4bc1e880b2e.wav", "src_ffprobe": {"path": "data/bench/maya_hf_v2/audio/tel/Kathbath__telugu__valid__990__844424932556595-170-f.m4a.wav", "duration_sec": 6.455188, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_v2/indic_whisper/audio/Kathbath__telugu__valid__990__844424932556595-170-f.m4a-b6ace4bc1e880b2e.wav", "duration_sec": 2.595063, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 2.5950625, "rms_dbfs": -31.330318265857265, "peak_dbfs": -17.831613332188315, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "tel", "prob": 0.997545063495636}, {"code": "mal", "prob": 0.0012452129740267992}, {"code": "kan", "prob": 0.00044176614028401673}, {"code": "tam", "prob": 0.0002700774057302624}, {"code": "sco", "prob": 4.403354614623822e-05}]}, "route": {"language_hint": "tel", "language_hint_source": "lid"}, "asr": {"faster-whisper:large-v3|det": {"model": "faster-whisper:large-v3", "text": "కొన్డల్లవో కూడా తైయావరు పేరు కొన్డల్లవో", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 29.98, "text": " కొన్డల్లవో కూడా తైయావరు పేరు కొన్డల్లవో", "avg_logprob": -0.20853365671176177, "no_speech_prob": 0.8330078125, "compression_ratio": 1.472972972972973}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల", "language": "te", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.59, "text": " కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల", "avg_logprob": -0.10013294749789768, "no_speech_prob": 0.8330078125, "compression_ratio": 4.4520547945205475}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "transformers:Vignesh-M/Indic-whisper": {"model": "transformers:Vignesh-M/Indic-whisper", "text": "கொண்டுள்ள கூட தயாரு பேருகோண்டி உ", "language": "te", "language_probability": null, "segments": [], "meta": {"device": "cuda", "dtype": "float16", "max_new_tokens": 256, "num_beams": 5}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.20853365671176177, "no_speech_prob_mean": 0.8330078125, "compression_ratio_mean": 1.472972972972973}, "faster-whisper:large-v3|samp_t0.2": {"language": "te", "language_probability": 1, "avg_logprob_mean": -0.10013294749789768, "no_speech_prob_mean": 0.8330078125, "compression_ratio_mean": 4.4520547945205475}, "transformers:Vignesh-M/Indic-whisper": {"language": "te", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "transformers:Vignesh-M/Indic-whisper"], "unique_backends": ["faster-whisper", "transformers"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "కొన్డల్లవో కూడా తైయావరు పేరు కొన్డల్లవో", "faster-whisper:large-v3|samp_t0.2": "కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల వాగుడా తైయావరు పేరు కొండల్ల", "transformers:Vignesh-M/Indic-whisper": "கொண்டுள்ள கூட தயாரு பேருகோண்டி உ"}, "rover": {"text": "కొన్డల్లవో కూడా తైయావరు పేరు కొన్డల్లవో", "skeleton_key": "faster-whisper:large-v3|det", "disagreement_rate": 0.45454545454545453, "disagreements": [{"kind": "token", "index": 0, "chosen": "కొన్డల్లవో", "options": {"கொண்டுள்ள": 1.0, "కొండల్ల": 1.0, "కొన్డల్లవో": 1.0}}, {"kind": "token", "index": 1, "chosen": "కూడా", "options": {"கூட": 1.0, "కూడా": 1.0, "వాగుడా": 1.0}}, {"kind": "token", "index": 2, "chosen": "తైయావరు", "options": {"தயாரு": 1.0, "తైయావరు": 2.0}}, {"kind": "token", "index": 3, "chosen": "పేరు", "options": {"பேருகோண்டி": 1.0, "పేరు": 2.0}}, {"kind": "token", "index": 4, "chosen": "కొన్డల్లవో", "options": {"உ": 1.0, "కొండల్ల": 1.0, "కొన్డల్లవో": 1.0}}]}, "ctc": null, "final": {"text": "కొన్డల్లవో కూడా తైయావరు పేరు కొన్డల్లవో", "source": "rover"}, "verification": {"decision": "reject", "reasons": ["asr_disagreement", "no_speech"], "scores": {"duration_sec": 2.5950625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.45454545454545453, "lid": {"available": true, "top1": "tel", "top1_prob": 0.997545063495636, "margin": 0.9962998505216092, "confident": true}, "no_speech_prob_max": 0.8330078125, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.20853365671176177, "compression_ratio_mean": 1.472972972972973, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.21047051018103957, "lid_sec": 0.09475642116740346, "asr_sec": 4.210717263165861, "normalize_sec": 0.0001339837908744812, "rover_sec": 0.0003678370267152786, "ctc_sec": 4.516914486885071e-07, "verify_sec": 5.550496280193329e-05, "total_sec": 4.5165926232002676}}


**Searching benchmark scripts**


sed -n '1,200p' /home/ubuntu/maya-transcribe/README.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# maya-transcribe

Offline, quality-first transcription pipeline for cleaned single-speaker audio segments (Indic + English).

## Install

```bash
python3 -m pip install -U pip
python3 -m pip install -e '.[full]'
```

If you only need R2 download + evaluation (no ASR), install the base deps:

```bash
python3 -m pip install -e .
```

## Configure

Create / update `.env` in the repo root. The CLI auto-loads it.

Minimum for R2:
- `R2_ENDPOINT_URL`
- `R2_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET` (defaults to `1-cleaned-data` if unset)
- `R2_PREFIX` (optional)

## Usage

```bash
maya-transcribe --help
maya-transcribe download --video-id <VIDEO_ID> --bucket 1-cleaned-data --out data/
maya-transcribe run --segments-dir data/<VIDEO_ID>/**/segments --out outputs/<VIDEO_ID> --whisper-model large-v3
```

Evaluation (requires a reference JSONL keyed by `segment_id`):

```bash
maya-transcribe eval --results outputs/<VIDEO_ID>/results.jsonl --refs refs.jsonl
```

Optional extras:
- Add additional Whisper-family checkpoints: `--transformers-whisper-model <hf_model_id>` (repeatable)
- Add Gemini as an extra hypothesis: `--gemini-model gemini-2.0-flash`
- Enable audio-grounded CTC fit scoring: `--ctc-model <hf_ctc_model_id>` (or `--ctc-model-map ctc_models.json`)


**Checking CLI support for new audio LLMs**

I'll search the codebase for mentions of "gemini" and "voxtral" to confirm if these audio-capable models are integrated or accessible before considering new ASR backend additions.


rg -n "gemini|voxtral|mistral|llm" /home/ubuntu/maya-transcribe