Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Step 1 — Task Understanding (from task.md)
-(TODO)
+### Task Summary (from task.md)
+- **Goal:** Produce highly accurate, offline transcriptions for cleaned, single-speaker audio segments so they are safe for training ASR/TTS.
+- **Languages:** Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (plus “unknown/unflagged” segments).
+- **Audio segment length range:** ~1s up to ~60s max (typical ~30–50s).
+- **Compute constraints:** None (can run multiple large models, offline, no latency requirements).
+- **Quality target (WER/CER):** “As low as possible” / “can’t afford mistakes” → treat as **precision-first** (accept fewer, near-perfect segments rather than keeping everything with hidden label noise).
+- **Acceptance philosophy (precision-first vs recall-first):** **Precision-first**; strict rejection + targeted human correction for the hard tail.
+- **Any required tools/frameworks:** Not specified; implied needs: spoken language ID, multi-model ASR, system-combination, and audio-grounded verification (forced alignment / CTC scoring).
+- **Any banned tools/constraints:** Not explicit, but the user explicitly worries about **hallucinations** → avoid any step that can invent content (especially text-only LLM “transcription”).
+
+### Success Metrics
+Define “good” in numbers and gates (must be tuned on a gold set; initial values are starting heuristics, not guarantees).
+- **Primary:** WER by language (space-delimited) + CER by language (robust for Indic scripts), reported on a stratified gold set.
+- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).
+- **Must-meet thresholds (calibrate):**
+  - **Max allowable WER/CER for accepted samples (measured on QA sample):** set a target like “≤2–5% WER” (inference; depends on domain + language) and tune gates until the accepted pool meets it.
+  - **Max disagreement allowed between systems:** require strong agreement among at least 2 diverse recognizers (ROVER consensus score / pairwise CER threshold; tune).
+  - **Minimum alignment confidence:** require high audio-text fit via alignment/CTC scoring (tune per language + duration).
+
+### Assumptions (explicit)
+- **A1:** Segments are already single-speaker, mostly speech-only (music largely removed) and broadly “podcast/explainer” quality.
+- **A2:** We can run and store outputs from multiple ASR systems per segment (large GPU RAM / multi-GPU available).
+- **A3:** We can afford a human correction workflow for a small reject tail (e.g., 1–10% depending on strictness).

 ## Step 2 — Evaluation Rubric
-(TODO)
+Use this rubric to judge every technique/model (0–5 each):
+- **Expected quality gain** (WER/CER reduction on gold set)
+- **Robustness across Indic + code-switching** (Hindi-English, etc.)
+- **Verification friendliness** (exposes confidence/logprobs; supports alignment/scoring)
+- **Implementation complexity** (dependencies, infra, licensing)
+- **Failure risk** (hallucinations, brittle LID, script errors, silent label noise)
+
+**Decision labels**
+- **Adopt:** Use as-is in the master pipeline.
+- **Adapt:** Use, but with constraints/thresholds or narrower scope.
+- **Reject:** Do not use (too risky / not verifiable / not supported).
+- **Park:** Optional experiment after the core pipeline is working.

 ---

 ## Reference1 — Analysis
-(TODO: fill using the per-reference template)
+### Reference1 — One-paragraph summary
+Proposes a best-practice offline pipeline: spoken language ID → diverse ASR ensemble per language → system combination (ROVER) → independent audio-grounded verification (forced alignment / CTC scoring) → strict rejection + small human correction tail; LLMs only as constrained tie-breakers.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **MMS LID (+ VoxLingua107)** with a clear confidence/margin routing rule.
+- C2: **Explicit system-combination (ROVER)** + “disagreement map” as a debug artifact.
+- C3: **Independent verification gate** via forced alignment or **CTC “fit” scoring** (don’t trust the generating model).
+
+### Strengths (why it could work well)
+- S1: Strong alignment with “quality > coverage” and “audio-grounded verification required.”
+- S2: Uses **error diversity** (Indic specialist vs Whisper generalist vs MMS) to make ensemble voting useful.
+
+### Risks / Failure Modes
+- R1: MFA forced alignment across 12+ languages may be operationally painful (lexicons/acoustic models per language).
+- R2: Voxtral claims for non-Hindi Indic languages are explicitly “test-and-measure”; avoid assuming it’s best everywhere.
+
+### Missing Pieces / Gaps
+- G1: No concrete thresholding protocol (needs gold-set tuning plan).
+- G2: No implementation details for storing hypotheses/scores and orchestration (but it hints at them).
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement LID with MMS-LID (+ optional VoxLingua107) and a “confident vs unknown/mixed” routing rule.
+- [ ] Run a 3-model ASR ensemble per segment (Indic specialist + Whisper + diversity model).
+- [ ] Normalize transcripts and combine via ROVER (or equivalent confusion-network voting).
+- [ ] Add an audio-grounded verification gate (CTC scoring first; MFA where feasible).
+- [ ] Add a strict reject → human correction loop (e.g., Shoonya).
+
+### Decisions
+- Suggestion: MMS LID (+ VoxLingua107) and strict routing thresholds
+  - Decision: **Adopt (with tuning)**
+  - Why: Spoken LID routing is foundational; thresholds are sensible starting points but must be calibrated.
+- Suggestion: IndicConformer + Whisper large-v3 (+ MMS ASR) ensemble
+  - Decision: **Adopt / Adapt**
+  - Why: Core diversity is strong; we’ll also incorporate **IndicWhisper** (from other refs) as an additional Indic specialist.
+- Suggestion: ROVER system combination
+  - Decision: **Adopt**
+  - Why: Clear, well-known method to reduce WER when recognizers have diverse errors.
+- Suggestion: Verification via CTC scoring / forced alignment
+  - Decision: **Adopt**
+  - Why: Directly enforces “audio-grounded verification required”; critical to control hallucinations.
+- Suggestion: Text-only LLMs for transcription
+  - Decision: **Reject**
+  - Why: Not audio-grounded; high hallucination risk. Use only constrained tie-breaks.
+
+### Integration Notes
+- Complements: Ref5’s concrete 4-stage pipeline; Ref4’s LLM-as-judge constraints; Ref3/6’s emphasis on strict gating.
+- Potential conflict: Ref6’s incorrect dismissal of Voxtral; resolve by “measure, don’t assume.”

 ---

 ## Reference2 — Analysis
-(TODO)
+### Reference2 — One-paragraph summary
+Advocates multi-model ensembling (Whisper/IndicWhisper, Voxtral, Gemma, etc.), ROVER-style consensus, plus an LLM-based correction stage and selective human review to reach near-human WER.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: Emphasizes **language-specific fine-tunes** (IndicWhisper; Hinglish variants) outperforming generic Whisper for Indic.
+- C2: Reinforces **ROVER + multi-candidate comparison** as the “clever” offline strategy.
+- C3: Highlights the importance of a **human layer for the tail** and iterative improvement.
+
+### Strengths (why it could work well)
+- S1: Correctly treats “no compute constraints” as permission to ensemble broadly.
+- S2: Explicitly calls out hallucination risk and the need for constraints when using LLMs.
+
+### Risks / Failure Modes
+- R1: Promotes “LLM correction” somewhat broadly; unconstrained “fixing” can invent content.
+- R2: Mentions several models (e.g., Gemma multimodal ASR claims) without a verification/gating specification; risk of adopting untested components.
+- R3: Contains truncated/incomplete sections (“…chars truncated…”), so details may be missing.
+
+### Missing Pieces / Gaps
+- G1: No audio-grounded verification gate definition (forced alignment/CTC scoring is not central here).
+- G2: No concrete data schema + orchestration plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Use IndicWhisper (language-specific) as a primary Indic recognizer in the ensemble.
+- [ ] Treat Voxtral/Gemma/etc as **“test-and-measure”** optional candidates; evaluate on a gold set before relying on them.
+- [ ] If using an LLM, constrain it to selecting among provided hypotheses/spans (no free generation).
+
+### Decisions
+- Suggestion: IndicWhisper as a key model
+  - Decision: **Adopt**
+  - Why: Multiple references converge on “IndicWhisper fine-tune beats generic Whisper for Indic.”
+- Suggestion: Broad LLM post-editing to “improve” transcripts
+  - Decision: **Adapt → constrain heavily**
+  - Why: Useful only as a tie-breaker among hypotheses; otherwise hallucination risk.
+- Suggestion: Gemma as a prompted transcriber
+  - Decision: **Park**
+  - Why: Not supported elsewhere; treat as optional experiment after core pipeline is validated.
+
+### Integration Notes
+- Complements: Ref1’s ensemble + ROVER framing.
+- Conflicts with: Precision-first “no invention” policy if LLM correction is unconstrained.

 ---

 ## Reference3 — Analysis
-(TODO)
+### Reference3 — One-paragraph summary
+Proposes a “Consensus & Arbitration” pipeline: triple-ensemble LID; a “council” of 3–4 ASR models (IndicWhisper, Whisper, Canary/Parakeet, MMS); forced alignment to pick the most acoustically faithful transcript; optional LLM arbitration; plus audio quality scoring for TTS readiness.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **Hardline LID gating**: disagreement → quarantine/discard to prevent dataset poisoning.
+- C2: **Acoustic truth filter** framing: choose among candidates using alignment likelihood.
+- C3: Adds **audio quality scoring** (beyond transcription accuracy) for TTS training.
+
+### Strengths (why it could work well)
+- S1: Strong alignment with “quality > coverage” and strict acceptance.
+- S2: Separation between generation (ASR models) and validation (alignment) is correct.
+
+### Risks / Failure Modes
+- R1: Claims about specific leaderboards/SOTA status (Canary/Parakeet) aren’t supported within the references.
+- R2: “LLM selects best transcript” is risky unless constrained; may introduce invented tokens.
+- R3: External “deep dive” link is not actionable for this repo plan.
+
+### Missing Pieces / Gaps
+- G1: No concrete combination method (ROVER specifics) beyond mentioning it.
+- G2: No language/script normalization plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Add strict LID gating logic (agreement required; otherwise flag/reject).
+- [ ] Implement forced alignment / alignment scoring to pick among hypotheses.
+- [ ] Add audio-quality scoring (SNR/clipping/reverb proxies) to protect TTS quality.
+
+### Decisions
+- Suggestion: 3-model LID ensemble and quarantine on disagreement
+  - Decision: **Adapt**
+  - Why: Good for precision; but make thresholds configurable so we don’t over-reject short segments; include an “unknown/mixed” route.
+- Suggestion: Add NVIDIA Canary/Parakeet as a challenger ASR
+  - Decision: **Park**
+  - Why: Could add diversity, but requires validation + integration work; not required for a strong first version.
+- Suggestion: LLM arbitration
+  - Decision: **Adapt → constrained tie-break only**
+  - Why: Must not generate new text; must be downstream of audio-grounded scoring.
+- Suggestion: Audio quality scoring for TTS
+  - Decision: **Adopt**
+  - Why: Even perfect transcripts don’t rescue low-SNR/reverberant speech for TTS.
+
+### Integration Notes
+- Complements: Ref1’s verification gate; Ref5’s preprocessing + confidence filtering.

 ---

 ## Reference4 — Analysis
-(TODO)
+### Reference4 — One-paragraph summary
+Defines a multi-stage “ensemble + consensus + verification” architecture with an explicit LLM-as-judge prompt (selection-only), disagreement detection, and a composite quality score that combines agreement, confidence, LM plausibility, and audio quality.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **Centroid transcript** idea (lowest average distance to others) as a simple “consensus” heuristic.
+- C2: **LLM-as-judge prompt template** that forbids generation and supports a LOW-confidence flag.
+- C3: Suggests **character-level voting** and **forced alignment post-processing** for verification.
+
+### Strengths (why it could work well)
+- S1: Explicitly frames LLM as a judge, not a generator (hallucination-aware).
+- S2: Provides practical engineering logic for disagreement routing and composite scoring.
+
+### Risks / Failure Modes
+- R1: Includes many models, including proprietary ones (Gemini/USM), which may not be feasible or consistent with “offline” assumptions.
+- R2: Perplexity/LM-based plausibility can be misleading for rare names, code-switching, and domain terms unless treated as a weak signal.
+- R3: Mentions NLLB-LID (primarily text LID) in an audio LID context; likely not directly applicable.
+
+### Missing Pieces / Gaps
+- G1: No specific, audio-grounded verification metric definition (alignment/CTC score thresholds).
+- G2: No plan for storing model outputs and auditability beyond a sketch.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement disagreement detection (pairwise CER/WER variance) and route only hard cases to the judge/human.
+- [ ] Use selection-only LLM prompts (no generation) if adding an LLM tie-breaker.
+- [ ] Consider char-level combination for Indic scripts.
+
+### Decisions
+- Suggestion: Centroid transcript / distance-based consensus
+  - Decision: **Adapt**
+  - Why: Useful as a fallback/diagnostic, but we’ll primarily use ROVER for combination.
+- Suggestion: LLM-as-judge selection-only prompt
+  - Decision: **Adopt**
+  - Why: Works with our “no invention” requirement when constrained; still must pass audio-grounded gating.
+- Suggestion: LM perplexity scoring (MuRIL/IndicBERT)
+  - Decision: **Park**
+  - Why: Potentially helpful as a weak feature, but not reliable enough to drive acceptance decisions alone.
+- Suggestion: Proprietary audio-native LLMs (Gemini) as verification
+  - Decision: **Reject (for the offline core plan)**
+  - Why: Offline + reproducibility constraints; keep the core pipeline fully local/open where possible.
+
+### Integration Notes
+- Complements: Ref1’s “LLM tie-break only” guidance and verification gates.

 ---

 ## Reference5 — Analysis
-(TODO)
+### Reference5 — One-paragraph summary
+Presents a concrete 4-stage pipeline: preprocessing (VAD/format), spoken LID (MMS-LID), 3-model ASR ensemble (IndicWhisper + Whisper large-v3 + MMS), ROVER combination, and confidence-gated correction with hallucination safeguards; also emphasizes Indic script normalization (IndicNLP).
+
+### Key Contributions (unique ideas worth preserving)
+- C1: A **clean, implementable baseline stack** (Silero VAD → MMS-LID → IndicWhisper/Whisper/MMS → ROVER).
+- C2: **Confidence-gated post-processing** with strict edit/length checks to prevent “over-correction.”
+- C3: **IndicNLP normalization** callout (preserving diacritics / script correctness).
+
+### Strengths (why it could work well)
+- S1: Practical and actionable; aligns with precision-first philosophy.
+- S2: Builds in hallucination safeguards (only touch low-confidence cases; reject large edits).
+
+### Risks / Failure Modes
+- R1: Some performance numbers and claims are presented without provenance inside the repo (treat as directional, not guarantees).
+- R2: LLM/correction stage still needs an audio-grounded verification gate; text-only checks aren’t enough to “prove” correctness.
+- R3: Some model mentions appear speculative/unclear (e.g., “Omnilingual ASR 2025”); validate before adoption.
+
+### Missing Pieces / Gaps
+- G1: Verification method is listed but not fully specified (how to compute/threshold alignment confidence).
+- G2: Orchestration and storage schema details aren’t spelled out.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement preprocessing: 16kHz mono, VAD trimming, basic audio QC.
+- [ ] Implement LID: MMS-LID + thresholds.
+- [ ] Implement ASR ensemble: IndicWhisper + Whisper large-v3 + MMS.
+- [ ] Implement ROVER combination and store full hypothesis set.
+- [ ] Add constrained correction only for low-confidence cases; enforce edit distance + length ratio constraints.
+
+### Decisions
+- Suggestion: Silero VAD + basic audio normalization
+  - Decision: **Adopt**
+  - Why: Cheap win; improves both LID and ASR stability.
+- Suggestion: IndicWhisper + Whisper + MMS as “core ensemble”
+  - Decision: **Adopt**
+  - Why: Strong diversity and consistent with other refs.
+- Suggestion: “LLM-based correction” stage
+  - Decision: **Adapt**
+  - Why: Only as constrained tie-break among options and always downstream of audio-grounded scoring.
+- Suggestion: IndicNLP normalization
+  - Decision: **Adopt**
+  - Why: Normalization is required for system combination and consistent training text.
+
+### Integration Notes
+- Complements: Ref1’s verification gate; Ref4’s LLM judge prompt discipline.

 ---

 ## Reference6 — Analysis
-(TODO)
+### Reference6 — One-paragraph summary
+Proposes a consensus MoE pipeline: triple LID check, 3-ASR “council,” extract N-best hypotheses, use an Indic LLM (Sarvam-1) to pick the lowest-perplexity option, and then enforce correctness with forced alignment (discard segments with poor alignment).
+
+### Key Contributions (unique ideas worth preserving)
+- C1: Emphasizes **N-best hypotheses**, not just 1-best transcripts (useful for later rescoring/ROVER).
+- C2: Reinforces **forced alignment as a “hallucination killer”** and strict discard logic.
+
+### Strengths (why it could work well)
+- S1: Clear insistence on verification and aggressive filtering.
+- S2: N-best extraction is a concrete engineering improvement for offline pipelines.
+
+### Risks / Failure Modes
+- R1: Uses **LLM perplexity as the main selector**, which is not audio-grounded and can prefer fluent-but-wrong text (names, code-switching, domain terms).
+- R2: Contains an incorrect/contradictory statement about Voxtral’s existence vs Ref1/2 (treat as unreliable).
+- R3: “Drop misaligned words” can silently delete content; for training data, it may be safer to reject the segment than to produce partial/edited transcripts unless you can verify the edit is correct.
+
+### Missing Pieces / Gaps
+- G1: No system combination method (ROVER/confusion network) despite multiple ASR sources.
+- G2: No script normalization plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Extract N-best hypotheses (top-k beams) where supported (Whisper/Conformer/MMS).
+- [ ] Use forced alignment/CTC scoring as a hard gate; discard segments that fail.
+
+### Decisions
+- Suggestion: N-best extraction
+  - Decision: **Adopt**
+  - Why: Enables better combination/verification and reduces reliance on any single decode.
+- Suggestion: Perplexity-based selection with Sarvam-1
+  - Decision: **Park**
+  - Why: Could be an auxiliary feature, but not acceptable as a primary correctness signal without audio grounding.
+- Suggestion: Aggressive forced alignment discard thresholds
+  - Decision: **Adapt**
+  - Why: Keep the “discard on poor fit” principle, but tune thresholds on a gold set and prefer “reject segment” over “silently drop words.”
+
+### Integration Notes
+- Complements: Ref1/5’s ensemble + verification pipeline.
+- Conflicts with: Ref1’s caution that text-only LLMs shouldn’t be used as generators/selectors without audio gating.

 ---

 ## Master Plan (Unified)
-(TODO)
+Goal: **near-zero-error accepted transcripts** via (1) diverse generation and (2) independent, audio-grounded verification; everything uncertain is rejected and routed to human correction.
+
+### 1) Pre-QC (audio sanity checks)
+- Enforce canonical format: 16kHz, mono, PCM/FLAC/WAV (adapt as needed).
+- Run VAD trim (e.g., Silero VAD) to hard-trim leading/trailing silence and remove non-speech tails.
+- Compute audio QC metrics and reject early if bad (precision-first): clipping %, extreme loudness, very low speech ratio, suspiciously short effective speech, etc. (Inference: exact metrics/thresholds must be tuned.)
+
+### 2) Language ID routing (handling unknown/mixed)
+- Run spoken LID on every segment:
+  - Primary: **MMS-LID** (Refs 1/5).
+  - Optional second opinion: **SpeechBrain VoxLingua107 ECAPA** (Refs 1/5) and/or Whisper LID (Refs 3/6) as a tie-breaker.
+- Routing (starting heuristic from Ref1; tune on gold set):
+  - If top-1 prob ≥ 0.90 and margin(top1-top2) ≥ 0.20 → **confident route**
+  - Else → **unknown/mixed route** (process with multilingual ensemble and stricter verification).
+
+### 3) ASR ensemble (at least 2–3 diverse systems)
+**Core idea (Refs 1/5):** combine an Indic specialist + a strong multilingual generalist + a diversity model.
+- **Indic segments (the 11 Indic languages):**
+  - **IndicWhisper** (AI4Bharat fine-tuned Whisper) (Refs 3/5, also implied in Ref2/4).
+  - **Whisper large-v3** as generalist anchor (Refs 1/3/5).
+  - **IndicConformer-600M multilingual** (hybrid CTC/RNNT; diversity in failure modes) (Refs 1/4/6).
+  - Optional diversity: **MMS ASR** (Refs 1/3/5) especially for lower-resource (Odia/Assamese) or when others disagree.
+- **English segments:**
+  - **Whisper large-v3** + **Voxtral** (Refs 1/2) (and optionally a third English ASR for diversity if needed).
+- **Unknown/mixed segments:**
+  - Run a multilingual-heavy set: Whisper large-v3 + IndicConformer multilingual + MMS (+ Voxtral if available) and later decide based on verification.
+
+Decoding policy (Ref1):
+- Start with deterministic/high-precision decoding (beam search, temperature=0 for Whisper-like models).
+- If decoding fails/degenerates, re-decode with alternate params; treat fallbacks as lower-confidence and require stricter verification.
+- Where supported, store **N-best hypotheses** (Ref6) not only 1-best.
+
+### 4) Normalization layer (mandatory before combination/eval)
+- Unicode normalize; normalize punctuation/numerals consistently; standardize whitespace.
+- Use **IndicNLP** (Ref5) or equivalent for Indic script normalization so system combination isn’t dominated by trivial differences.
+
+### 5) System combination (don’t just pick one)
+- Use **ROVER / confusion-network voting** (Refs 1/4/5) to produce:
+  - `combined_transcript`
+  - a per-token **consensus/conflict map**
+- Consider character-level voting for Indic scripts as an adaptation (Ref4) if word tokenization is unreliable.
+
+### 6) Audio-grounded verification gate (non-negotiable)
+Use at least one **independent** audio-grounded check before accepting a transcript.
+- **Primary gate:** **CTC-based “fit” scoring** (Ref1) on:
+  - top candidates (best per-model 1-best and/or N-best)
+  - the ROVER-combined transcript
+  - Choose the best-scoring candidate and compute pass/fail metrics (coverage, mean posterior, confidence dips).
+- **Secondary/optional gate:** forced alignment (MFA / NeMo aligner) (Refs 1/3/6) where language resources support it cleanly.
+- **Reject policy:** if alignment/CTC fit is poor, **reject the whole segment** rather than “editing” words out (precision-first; avoid silent corruption).
+
+### 7) LLM usage (only if constrained, never as free generator)
+- Only use an LLM as a **selection/tie-breaker among existing hypotheses/spans** (Refs 1/4/5).
+- The LLM output is still provisional until it passes the same audio-grounded verification gate.
+- Prefer audio-native models (e.g., Voxtral) over text-only models when possible (Ref1), but still enforce verification.
+
+### 8) Reject + re-decode logic
+- If a segment fails the gate:
+  - Try an alternate decode (different beam/LM, add a diversity model like MMS).
+  - If still failing → send to human queue (don’t “force accept”).
+
+### 9) Human-in-the-loop for rejects
+- Only humans touch the reject tail (Refs 1/2/5).
+- Use an annotation UI that shows: audio + top hypotheses + disagreement highlights (Ref1) and lets the annotator pick/edit.
+- If you want OSS tooling: **Shoonya** (AI4Bharat) is suggested in Ref1.
+
+### 10) Storage schema (auditability)
+Store everything needed to debug and to re-run deterministically:
+- `segment_id`, source metadata, duration, audio QC metrics
+- `language_probs[]` (per LID model)
+- `hypotheses[]` (model, decode params, 1-best + N-best, confidences/logprobs if available)
+- `normalized_hypotheses[]`
+- `rover_output` + `disagreement_map`
+- `verification_scores[]` (CTC/aligner metrics)
+- `decision` (accept/reject/human) + `reason_codes` + timestamps
+
 ## Implementation Blueprint
-(TODO)
+### Engineering scope (minimal first, then scale)
+**Phase A (strong baseline):** LID + 3-model ASR ensemble + normalization + ROVER + CTC scoring gate + reject queue.
+**Phase B (quality hardening):** add N-best storage, better disagreement analytics, forced alignment where feasible, and a constrained LLM tie-breaker only for narrow cases.
+**Phase C (operations):** human review UI integration, dashboards, and active learning loop.
+
+### Orchestration
+- Batch segments by language route to maximize GPU throughput (Inference).
+- Run models in parallel across available GPUs; treat each model decode as an independent job with retries.
+- Make everything config-driven (`models`, `decode_params`, `thresholds`, `weights`), and record config snapshots with outputs.
+
+### Data structures (suggested)
+- `Segment`: `{segment_id, source_id, path, duration_s, qc_metrics}`
+- `LidResult`: `{model, probs: {lang: p}, top1, top2, margin}`
+- `Hypothesis`: `{model, lang, text, nbest[], token_logprobs?, avg_logprob?, decode_params}`
+- `CombineResult`: `{method: rover, text, consensus_score, disagreement_spans[]}`
+- `VerifyResult`: `{method: ctc_fit|forced_align, score, coverage, pass, diagnostics}`
+- `Decision`: `{status: accept|reject|human, reason_codes[], chosen_text}`
+
+### Model integration notes (practical)
+- Prefer models that can expose confidence/logprob signals; if not, lean more on independent CTC scoring + agreement.
+- Keep model versions pinned (Hugging Face commit hash / NeMo checkpoint version) for reproducibility.
+- Persist raw hypotheses + scores; never overwrite outputs in-place without versioning.
+
+### System combination (ROVER)
+- Use an existing implementation/toolkit where possible (Ref1/5 mention NIST SCTK; Ref3 suggests a simple script).
+- Maintain per-language model weights learned on a dev set (Inference; implied by Ref1/4/5).
+
+### Observability
+- Aggregate metrics by language, duration bucket, and source video.
+- Track top reject reasons (low LID confidence, high disagreement, low alignment score, audio QC failure).
+- Sample a fixed % of “accepted” for spot-check human QA to estimate residual label noise (Inference).

 ## Validation Protocol
-(TODO)
+### Build a gold set (required before tuning gates)
+- Stratified sample across:
+  - all languages
+  - duration buckets (1–3s, 3–10s, 10–30s, 30–60s)
+  - audio conditions (clean vs mildly noisy)
+  - code-switching presence (at least Hindi-English; if present in your data)
+- Human-transcribe this gold set with strict guidelines (verbatim, include disfluencies if desired for training consistency).
+
+### Evaluate components and the full pipeline
+- Score each base ASR model (WER/CER) by language and condition.
+- Score ensemble variants:
+  - best-single-model
+  - ROVER combination (with/without MMS/IndicConformer)
+  - verification gate impact (precision/recall tradeoff vs gold)
+
+### Tune thresholds and weights
+- Tune:
+  - LID “confident route” thresholds
+  - model weights for ROVER per language
+  - verification gate thresholds (CTC fit / alignment pass)
+- Objective: maximize **precision** of accepted set subject to a feasible accept rate (precision-first).
+
+### Active learning loop
+- Collect rejects + hard disagreements as an “error discovery” pool.
+- Human-correct a subset, then:
+  - recalibrate thresholds/weights
+  - optionally fine-tune the best-performing model(s) for your domain (Inference; consistent with Ref2’s iterative improvement idea)

 ## Do This Next (Execution Checklist)
-(TODO)
+1. Create a stratified gold set (per language + durations) and produce human transcripts.
+2. Stand up the baseline pipeline: `MMS-LID → (IndicWhisper + Whisper large-v3 + IndicConformer [+ MMS]) → normalize → ROVER → CTC fit gate → accept/reject`.
+3. Store *all* hypotheses + scores in an auditable schema (JSONL/Parquet) and build simple per-language dashboards.
+4. Tune routing/verification thresholds on the gold set until the accepted pool hits your precision target.
+5. Add a human review workflow for rejects (UI + queue); feed corrected tail back into evaluation and (optional) fine-tuning.
*** End Patch


Output:
apply_patch verification failed: Failed to find expected lines in /home/ubuntu/maya-transcribe/Agents.md:
## Step 1 — Task Understanding (from task.md)
(TODO)

## Step 2 — Evaluation Rubric
(TODO)

---

## Reference1 — Analysis
(TODO: fill using the per-reference template)

---

## Reference2 — Analysis
(TODO)

---

## Reference3 — Analysis
(TODO)

---

## Reference4 — Analysis
(TODO)

---

## Reference5 — Analysis
(TODO)

---

## Reference6 — Analysis
(TODO)

---

## Master Plan (Unified)
(TODO)
## Implementation Blueprint
(TODO)

## Validation Protocol
(TODO)

## Do This Next (Execution Checklist)
(TODO)


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Inspecting file content**

It seems like the patch failed because the content of the file is different than expected. We read earlier, and while there are placeholders, there might be extra lines or spaces causing the issue. I think I should inspect the portion around Step 1 in the Agents.md file to ensure I can patch it correctly. I’ll focus on everything from "Analysis Workspace" to the end to get a better idea of what's going on.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
(TODO)

---

## Reference3 — Analysis
(TODO)

---

## Reference4 — Analysis
(TODO)

---

## Reference5 — Analysis
(TODO)

---

## Reference6 — Analysis
(TODO)

---

## Master Plan (Unified)
(TODO)

## Implementation Blueprint
(TODO)

## Validation Protocol
(TODO)

## Do This Next (Execution Checklist)
(TODO)


Summary: auto


Output:
Exit code: 0
Wall time: 0 seconds
Output:
2. **Language ID routing** (include handling for uncertain/mixed)
3. **ASR ensemble** (per language group; include at least 2 diverse systems)
4. **Normalization layer** (unicode, punctuation, numerals, script normalization)
5. **System combination** (e.g., voting/ROVER or other)
6. **Audio-grounded verification gate** (forced alignment / CTC scoring / confidence)
7. **Reject + re-decode logic** (different decoding params / alternative model)
8. **Human-in-the-loop for rejects** (only the tail; scalable workflow)
9. **Storage schema** (store all hypotheses + scores for auditability)

### Implementation Blueprint (engineering plan)
- Orchestration: batching, GPU scheduling, retries
- Data structures:
  - `segment_id`, `language_probs`, `hypotheses[]`, `rover_output`, `alignment_scores`, `decision`, `audit_log`
- Reproducibility: config-driven pipeline, versioned models, deterministic decoding where possible
- Observability: dashboards for rejection reasons, per-language error patterns

### Validation Protocol (prove it works)
- Build a **gold set** (stratified across languages, speakers, domains, lengths)
- Compute WER/CER by language + conditions
- Threshold tuning approach
- Active learning loop: rejects → human fix → retrain / recalibrate

### Final Deliverable
- A short “Do This Next” checklist that an engineer can execute immediately.

---

## Global TODO Checklist (tick as you complete)
- [ ] Read `task.md` and fill **Step 1**
- [ ] Create evaluation rubric (**Step 2**) and decision labels
- [ ] Complete Reference1 analysis
- [ ] Complete Reference2 analysis
- [ ] Complete Reference3 analysis
- [ ] Complete Reference4 analysis
- [ ] Complete Reference5 analysis
- [ ] Complete Reference6 analysis
- [ ] Write **Master Plan**
- [ ] Write **Implementation Blueprint**
- [ ] Write **Validation Protocol**
- [ ] Produce final “Do This Next” execution checklist

---

# Analysis Workspace (Fill below as you go)
> Everything below this line is your working output. Keep it clean and structured.

## Step 1 — Task Understanding (from task.md)
(TODO)

## Step 2 — Evaluation Rubric
(TODO)

---

## Reference1 — Analysis
(TODO: fill using the per-reference template)

---

## Reference2 — Analysis
(TODO)

---

## Reference3 — Analysis
(TODO)

---

## Reference4 — Analysis
(TODO)

---

## Reference5 — Analysis
(TODO)

---

## Reference6 — Analysis
(TODO)

---

## Master Plan (Unified)
(TODO)

## Implementation Blueprint
(TODO)

## Validation Protocol
(TODO)

## Do This Next (Execution Checklist)
(TODO)


python3 - <<'PY'
import pathlib
text = pathlib.Path('Agents.md').read_text(encoding='utf-8')
for line in text.splitlines():
    if line.startswith('## Step 1'):
        print(line)
        print([hex(ord(ch)) for ch in line])
        break
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
## Step 1 — Extract Requirements from `task.md`
['0x23', '0x23', '0x20', '0x53', '0x74', '0x65', '0x70', '0x20', '0x31', '0x20', '0x2014', '0x20', '0x45', '0x78', '0x74', '0x72', '0x61', '0x63', '0x74', '0x20', '0x52', '0x65', '0x71', '0x75', '0x69', '0x72', '0x65', '0x6d', '0x65', '0x6e', '0x74', '0x73', '0x20', '0x66', '0x72', '0x6f', '0x6d', '0x20', '0x60', '0x74', '0x61', '0x73', '0x6b', '0x2e', '0x6d', '0x64', '0x60']


**Adjusting requirement extraction**

I noticed that the script printed "## Step 1 — Extract Requirements from `task.md`," which is from the top of the file instead of the analysis workspace. It seems we expected to find the line "## Step 1 — Task Understanding (from task.md)" but instead, we found the first matching line incorrectly. I need to refine our search to locate that specific line properly so we can adjust accordingly.


python3 - <<'PY'
import pathlib
text = pathlib.Path('Agents.md').read_text(encoding='utf-8')
for line in text.splitlines():
    if line.strip() == '## Step 1 — Task Understanding (from task.md)':
        print('FOUND')
        print(line)
        print([hex(ord(ch)) for ch in line])
        break
else:
    print('NOT FOUND')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
FOUND
## Step 1 — Task Understanding (from task.md)
['0x23', '0x23', '0x20', '0x53', '0x74', '0x65', '0x70', '0x20', '0x31', '0x20', '0x2014', '0x20', '0x54', '0x61', '0x73', '0x6b', '0x20', '0x55', '0x6e', '0x64', '0x65', '0x72', '0x73', '0x74', '0x61', '0x6e', '0x64', '0x69', '0x6e', '0x67', '0x20', '0x28', '0x66', '0x72', '0x6f', '0x6d', '0x20', '0x74', '0x61', '0x73', '0x6b', '0x2e', '0x6d', '0x64', '0x29']


**Investigating patch issues**

I noticed that we're using the em dash correctly, which is good! The patch should have matched the target line, but it's possible the apply_patch tool failed due to the large size or context issues. If any part of the context doesn't match, the entire patch might fail. I think we attempted to update multiple sections at once without specifying separate hunks. Each hunk needs to match the whole context for the patch to be successfully applied.


**Analyzing patch file structure**

I'm looking at the patch file for updates in "Agents.md." It looks like I've got a single hunk starting with "@@", containing multiple modifications throughout. The apply_patch grammar allows for these sequences, so that seems fine. However, if the file content doesn't match exactly—due to missing or extra lines—this could cause the patch to fail. I need to double-check that every line aligns correctly to prevent any issues when applying the patch.


**Evaluating patching issues**

I included the whole set in the patch, but it might fail if any part doesn’t match exactly. There could be an issue with the “## Master Plan (Unified)” line followed by “(TODO)” and a blank line before the Implementation section. I think the discrepancy lies in the line structure; the patch requires exact matches from the "@@" context marker, and even small differences can lead to failure. It’s essential to double-check the structure to ensure everything aligns correctly.


**Reassessing patch context**

I need to think about the patch context at the beginning. We used "## Step 1 — Task Understanding (from task.md)" preceded by "@@", where we removed "(TODO)." There’s likely only one occurrence of "(TODO)" after the Step 1 header. The apply_patch tool claims it couldn't find the expected lines at all, which makes me wonder if it’s scanning for an exact sequence from the start of the hunk. I think I need to clarify the context marker usage, since "@@" might not provide enough detail for the tool to find the exact matching lines.


**Examining patch failure reasons**

The patch might fail if there are multiple "@@" sections or if it's large with lines that don’t match exactly. For instance, I included "## Step 1 — Task Understanding (from task.md)\n-(TODO)," while the file has "## Step 1 — Task Understanding (from task.md)\n(TODO)." The apply_patch tool uses regex to find exact sequences, so if the block changes context, it may not match. It’s important to ensure all modifications align with the context. I’ll need to check for any small variations, like extra spaces or blank lines, and look for CRLF issues as well.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Agents.md: Unicode text, UTF-8 text


**Evaluating patching approach**

I'm wondering why the apply_patch tool couldn't find the expected lines. It seems like the tool might look for exact phrases, which isn't what I have. I think I’ll try patching in smaller chunks, focusing on the first section specifically to avoid matching issues. Another option is to replace the entire "Analysis Workspace" section by searching for its starting line and swapping in my new text. However, I need to be cautious because patch grammar doesn’t support regex; context needs to match exactly.


**Exploring patch limitations**

I'm considering that the patch might have failed because it was too large. Perhaps there’s an internal limit on size or complexity that I wasn’t aware of. It’s possible that the failure isn't just about mismatched lines but also the patch length. So, I'll try a more incremental approach by replacing the (TODO) line in Step 1 with my new content while keeping the rest unchanged for now. I’ll also include context lines around Step 2 to ensure it matches properly.


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Step 1 — Task Understanding (from task.md)
-(TODO)
+### Task Summary (from task.md)
+- **Goal:** Produce highly accurate, offline transcriptions for cleaned, single-speaker audio segments so they are safe for training ASR/TTS.
+- **Languages:** Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (plus “unknown/unflagged” segments).
+- **Audio segment length range:** ~1s up to ~60s max (typical ~30–50s).
+- **Compute constraints:** None (can run multiple large models, offline, no latency requirements).
+- **Quality target (WER/CER):** “As low as possible” / “can’t afford mistakes” → treat as **precision-first** (accept fewer, near-perfect segments rather than keeping everything with hidden label noise).
+- **Acceptance philosophy (precision-first vs recall-first):** **Precision-first**; strict rejection + targeted human correction for the hard tail.
+- **Any required tools/frameworks:** Not specified; implied needs: spoken language ID, multi-model ASR, system-combination, and audio-grounded verification (forced alignment / CTC scoring).
+- **Any banned tools/constraints:** Not explicit, but the user explicitly worries about **hallucinations** → avoid any step that can invent content (especially text-only LLM “transcription”).
+
+### Success Metrics
+Define “good” in numbers and gates (must be tuned on a gold set; initial values are starting heuristics, not guarantees).
+- **Primary:** WER by language (space-delimited) + CER by language (robust for Indic scripts), reported on a stratified gold set.
+- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).
+- **Must-meet thresholds (calibrate):**
+  - **Max allowable WER/CER for accepted samples (measured on QA sample):** set a target like “≤2–5% WER” (inference; depends on domain + language) and tune gates until the accepted pool meets it.
+  - **Max disagreement allowed between systems:** require strong agreement among at least 2 diverse recognizers (ROVER consensus score / pairwise CER threshold; tune).
+  - **Minimum alignment confidence:** require high audio-text fit via alignment/CTC scoring (tune per language + duration).
+
+### Assumptions (explicit)
+- **A1:** Segments are already single-speaker, mostly speech-only (music largely removed) and broadly “podcast/explainer” quality.
+- **A2:** We can run and store outputs from multiple ASR systems per segment (large GPU RAM / multi-GPU available).
+- **A3:** We can afford a human correction workflow for a small reject tail (e.g., 1–10% depending on strictness).

 ## Step 2 — Evaluation Rubric
 (TODO)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating project plan**

I’ve successfully updated Step 1. Now, it’s time to update the plan status: the first step, "Extract requirements from task.md," is complete, and I'll note that the next step, "Review Reference1–Reference6," is in progress. However, we've already read the references, so I’m realizing we still need to fill in their analyses. It seems like I’ll mark Step 1 as completed and will keep Step 2 in progress. I’ll update the plan accordingly!


Explanation: Filled Step 1 task requirements in Agents.md; next I’ll complete per-reference analyses based on Reference1–Reference6.


Output: Plan updated


**Updating evaluation rubric**

I need to update the evaluation rubric for Step 2. I'll patch the section, replacing the (TODO) with the necessary details. It seems like I have a clear direction on how to apply the patch similarly as before. It’s important for me to get this right, so I'll make sure I'm following the format and process exactly. This way, the updates will be coherent and effective for the users. Let's dive into it!


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Step 2 — Evaluation Rubric
-(TODO)
+Use this rubric to judge every technique/model (0–5 each):
+- **Expected quality gain** (WER/CER reduction on gold set)
+- **Robustness across Indic + code-switching** (Hindi-English, etc.)
+- **Verification friendliness** (exposes confidence/logprobs; supports alignment/scoring)
+- **Implementation complexity** (dependencies, infra, licensing)
+- **Failure risk** (hallucinations, brittle LID, script errors, silent label noise)
+
+**Decision labels**
+- **Adopt:** Use as-is in the master pipeline.
+- **Adapt:** Use, but with constraints/thresholds or narrower scope.
+- **Reject:** Do not use (too risky / not verifiable / not supported).
+- **Park:** Optional experiment after the core pipeline is working.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating Reference Analysis**

I'm starting to fill in the Reference1 analysis. My plan is to patch each section one at a time. First, I'll locate the Reference1 placeholder in the document. Then, I will replace the text "(TODO: fill using the per-reference template)" with the relevant content I've prepared. It's a straightforward process, but I want to make sure everything is accurate and organized as I go!


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference1 — Analysis
-(TODO: fill using the per-reference template)
+### Reference1 — One-paragraph summary
+Proposes a best-practice offline pipeline: spoken language ID → diverse ASR ensemble per language → system combination (ROVER) → independent audio-grounded verification (forced alignment / CTC scoring) → strict rejection + small human correction tail; LLMs only as constrained tie-breakers.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **MMS LID (+ VoxLingua107)** with a clear confidence/margin routing rule.
+- C2: **Explicit system-combination (ROVER)** + “disagreement map” as a debug artifact.
+- C3: **Independent verification gate** via forced alignment or **CTC “fit” scoring** (don’t trust the generating model).
+
+### Strengths (why it could work well)
+- S1: Strong alignment with “quality > coverage” and “audio-grounded verification required.”
+- S2: Uses **error diversity** (Indic specialist vs Whisper generalist vs MMS) to make ensemble voting useful.
+
+### Risks / Failure Modes
+- R1: MFA forced alignment across 12+ languages may be operationally painful (lexicons/acoustic models per language).
+- R2: Voxtral claims for non-Hindi Indic languages are explicitly “test-and-measure”; avoid assuming it’s best everywhere.
+
+### Missing Pieces / Gaps
+- G1: No concrete thresholding protocol (needs gold-set tuning plan).
+- G2: No implementation details for storing hypotheses/scores and orchestration (but it hints at them).
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement LID with MMS-LID (+ optional VoxLingua107) and a “confident vs unknown/mixed” routing rule.
+- [ ] Run a 3-model ASR ensemble per segment (Indic specialist + Whisper + diversity model).
+- [ ] Normalize transcripts and combine via ROVER (or equivalent confusion-network voting).
+- [ ] Add an audio-grounded verification gate (CTC scoring first; MFA where feasible).
+- [ ] Add a strict reject → human correction loop (e.g., Shoonya).
+
+### Decisions
+- Suggestion: MMS LID (+ VoxLingua107) and strict routing thresholds
+  - Decision: **Adopt (with tuning)**
+  - Why: Spoken LID routing is foundational; thresholds are sensible starting points but must be calibrated.
+- Suggestion: IndicConformer + Whisper large-v3 (+ MMS ASR) ensemble
+  - Decision: **Adopt / Adapt**
+  - Why: Core diversity is strong; we’ll also incorporate **IndicWhisper** (from other refs) as an additional Indic specialist.
+- Suggestion: ROVER system combination
+  - Decision: **Adopt**
+  - Why: Clear, well-known method to reduce WER when recognizers have diverse errors.
+- Suggestion: Verification via CTC scoring / forced alignment
+  - Decision: **Adopt**
+  - Why: Directly enforces “audio-grounded verification required”; critical to control hallucinations.
+- Suggestion: Text-only LLMs for transcription
+  - Decision: **Reject**
+  - Why: Not audio-grounded; high hallucination risk. Use only constrained tie-breaks.
+
+### Integration Notes
+- Complements: Ref5’s concrete 4-stage pipeline; Ref4’s LLM-as-judge constraints; Ref3/6’s emphasis on strict gating.
+- Potential conflict: Ref6’s incorrect dismissal of Voxtral; resolve by “measure, don’t assume.”

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference2 — Analysis
-(TODO)
+### Reference2 — One-paragraph summary
+Advocates multi-model ensembling (Whisper/IndicWhisper, Voxtral, Gemma, etc.), ROVER-style consensus, plus an LLM-based correction stage and selective human review to reach near-human WER.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: Emphasizes **language-specific fine-tunes** (IndicWhisper; Hinglish variants) outperforming generic Whisper for Indic.
+- C2: Reinforces **ROVER + multi-candidate comparison** as the “clever” offline strategy.
+- C3: Highlights the importance of a **human layer for the tail** and iterative improvement.
+
+### Strengths (why it could work well)
+- S1: Correctly treats “no compute constraints” as permission to ensemble broadly.
+- S2: Explicitly calls out hallucination risk and the need for constraints when using LLMs.
+
+### Risks / Failure Modes
+- R1: Promotes “LLM correction” somewhat broadly; unconstrained “fixing” can invent content.
+- R2: Mentions several models (e.g., Gemma multimodal ASR claims) without a verification/gating specification; risk of adopting untested components.
+- R3: Contains truncated/incomplete sections (“…chars truncated…”), so details may be missing.
+
+### Missing Pieces / Gaps
+- G1: No audio-grounded verification gate definition (forced alignment/CTC scoring is not central here).
+- G2: No concrete data schema + orchestration plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Use IndicWhisper (language-specific) as a primary Indic recognizer in the ensemble.
+- [ ] Treat Voxtral/Gemma/etc as **“test-and-measure”** optional candidates; evaluate on a gold set before relying on them.
+- [ ] If using an LLM, constrain it to selecting among provided hypotheses/spans (no free generation).
+
+### Decisions
+- Suggestion: IndicWhisper as a key model
+  - Decision: **Adopt**
+  - Why: Multiple references converge on “IndicWhisper fine-tune beats generic Whisper for Indic.”
+- Suggestion: Broad LLM post-editing to “improve” transcripts
+  - Decision: **Adapt → constrain heavily**
+  - Why: Useful only as a tie-breaker among hypotheses; otherwise hallucination risk.
+- Suggestion: Gemma as a prompted transcriber
+  - Decision: **Park**
+  - Why: Not supported elsewhere; treat as optional experiment after core pipeline is validated.
+
+### Integration Notes
+- Complements: Ref1’s ensemble + ROVER framing.
+- Conflicts with: Precision-first “no invention” policy if LLM correction is unconstrained.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference3 — Analysis
-(TODO)
+### Reference3 — One-paragraph summary
+Proposes a “Consensus & Arbitration” pipeline: triple-ensemble LID; a “council” of 3–4 ASR models (IndicWhisper, Whisper, Canary/Parakeet, MMS); forced alignment to pick the most acoustically faithful transcript; optional LLM arbitration; plus audio quality scoring for TTS readiness.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **Hardline LID gating**: disagreement → quarantine/discard to prevent dataset poisoning.
+- C2: **Acoustic truth filter** framing: choose among candidates using alignment likelihood.
+- C3: Adds **audio quality scoring** (beyond transcription accuracy) for TTS training.
+
+### Strengths (why it could work well)
+- S1: Strong alignment with “quality > coverage” and strict acceptance.
+- S2: Separation between generation (ASR models) and validation (alignment) is correct.
+
+### Risks / Failure Modes
+- R1: Claims about specific leaderboards/SOTA status (Canary/Parakeet) aren’t supported within the references.
+- R2: “LLM selects best transcript” is risky unless constrained; may introduce invented tokens.
+- R3: External “deep dive” link is not actionable for this repo plan.
+
+### Missing Pieces / Gaps
+- G1: No concrete combination method (ROVER specifics) beyond mentioning it.
+- G2: No language/script normalization plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Add strict LID gating logic (agreement required; otherwise flag/reject).
+- [ ] Implement forced alignment / alignment scoring to pick among hypotheses.
+- [ ] Add audio-quality scoring (SNR/clipping/reverb proxies) to protect TTS quality.
+
+### Decisions
+- Suggestion: 3-model LID ensemble and quarantine on disagreement
+  - Decision: **Adapt**
+  - Why: Good for precision; but make thresholds configurable so we don’t over-reject short segments; include an “unknown/mixed” route.
+- Suggestion: Add NVIDIA Canary/Parakeet as a challenger ASR
+  - Decision: **Park**
+  - Why: Could add diversity, but requires validation + integration work; not required for a strong first version.
+- Suggestion: LLM arbitration
+  - Decision: **Adapt → constrained tie-break only**
+  - Why: Must not generate new text; must be downstream of audio-grounded scoring.
+- Suggestion: Audio quality scoring for TTS
+  - Decision: **Adopt**
+  - Why: Even perfect transcripts don’t rescue low-SNR/reverberant speech for TTS.
+
+### Integration Notes
+- Complements: Ref1’s verification gate; Ref5’s preprocessing + confidence filtering.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference4 — Analysis
-(TODO)
+### Reference4 — One-paragraph summary
+Defines a multi-stage “ensemble + consensus + verification” architecture with an explicit LLM-as-judge prompt (selection-only), disagreement detection, and a composite quality score that combines agreement, confidence, LM plausibility, and audio quality.
+
+### Key Contributions (unique ideas worth preserving)
+- C1: **Centroid transcript** idea (lowest average distance to others) as a simple “consensus” heuristic.
+- C2: **LLM-as-judge prompt template** that forbids generation and supports a LOW-confidence flag.
+- C3: Suggests **character-level voting** and **forced alignment post-processing** for verification.
+
+### Strengths (why it could work well)
+- S1: Explicitly frames LLM as a judge, not a generator (hallucination-aware).
+- S2: Provides practical engineering logic for disagreement routing and composite scoring.
+
+### Risks / Failure Modes
+- R1: Includes many models, including proprietary ones (Gemini/USM), which may not be feasible or consistent with “offline” assumptions.
+- R2: Perplexity/LM-based plausibility can be misleading for rare names, code-switching, and domain terms unless treated as a weak signal.
+- R3: Mentions NLLB-LID (primarily text LID) in an audio LID context; likely not directly applicable.
+
+### Missing Pieces / Gaps
+- G1: No specific, audio-grounded verification metric definition (alignment/CTC score thresholds).
+- G2: No plan for storing model outputs and auditability beyond a sketch.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement disagreement detection (pairwise CER/WER variance) and route only hard cases to the judge/human.
+- [ ] Use selection-only LLM prompts (no generation) if adding an LLM tie-breaker.
+- [ ] Consider char-level combination for Indic scripts.
+
+### Decisions
+- Suggestion: Centroid transcript / distance-based consensus
+  - Decision: **Adapt**
+  - Why: Useful as a fallback/diagnostic, but we’ll primarily use ROVER for combination.
+- Suggestion: LLM-as-judge selection-only prompt
+  - Decision: **Adopt**
+  - Why: Works with our “no invention” requirement when constrained; still must pass audio-grounded gating.
+- Suggestion: LM perplexity scoring (MuRIL/IndicBERT)
+  - Decision: **Park**
+  - Why: Potentially helpful as a weak feature, but not reliable enough to drive acceptance decisions alone.
+- Suggestion: Proprietary audio-native LLMs (Gemini) as verification
+  - Decision: **Reject (for the offline core plan)**
+  - Why: Offline + reproducibility constraints; keep the core pipeline fully local/open where possible.
+
+### Integration Notes
+- Complements: Ref1’s “LLM tie-break only” guidance and verification gates.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference5 — Analysis
-(TODO)
+### Reference5 — One-paragraph summary
+Presents a concrete 4-stage pipeline: preprocessing (VAD/format), spoken LID (MMS-LID), 3-model ASR ensemble (IndicWhisper + Whisper large-v3 + MMS), ROVER combination, and confidence-gated correction with hallucination safeguards; also emphasizes Indic script normalization (IndicNLP).
+
+### Key Contributions (unique ideas worth preserving)
+- C1: A **clean, implementable baseline stack** (Silero VAD → MMS-LID → IndicWhisper/Whisper/MMS → ROVER).
+- C2: **Confidence-gated post-processing** with strict edit/length checks to prevent “over-correction.”
+- C3: **IndicNLP normalization** callout (preserving diacritics / script correctness).
+
+### Strengths (why it could work well)
+- S1: Practical and actionable; aligns with precision-first philosophy.
+- S2: Builds in hallucination safeguards (only touch low-confidence cases; reject large edits).
+
+### Risks / Failure Modes
+- R1: Some performance numbers and claims are presented without provenance inside the repo (treat as directional, not guarantees).
+- R2: LLM/correction stage still needs an audio-grounded verification gate; text-only checks aren’t enough to “prove” correctness.
+- R3: Some model mentions appear speculative/unclear (e.g., “Omnilingual ASR 2025”); validate before adoption.
+
+### Missing Pieces / Gaps
+- G1: Verification method is listed but not fully specified (how to compute/threshold alignment confidence).
+- G2: Orchestration and storage schema details aren’t spelled out.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Implement preprocessing: 16kHz mono, VAD trimming, basic audio QC.
+- [ ] Implement LID: MMS-LID + thresholds.
+- [ ] Implement ASR ensemble: IndicWhisper + Whisper large-v3 + MMS.
+- [ ] Implement ROVER combination and store full hypothesis set.
+- [ ] Add constrained correction only for low-confidence cases; enforce edit distance + length ratio constraints.
+
+### Decisions
+- Suggestion: Silero VAD + basic audio normalization
+  - Decision: **Adopt**
+  - Why: Cheap win; improves both LID and ASR stability.
+- Suggestion: IndicWhisper + Whisper + MMS as “core ensemble”
+  - Decision: **Adopt**
+  - Why: Strong diversity and consistent with other refs.
+- Suggestion: “LLM-based correction” stage
+  - Decision: **Adapt**
+  - Why: Only as constrained tie-break among options and always downstream of audio-grounded scoring.
+- Suggestion: IndicNLP normalization
+  - Decision: **Adopt**
+  - Why: Normalization is required for system combination and consistent training text.
+
+### Integration Notes
+- Complements: Ref1’s verification gate; Ref4’s LLM judge prompt discipline.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Reference6 — Analysis
-(TODO)
+### Reference6 — One-paragraph summary
+Proposes a consensus MoE pipeline: triple LID check, 3-ASR “council,” extract N-best hypotheses, use an Indic LLM (Sarvam-1) to pick the lowest-perplexity option, and then enforce correctness with forced alignment (discard segments with poor alignment).
+
+### Key Contributions (unique ideas worth preserving)
+- C1: Emphasizes **N-best hypotheses**, not just 1-best transcripts (useful for later rescoring/ROVER).
+- C2: Reinforces **forced alignment as a “hallucination killer”** and strict discard logic.
+
+### Strengths (why it could work well)
+- S1: Clear insistence on verification and aggressive filtering.
+- S2: N-best extraction is a concrete engineering improvement for offline pipelines.
+
+### Risks / Failure Modes
+- R1: Uses **LLM perplexity as the main selector**, which is not audio-grounded and can prefer fluent-but-wrong text (names, code-switching, domain terms).
+- R2: Contains an incorrect/contradictory statement about Voxtral’s existence vs Ref1/2 (treat as unreliable).
+- R3: “Drop misaligned words” can silently delete content; for training data, it may be safer to reject the segment than to produce partial/edited transcripts unless you can verify the edit is correct.
+
+### Missing Pieces / Gaps
+- G1: No system combination method (ROVER/confusion network) despite multiple ASR sources.
+- G2: No script normalization plan.
+
+### Actionable Extract (convert into implementable tasks)
+- [ ] Extract N-best hypotheses (top-k beams) where supported (Whisper/Conformer/MMS).
+- [ ] Use forced alignment/CTC scoring as a hard gate; discard segments that fail.
+
+### Decisions
+- Suggestion: N-best extraction
+  - Decision: **Adopt**
+  - Why: Enables better combination/verification and reduces reliance on any single decode.
+- Suggestion: Perplexity-based selection with Sarvam-1
+  - Decision: **Park**
+  - Why: Could be an auxiliary feature, but not acceptable as a primary correctness signal without audio grounding.
+- Suggestion: Aggressive forced alignment discard thresholds
+  - Decision: **Adapt**
+  - Why: Keep the “discard on poor fit” principle, but tune thresholds on a gold set and prefer “reject segment” over “silently drop words.”
+
+### Integration Notes
+- Complements: Ref1/5’s ensemble + verification pipeline.
+- Conflicts with: Ref1’s caution that text-only LLMs shouldn’t be used as generators/selectors without audio gating.

 ---
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating the review status**

I'm currently in the process of reviewing a plan. I can go ahead and mark this review as completed. Now, I should set the next step—deciding whether to adopt, adapt, or reject the plan—to in progress. This means I’ll need to call the update plan function accordingly. It feels like the right track to ensure everything moves smoothly!


Explanation: Filled per-reference analyses; next I’ll finalize the cross-reference decisions and synthesize them into the unified master plan.


Output: Plan updated


Summary: auto


**Updating documentation sections**

I need to update the "Master Plan (Unified)" section in Agents.md, specifically the Implementation Blueprint and Validation Protocol. I'll start by replacing the "(TODO)" placeholder in the Master Plan section with the content we've already drafted. It seems like we wrote this in a previous attempt, so reusing that information will save time. Alright, let's go ahead and patch the Master Plan section!


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Master Plan (Unified)
-(TODO)
+Goal: **near-zero-error accepted transcripts** via (1) diverse generation and (2) independent, audio-grounded verification; everything uncertain is rejected and routed to human correction.
+
+### 1) Pre-QC (audio sanity checks)
+- Enforce canonical format: 16kHz, mono, PCM/FLAC/WAV (adapt as needed).
+- Run VAD trim (e.g., Silero VAD) to hard-trim leading/trailing silence and remove non-speech tails.
+- Compute audio QC metrics and reject early if bad (precision-first): clipping %, extreme loudness, very low speech ratio, suspiciously short effective speech, etc. (Inference: exact metrics/thresholds must be tuned.)
+
+### 2) Language ID routing (handling unknown/mixed)
+- Run spoken LID on every segment:
+  - Primary: **MMS-LID** (Refs 1/5).
+  - Optional second opinion: **SpeechBrain VoxLingua107 ECAPA** (Refs 1/5) and/or Whisper LID (Refs 3/6) as a tie-breaker.
+- Routing (starting heuristic from Ref1; tune on gold set):
+  - If top-1 prob ≥ 0.90 and margin(top1-top2) ≥ 0.20 → **confident route**
+  - Else → **unknown/mixed route** (process with multilingual ensemble and stricter verification).
+
+### 3) ASR ensemble (at least 2–3 diverse systems)
+**Core idea (Refs 1/5):** combine an Indic specialist + a strong multilingual generalist + a diversity model.
+- **Indic segments (the 11 Indic languages):**
+  - **IndicWhisper** (AI4Bharat fine-tuned Whisper) (Refs 3/5, also implied in Ref2/4).
+  - **Whisper large-v3** as generalist anchor (Refs 1/3/5).
+  - **IndicConformer-600M multilingual** (hybrid CTC/RNNT; diversity in failure modes) (Refs 1/4/6).
+  - Optional diversity: **MMS ASR** (Refs 1/3/5) especially for lower-resource (Odia/Assamese) or when others disagree.
+- **English segments:**
+  - **Whisper large-v3** + **Voxtral** (Refs 1/2) (and optionally a third English ASR for diversity if needed).
+- **Unknown/mixed segments:**
+  - Run a multilingual-heavy set: Whisper large-v3 + IndicConformer multilingual + MMS (+ Voxtral if available) and later decide based on verification.
+
+Decoding policy (Ref1):
+- Start with deterministic/high-precision decoding (beam search, temperature=0 for Whisper-like models).
+- If decoding fails/degenerates, re-decode with alternate params; treat fallbacks as lower-confidence and require stricter verification.
+- Where supported, store **N-best hypotheses** (Ref6) not only 1-best.
+
+### 4) Normalization layer (mandatory before combination/eval)
+- Unicode normalize; normalize punctuation/numerals consistently; standardize whitespace.
+- Use **IndicNLP** (Ref5) or equivalent for Indic script normalization so system combination isn’t dominated by trivial differences.
+
+### 5) System combination (don’t just pick one)
+- Use **ROVER / confusion-network voting** (Refs 1/4/5) to produce:
+  - `combined_transcript`
+  - a per-token **consensus/conflict map**
+- Consider character-level voting for Indic scripts as an adaptation (Ref4) if word tokenization is unreliable.
+
+### 6) Audio-grounded verification gate (non-negotiable)
+Use at least one **independent** audio-grounded check before accepting a transcript.
+- **Primary gate:** **CTC-based “fit” scoring** (Ref1) on:
+  - top candidates (best per-model 1-best and/or N-best)
+  - the ROVER-combined transcript
+  - Choose the best-scoring candidate and compute pass/fail metrics (coverage, mean posterior, confidence dips).
+- **Secondary/optional gate:** forced alignment (MFA / NeMo aligner) (Refs 1/3/6) where language resources support it cleanly.
+- **Reject policy:** if alignment/CTC fit is poor, **reject the whole segment** rather than “editing” words out (precision-first; avoid silent corruption).
+
+### 7) LLM usage (only if constrained, never as free generator)
+- Only use an LLM as a **selection/tie-breaker among existing hypotheses/spans** (Refs 1/4/5).
+- The LLM output is still provisional until it passes the same audio-grounded verification gate.
+- Prefer audio-native models (e.g., Voxtral) over text-only models when possible (Ref1), but still enforce verification.
+
+### 8) Reject + re-decode logic
+- If a segment fails the gate:
+  - Try an alternate decode (different beam/LM, add a diversity model like MMS).
+  - If still failing → send to human queue (don’t “force accept”).
+
+### 9) Human-in-the-loop for rejects
+- Only humans touch the reject tail (Refs 1/2/5).
+- Use an annotation UI that shows: audio + top hypotheses + disagreement highlights (Ref1) and lets the annotator pick/edit.
+- If you want OSS tooling: **Shoonya** (AI4Bharat) is suggested in Ref1.
+
+### 10) Storage schema (auditability)
+Store everything needed to debug and to re-run deterministically:
+- `segment_id`, source metadata, duration, audio QC metrics
+- `language_probs[]` (per LID model)
+- `hypotheses[]` (model, decode params, 1-best + N-best, confidences/logprobs if available)
+- `normalized_hypotheses[]`
+- `rover_output` + `disagreement_map`
+- `verification_scores[]` (CTC/aligner metrics)
+- `decision` (accept/reject/human) + `reason_codes` + timestamps

 ## Implementation Blueprint
 (TODO)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Implementation Blueprint
-(TODO)
+### Engineering scope (minimal first, then scale)
+**Phase A (strong baseline):** LID + 3-model ASR ensemble + normalization + ROVER + CTC scoring gate + reject queue.
+**Phase B (quality hardening):** add N-best storage, better disagreement analytics, forced alignment where feasible, and a constrained LLM tie-breaker only for narrow cases.
+**Phase C (operations):** human review UI integration, dashboards, and active learning loop.
+
+### Orchestration
+- Batch segments by language route to maximize GPU throughput (Inference).
+- Run models in parallel across available GPUs; treat each model decode as an independent job with retries.
+- Make everything config-driven (`models`, `decode_params`, `thresholds`, `weights`), and record config snapshots with outputs.
+
+### Data structures (suggested)
+- `Segment`: `{segment_id, source_id, path, duration_s, qc_metrics}`
+- `LidResult`: `{model, probs: {lang: p}, top1, top2, margin}`
+- `Hypothesis`: `{model, lang, text, nbest[], token_logprobs?, avg_logprob?, decode_params}`
+- `CombineResult`: `{method: rover, text, consensus_score, disagreement_spans[]}`
+- `VerifyResult`: `{method: ctc_fit|forced_align, score, coverage, pass, diagnostics}`
+- `Decision`: `{status: accept|reject|human, reason_codes[], chosen_text}`
+
+### Model integration notes (practical)
+- Prefer models that can expose confidence/logprob signals; if not, lean more on independent CTC scoring + agreement.
+- Keep model versions pinned (Hugging Face commit hash / NeMo checkpoint version) for reproducibility.
+- Persist raw hypotheses + scores; never overwrite outputs in-place without versioning.
+
+### System combination (ROVER)
+- Use an existing implementation/toolkit where possible (Ref1/5 mention NIST SCTK; Ref3 suggests a simple script).
+- Maintain per-language model weights learned on a dev set (Inference; implied by Ref1/4/5).
+
+### Observability
+- Aggregate metrics by language, duration bucket, and source video.
+- Track top reject reasons (low LID confidence, high disagreement, low alignment score, audio QC failure).
+- Sample a fixed % of “accepted” for spot-check human QA to estimate residual label noise (Inference).

 ## Validation Protocol
 (TODO)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Validation Protocol
-(TODO)
+### Build a gold set (required before tuning gates)
+- Stratified sample across:
+  - all languages
+  - duration buckets (1–3s, 3–10s, 10–30s, 30–60s)
+  - audio conditions (clean vs mildly noisy)
+  - code-switching presence (at least Hindi-English; if present in your data)
+- Human-transcribe this gold set with strict guidelines (verbatim, include disfluencies if desired for training consistency).
+
+### Evaluate components and the full pipeline
+- Score each base ASR model (WER/CER) by language and condition.
+- Score ensemble variants:
+  - best-single-model
+  - ROVER combination (with/without MMS/IndicConformer)
+  - verification gate impact (precision/recall tradeoff vs gold)
+
+### Tune thresholds and weights
+- Tune:
+  - LID “confident route” thresholds
+  - model weights for ROVER per language
+  - verification gate thresholds (CTC fit / alignment pass)
+- Objective: maximize **precision** of accepted set subject to a feasible accept rate (precision-first).
+
+### Active learning loop
+- Collect rejects + hard disagreements as an “error discovery” pool.
+- Human-correct a subset, then:
+  - recalibrate thresholds/weights
+  - optionally fine-tune the best-performing model(s) for your domain (Inference; consistent with Ref2’s iterative improvement idea)

 ## Do This Next (Execution Checklist)
 (TODO)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Do This Next (Execution Checklist)
-(TODO)
+1. Create a stratified gold set (per language + durations) and produce human transcripts.
+2. Stand up the baseline pipeline: `MMS-LID → (IndicWhisper + Whisper large-v3 + IndicConformer [+ MMS]) → normalize → ROVER → CTC fit gate → accept/reject`.
+3. Store *all* hypotheses + scores in an auditable schema (JSONL/Parquet) and build simple per-language dashboards.
+4. Tune routing/verification thresholds on the gold set until the accepted pool hits your precision target.
+5. Add a human review workflow for rejects (UI + queue); feed corrected tail back into evaluation and (optional) fine-tuning.
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating checklist completion**

I need to update the Global TODO Checklist to show that everything is complete. I’ll make sure to patch that section and check all items off. But first, I want to double-check that we filled in every part correctly. We've completed Step 1, Step 2, all the reference analyses, the Master Plan, Implementation Blueprint, and Validation Protocol. Once I verify that, I can tick everything off. Time to patch it!


Input:
*** Begin Patch
*** Update File: Agents.md
@@
 ## Global TODO Checklist (tick as you complete)
-- [ ] Read `task.md` and fill **Step 1**
-- [ ] Create evaluation rubric (**Step 2**) and decision labels
-- [ ] Complete Reference1 analysis
-- [ ] Complete Reference2 analysis
-- [ ] Complete Reference3 analysis
-- [ ] Complete Reference4 analysis
-- [ ] Complete Reference5 analysis
-- [ ] Complete Reference6 analysis
-- [ ] Write **Master Plan**
-- [ ] Write **Implementation Blueprint**
-- [ ] Write **Validation Protocol**
-- [ ] Produce final “Do This Next” execution checklist
+- [x] Read `task.md` and fill **Step 1**
+- [x] Create evaluation rubric (**Step 2**) and decision labels
+- [x] Complete Reference1 analysis
+- [x] Complete Reference2 analysis
+- [x] Complete Reference3 analysis
+- [x] Complete Reference4 analysis
+- [x] Complete Reference5 analysis
+- [x] Complete Reference6 analysis
+- [x] Write **Master Plan**
+- [x] Write **Implementation Blueprint**
+- [x] Write **Validation Protocol**
+- [x] Produce final “Do This Next” execution checklist
*** End Patch


Output: {"output":"Success. Updated the following files:\nM Agents.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating plan statuses**

I need to focus on updating the plan statuses correctly. We’ve completed several steps like writing the master plan and validating it. However, the current step, "Decide Adopt/Adapt/Reject per idea," is still in progress, and I need to mark the next step sequentially. I can’t skip to marking things as completed; I have to set them as in progress first. Even though some tasks are done, I must reflect that accurately without jumping ahead in the process.


**Updating items sequentially**

I think we’ve completed quite a bit already, but I need to update the plan to reflect that. I can mark "Write unified master plan" as in progress, even if it’s already done, along with other tasks in separate updates. It's important I follow the rule of transitioning from pending to in progress, and then to completed. Since I can't have multiple tasks in progress at once, I'll do several updates in a sequence to track everything accurately.


**Updating plan steps**

I realize there are additional steps to address, such as "Add validation protocol and checklist" and "Update Agents.md artifact." My current plan includes extracting requirements and reviewing references, which are both completed. The current step, deciding how to adopt or adapt ideas, is in progress. To accurately reflect our progress, I’ll update the plan sequentially. First, I’ll mark steps as completed or in progress, ensuring that each transition follows the rules before I finalize everything. Let’s get started on those updates!


Explanation: Captured Adopt/Adapt/Reject decisions in the per-reference sections; moving on to the unified master plan write-up.


Output: Plan updated


**Finalizing plans**

I'm making sure to mark the master plan as completed, which feels satisfying! Now, it looks like I'm transitioning into the next phase with the blueprint in progress. I want to ensure I’m organized and clear about the steps ahead. Keeping track of everything really helps me feel more confident in moving forward. It's all about staying on top of things, right? Let's keep it going!


Explanation: Wrote the unified master plan section; next is drafting the concrete implementation blueprint.


Output: Plan updated


Explanation: Added the implementation blueprint; next is the validation protocol and execution checklist.


Output: Plan updated


Explanation: Completed validation protocol and the 'Do This Next' checklist; final step is ensuring Agents.md is fully updated and consistent.


Output: Plan updated


**Ensuring consistency in Agents.md**

I need to make sure that Agents.md is consistent by checking that all TODOs are replaced. This means I should look for any lingering "(TODO)" lines that might have been missed. I'll go ahead and search through the document to find those remaining items and ensure everything is updated. It's a bit of a critical step to keep things organized and functioning smoothly. Let's make sure it's done right!


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Finalizing tasks**

I’ve checked that there are no TODOs left, but I wonder if there are other tasks I might have missed. It’s essential to ensure that the global checklist is also ticked. I should check for formatting issues like double spaces, just to be thorough, but it seems fine overall. Now, I’ll mark the plan as the last step completed! It feels good to have things wrapped up neatly.


Explanation: Agents.md now contains the full synthesized plan, per-reference decisions, implementation blueprint, validation protocol, and an execution checklist.


Output: Plan updated


Output:
Exit code: 0
Wall time: 0 seconds
Output:
For each major suggestion:
- Suggestion: …
  - Decision: Adopt / Adapt / Reject / Park
  - Why:
  - What to do instead (if rejected):

### Integration Notes
How this reference fits with others.
- Conflicts with:
- Complements:
- Dependencies:

---

## Step 4 — Synthesis After All References
After finishing Reference1–Reference6, produce:

### Master Plan (single best pipeline)
Must include:
1. **Pre-QC** (audio sanity checks, VAD trimming, clipping/SNR checks)
2. **Language ID routing** (include handling for uncertain/mixed)
3. **ASR ensemble** (per language group; include at least 2 diverse systems)
4. **Normalization layer** (unicode, punctuation, numerals, script normalization)
5. **System combination** (e.g., voting/ROVER or other)
6. **Audio-grounded verification gate** (forced alignment / CTC scoring / confidence)
7. **Reject + re-decode logic** (different decoding params / alternative model)
8. **Human-in-the-loop for rejects** (only the tail; scalable workflow)
9. **Storage schema** (store all hypotheses + scores for auditability)

### Implementation Blueprint (engineering plan)
- Orchestration: batching, GPU scheduling, retries
- Data structures:
  - `segment_id`, `language_probs`, `hypotheses[]`, `rover_output`, `alignment_scores`, `decision`, `audit_log`
- Reproducibility: config-driven pipeline, versioned models, deterministic decoding where possible
- Observability: dashboards for rejection reasons, per-language error patterns

### Validation Protocol (prove it works)
- Build a **gold set** (stratified across languages, speakers, domains, lengths)
- Compute WER/CER by language + conditions
- Threshold tuning approach
- Active learning loop: rejects → human fix → retrain / recalibrate

### Final Deliverable
- A short “Do This Next” checklist that an engineer can execute immediately.

---

## Global TODO Checklist (tick as you complete)
- [x] Read `task.md` and fill **Step 1**
- [x] Create evaluation rubric (**Step 2**) and decision labels
- [x] Complete Reference1 analysis
- [x] Complete Reference2 analysis
- [x] Complete Reference3 analysis
- [x] Complete Reference4 analysis
- [x] Complete Reference5 analysis
- [x] Complete Reference6 analysis
- [x] Write **Master Plan**
- [x] Write **Implementation Blueprint**
- [x] Write **Validation Protocol**
- [x] Produce final “Do This Next” execution checklist

---

# Analysis Workspace (Fill below as you go)
> Everything below this line is your working output. Keep it clean and structured.

## Step 1 — Task Understanding (from task.md)
### Task Summary (from task.md)
- **Goal:** Produce highly accurate, offline transcriptions for cleaned, single-speaker audio segments so they are safe for training ASR/TTS.
- **Languages:** Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (plus “unknown/unflagged” segments).
- **Audio segment length range:** ~1s up to ~60s max (typical ~30–50s).
- **Compute constraints:** None (can run multiple large models, offline, no latency requirements).
- **Quality target (WER/CER):** “As low as possible” / “can’t afford mistakes” → treat as **precision-first** (accept fewer, near-perfect segments rather than keeping everything with hidden label noise).
- **Acceptance philosophy (precision-first vs recall-first):** **Precision-first**; strict rejection + targeted human correction for the hard tail.
- **Any required tools/frameworks:** Not specified; implied needs: spoken language ID, multi-model ASR, system-combination, and audio-grounded verification (forced alignment / CTC scoring).
- **Any banned tools/constraints:** Not explicit, but the user explicitly worries about **hallucinations** → avoid any step that can invent content (especially text-only LLM “transcription”).

### Success Metrics
Define “good” in numbers and gates (must be tuned on a gold set; initial values are starting heuristics, not guarantees).
- **Primary:** WER by language (space-delimited) + CER by language (robust for Indic scripts), reported on a stratified gold set.
- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).
- **Must-meet thresholds (calibrate):**
  - **Max allowable WER/CER for accepted samples (measured on QA sample):** set a target like “≤2–5% WER” (inference; depends on domain + language) and tune gates until the accepted pool meets it.
  - **Max disagreement allowed between systems:** require strong agreement among at least 2 diverse recognizers (ROVER consensus score / pairwise CER threshold; tune).
  - **Minimum alignment confidence:** require high audio-text fit via alignment/CTC scoring (tune per language + duration).

### Assumptions (explicit)
- **A1:** Segments are already single-speaker, mostly speech-only (music largely removed) and broadly “podcast/explainer” quality.
- **A2:** We can run and store outputs from multiple ASR systems per segment (large GPU RAM / multi-GPU available).
- **A3:** We can afford a human correction workflow for a small reject tail (e.g., 1–10% depending on strictness).

## Step 2 — Evaluation Rubric
Use this rubric to judge every technique/model (0–5 each):
- **Expected quality gain** (WER/CER reduction on gold set)
- **Robustness across Indic + code-switching** (Hindi-English, etc.)
- **Verification friendliness** (exposes confidence/logprobs; supports alignment/scoring)
- **Implementation complexity** (dependencies, infra, licensing)
- **Failure risk** (hallucinations, brittle LID, script errors, silent label noise)

**Decision labels**
- **Adopt:** Use as-is in the master pipeline.


**Scaffolding a Python Pipeline**

I'm considering whether to ask if the user wants a Python pipeline skeleton for Phase A, which could be helpful. I need to be concise and mention the file path: `Agents.md` as the root. I also need to think about including line number references — it seems the guidelines suggest this. However, while it’s recommended to include line numbers when referencing files, it might not be strictly necessary. I'll clarify that detail.


**Referencing Line Numbers**

I'm thinking we should provide a reference like `Agents.md:1`, focusing on the relevant start line instead of a full range. Using the command `nl -ba` is a good way to get line numbers, and I suspect the Step 1 section starts around line 170. But we can reference the general file with line 1, which might suffice for clarity since they want each reference to be a stand-alone path. So, referencing `Agents.md:1` looks like the best option here. I'll confirm the line numbers quickly.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	# Agents.md — Meta-Prompt + Workflow for “Reference Ensemble” Research (ASR Transcription Quality First)
     2
     3	## Role
     4	You are a **coding + research agent** operating inside a repo/workspace that contains:
     5	- `task.md` (the original problem statement + requirements)
     6	- `Reference1.md` … `Reference6.md` (different LLM-proposed approaches + code snippets)
     7	- Any supporting code/config files already present
     8
     9	Your job is to **read everything**, extract the best ideas, **reject weak or risky suggestions**, and synthesize a **single optimal plan** to produce **near-zero-error transcripts** for multi-lingual audio (offline; compute not a constraint; quality is).
    10
    11	---
    12
    13	## Non-Negotiables
    14	1. **No invention.** Only claim something is “recommended” if it is supported by the references or is a clearly-labeled inference.
    15	2. **Quality > Coverage.** If forced to choose, reject questionable segments rather than poison training data.
    16	3. **Audio-grounded verification required.** Any transcript must be **verified against the audio** before being accepted.
    17	4. **Make decisions.** For every major technique, explicitly decide: **Adopt / Adapt / Reject / Park** and justify.
    18	5. **Keep this file updated.** This file (`Agents.md`) is the living artifact: top section = this prompt; below it = your analysis, TODOs, decisions, and final plan.
    19
    20	---
    21
    22	## Output Contract (What you must produce in THIS file)
    23	By the end, `Agents.md` must contain:
    24	- A crisp understanding of `task.md` (constraints, metrics, languages, tooling assumptions)
    25	- A per-reference analysis for Reference1–Reference6:
    26	  - Summary, unique contributions, strengths, failure modes, missing pieces
    27	  - Concrete actionable items + what to ignore
    28	  - Adopt/Adapt/Reject/Park decisions
    29	- A unified **Master Plan** (pipeline + thresholds + verification + human loop)
    30	- An implementation blueprint (data structures, job orchestration, evaluation harness)
    31	- A validation protocol (gold set, WER/CER reporting, active learning loop)
    32
    33	---
    34
    35	## Step 0 — Read Order (do this first)
    36	1. Read `task.md` fully.
    37	2. Skim all `Reference*.md` quickly to build a map of what exists.
    38	3. Then deep-read each reference, one by one, and complete the per-reference template below.
    39
    40	---


**Finalizing file update details**

I’m looking to finalize my response about updating our files. I might reference `Agents.md:170`, and `task.md:1` could be relevant but not essential.

In the response, I plan to mention the file update and outline key pipeline components like MMS-LID, ensemble models including IndicWhisper and Whisper large-v3, ROVER, CTC fit verification, and a human review process.

I could also ask if the user wants a code skeleton next. I’ll keep the response small, using about 4-6 bullet points.


Updated `Agents.md:1` with the full synthesis:
- Filled Step 1 requirements + Step 2 rubric
- Completed Reference1–Reference6 analyses with Adopt/Adapt/Reject/Park decisions
- Wrote a unified precision-first pipeline: `MMS-LID → ASR ensemble (IndicWhisper + Whisper large-v3 + IndicConformer [+ MMS]) → normalization → ROVER → CTC/forced-alignment verification gate → accept/reject → human tail`
- Added implementation blueprint (data schema/orchestration) + validation protocol (gold set + threshold tuning) + “Do This Next” checklist

If you want, I can scaffold Phase A as an actual runnable Python pipeline next (config + schema + stubs for LID/ASR/ROVER/CTC gating).


# Context from my IDE setup:

## Open tabs:
- Agents.md: Agents.md
- task.md: task.md
- Reference1.md: docs/Reference1.md
- task.md: docs/task.md
- Reference5.md: docs/Reference5.md

## My request for Codex:
Can you now create the master.md in the root folder here along with agents.md with a block diagram and execution plan and why this current plan gives the minimal WER possible till date as we are generating transcriptions offline and works smartly and cleverly to minimize WER.


Summary: auto


**Focusing on user request**

The user is asking me to create a master.md file in the root folder, along with an agents.md that includes a block diagram and an execution plan. They want to know why this current plan minimizes the Word Error Rate (WER) while working offline. However, I need to be careful not to claim it achieves the absolute minimal WER, but rather that it aims for that by using proven techniques. I'll clarify it follows best practices without any unsupported claims.


**Clarifying the task**

I need to maintain the updates in Agents.md as the user has asked for both a master.md and enhancements to Agents.md, including a block diagram and an execution plan. Since Agents.md currently has sections like Master Plan and Do This Next, adding a block diagram there makes sense. It looks like I’ll be creating a concise master.md as a deliverable summary for engineering, ensuring it includes the necessary reasoning. I'll apply the updates using the correct patch function.


**Understanding the environment**

I realize that I don’t need to use skills here because I've been given the context. The previous setups indicated that I can operate in a restricted environment with network access, allowing me to run commands without needing additional approval. My main task is to create the master.md in the root folder, which should include a block diagram, an execution plan, and a grounded explanation of how the plan minimizes WER using various techniques like ensemble methods and audio-grounded verification.


**Formulating the explanation**

I plan to state that using an ensemble approach with verification yields lower WER than a single model due to error diversity and consensus, supported by references. I'll highlight techniques like ROVER to reduce WER and gating to prevent hallucinations. I need to be cautious with the phrase "minimal WER possible till date," so I'll clarify that given unlimited compute offline, this strategy effectively reduces WER by maximizing model diversity, combining outputs, and verifying against audio. I’ll also mention the constraints on LLMs to avoid hallucinations since they can’t process audio. Lastly, I should add a block diagram using Mermaid or ASCII.


**Planning the updates**

I need to create a mermaid code block for the block diagram and detail an execution plan that includes phases, tasks, timelines, priorities, and gating thresholds. I'll update Agents.md by adding a section for the block diagram near the existing Master Plan or under Do This Next. The execution plan will reference sections like Do This Next and the Implementation Blueprint, but I must also ensure to explain why this plan minimizes WER in both master.md and Agents.md.

Master.md should summarize the problem, state principles, include the block diagram, and detail the execution phases while explaining how the various strategies minimize WER. Finally, I'll be very cautious about not revealing any sensitive information. I'll apply the necessary updates via patching while ensuring I inspect the existing Agents.md document for appropriate placement of content.


rg -n "## Master Plan \(Unified\)" -n Agents.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
504:## Master Plan (Unified)