Output:
Exit code: 0
Wall time: 9606.5 seconds
Total output lines: 486
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 19:20:18.753824: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
`torch_dtype` is deprecated! Use `dtype` instead!
`generation_config` default values have been modified to match model-specific defaults: {'suppress_tokens': [], 'begin_suppress_tokens': [220, 50257]}. If this is not desired, please set these values explicitly.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> to see related `.generate()` flags.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> to see related `.generate()` flags.

Fetching 2 files:   0%|          | 0/2 [00:00<?, ?it/s]
Fetching 2 files:  50%|█████     | 1/2 [00:12<00:12, 12.90s/it]
Fetching 2 files: 100%|██████████| 2/2 [00:12<00:00,  6.49s/it]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.42it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.50it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.24it/s]

Fetching 2 files:   0%|          | 0/2 [00:00<?, ?it/s]
Fetching 2 files:  50%|█████     | 1/2 [00:13<00:13, 13.90s/it]
Fetching 2 files: 100%|██████████| 2/2 [00:13<00:00,  6.95s/it]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.33it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.33it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.09it/s]

Fetching 2 files:   0%|          | 0/2 [00:00<?, ?it/s]
Fetching 2 files:  50%|█████     | 1/2 [00:12<00:12, 12.42s/it]
Fetching 2 files: 100%|██████████| 2/2 [00:12<00:00,  6.21s/it]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.31it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.34it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.09it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.07it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.97it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.75it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  1.51it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.56it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  2.32it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00, …110160 chars truncated… 50%|█████     | 1/2 [00:00<00:00,  2.15it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.42it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.14it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.12it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.40it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.11it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.16it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.45it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.17it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.08it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.36it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.08it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.08it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.39it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.10it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.10it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.36it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.08it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.09it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.34it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.06it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.10it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.34it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.06it/s]

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards:  50%|█████     | 1/2 [00:00<00:00,  2.11it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.36it/s]
Loading checkpoint shards: 100%|██████████| 2/2 [00:00<00:00,  3.09it/s]
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 2:39:53
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani/results.jsonl
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani/summary.json
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 692,
  "wer_mean": 0.9472045966164909,
  "cer_mean": 0.8145643353155686,
  "wer_p50": 1.0,
  "wer_p90": 1.0833333333333333,
  "wer_p95": 1.2,
  "cer_p50": 1.0,
  "cer_p90": 1.0,
  "cer_p95": 1.0461538461538462,
  "segment_error_rate": 0.9985549132947977,
  "pct_wer_gt_5pct": 0.9985549132947977,
  "pct_wer_gt_10pct": 0.9985549132947977,
  "wer_by_language_mean": {
    "asm": 1.0340197818599366,
    "ben": 0.7649698940515266,
    "eng": 1.0,
    "guj": 1.0,
    "kan": 1.0,
    "mal": 1.0291666666666666,
    "mar": 0.9106511845642281,
    "ory": 1.0910669919929181,
    "pan": 1.0,
    "tam": 0.6187301587301587,
    "tel": 0.9092241092241092
  },
  "wer_by_decision_mean": {
    "accept": 0.8188852652515446,
    "human": 0.9655085270555523,
    "reject": 1.0250008548586749
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.995572454841771,
    "10-30s": 1.0194464506634158,
    "3-10s": 0.862549973259891
  },
  "cer_by_language_mean": {
    "asm": 0.9557039595095098,
    "ben": 0.4789955514801865,
    "eng": 1.0,
    "guj": 1.0,
    "kan": 1.0,
    "mal": 0.9901859609249734,
    "mar": 0.526638648951847,
    "ory": 1.040861869918065,
    "pan": 0.8813559322033898,
    "tam": 0.2648010370928059,
    "tel": 0.5873585229237184
  },
  "cer_by_decision_mean": {
    "accept": 0.5848086448920116,
    "human": 0.8075798994698656,
    "reject": 1.0120817259751598
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.9417132963378172,
    "10-30s": 0.8604324138772151,
    "3-10s": 0.6088391541059958
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.8602210671031469,
    "selection_gap_wer_mean": 0.08698352951334401,
    "selection_gap_wer_p90": 0.3333333333333333
  }
}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Analyzing evaluation results**

I'm taking a look at the evaluation results, and wow, they’re not great. It seems like only 692 pairs have been evaluated, which suggests there might be many errors or even missing final text. That’s concerning. It looks like the WER (Word Error Rate) is showing 1.0 for many languages, which indicates the evaluation might only be on a subset due to these errors. I’ll need to inspect the summary.json for more details.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "segments_total": 2400,
  "counts": {
    "accept": 172,
    "human": 309,
    "reject": 211,
    "error": 1708
  },
  "avg_total_sec": 4.784006895194627,
  "p50_total_sec": 3.6584825259633362,
  "p90_total_sec": 10.181923252996057
}


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <approval_policy>on-request</approval_policy>
  <sandbox_mode>workspace-write</sandbox_mode>
  <network_access>restricted</network_access>
  <shell>bash</shell>
</environment_context>

<environment_context>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>

# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
1) Code readiness for open benchmarking

Overall yes: you now have reproducible IDs (ids.py (line 16)), audit-friendly outputs (pipeline.py (line 262)), and eval that reports WER+CER+tails+oracle gap (eval.py (line 54)).
One critical check before benchmarking with MMS CTC: facebook/mms-1b-all (and similar) needs processor.tokenizer.set_target_lang(<iso3>) before tokenizing text; otherwise Indic scripts become mostly <unk> and CTC scoring will silently fail via too_many_unk_tokens. I don’t see any set_target_lang call in ctc_fit.py (line 101) yet. If you’re using MMS as the fallback CTC model, patch this (or ensure your ctc_model_map always points to monolingual CTC models that don’t require it).
Benchmarking will work best only if you actually run ≥2 diverse ASR hypotheses; otherwise ROVER/CTC selection is meaningless (watch ensemble.warnings in results.jsonl).
2) Benchmarks/datasets to use (internet findings)

AI4Bharat Vistaar (benchmarks + downloads + baseline WER tables): https://github.com/AI4Bharat/vistaar
Includes benchmark zips for Kathbath, Kathbath-Hard, FLEURS, CommonVoice, IndicTTS, MUCS, GramVaani and reports IndicWhisper WER (e.g., overall avg WER 24.6%; Hindi subset avg 13.6%) in the repo README.
Direct benchmark downloads are linked in their README (example): https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/kathbath.zip (same pattern for others).
For Assamese + English (not covered in Vistaar’s 12-language table): use google/fleurs and Common Voice (HuggingFace datasets) and evaluate the same way.
Optional additional benchmark: collabora/indic-superb (HF dataset; has audio + transcription, but is large).
3) Handover: how to run benchmarks + analyze language patterns (no new files)

Download + unpack one benchmark (Vistaar example):
kathbath.zip
unzip kathbath_bench.zip -d bench/vistaar/kathbath
Create refs.jsonl for one language folder (Vistaar structure is *.wav + transcript.txt per their README). Run this from repo root:
python3 - <<'PY'
from pathlib import Path
import json
from maya_transcribe.ids import segment_identity

lang_dir = Path("bench/vistaar/kathbath/hindi")  # adjust
audio_dir = lang_dir / "audio"
txt_path = lang_dir / "transcript.txt"
out_path = lang_dir / "refs.jsonl"

def parse_line(line: str):
    line=line.strip()
    for sep in ("\t","|"," ",","):
        if sep in line:
            a,b=line.split(sep,1)
            return a.strip(), b.strip()
    return None

rows=[]
for line in txt_path.read_text(encoding="utf-8").splitlines():
    if not line.strip():
        continue
    parsed=parse_line(line)
    if not parsed:
        continue
    fname, text = parsed
    src = audio_dir / fname
    ident = segment_identity(src, root=audio_dir)
    rows.append({"segment_id": ident.segment_id, "text": text})

out_path.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n", encoding="utf-8")
print("wrote", out_path, "rows", len(rows))
PY
Run pipeline + eval (start with no CTC to validate end-to-end; then enable CTC):
maya-transcribe run --segments-dir bench/vistaar/kathbath/hindi/audio --out bench_runs/kathbath_hi --whisper-model large-v3 \
  --transformers-whisper-model <indic_whisper_hf_id>

maya-transcribe eval --results bench_runs/kathbath_hi/results.jsonl --refs bench/vistaar/kathbath/hindi/refs.jsonl --out bench_runs/kathbath_hi/eval.json
Pattern analysis to tune (estimate human workload + where WER comes from):
python3 - <<'PY'
import json, collections
p="bench_runs/kathbath_hi/results.jsonl"
dec=collections.Counter(); reasons=collections.Counter()
for line in open(p,encoding="utf-8"):
    r=json.loads(line)
    v=r.get("verification",{})
    dec[v.get("decision")] += 1
    for reason in (v.get("reasons") or []):
        reasons[reason]+=1
print("decisions:",dec)
print("top reasons:",reasons.most_common(20))
PY
Repeat per language/dataset; use eval.json + oracle stats to decide:
High oracle_best_wer ⇒ you need a better base ASR for that language/domain.
High selection_gap_wer_mean ⇒ selection/gating (ROVER/CTC thresholds) needs tuning or the CTC model is mismatched.
If you tell me which exact datasets you want first (Vistaar-only vs Vistaar+FLEURS for Assamese/English), I can give a minimal “benchmark matrix” (languages × datasets × model configs) to run as ablations.

can you run the benchmarks with few 100s of samples for each language. maybe if it's nto available in a single dataset, please pull from various sources but in overall, like get some 1000-2000 samples per each language and do a benchmark with our pipeline. goal is minimal WER as possible, if particular languages show higher, analyse why ? and fix or tweak them but make it adjustable tweakable and make it robust. this is the place for us to run the automated benchmarking tests on our pipeline here. Constantly keep evaluating, evolving and only conclude when you feel like this is the lowest WER possible and i refuse to believe until we're like at 2.0 wer or achieve even lesser. just validated that i now have access to shrutilipi, svarah, kathbath and etc.

And to make it even easier, create a new benchmark dataset under our huggingface ID using the key in .env, ensure it's private.

more info on datasets to choose samples from huggingface.

For the benchmark datasets we discussed, these are the Hugging Face dataset repos to check access for:

AI4Bharat (these are gated=auto → you must be logged in and click “Agree/Request access” on HF):

ai4bharat/Kathbath
ai4bharat/IndicVoices
ai4bharat/indicvoices_r
ai4bharat/Lahaja
ai4bharat/Svarah
(optional, not ASR) ai4bharat/IndicVoices-ST
Open / ungated (should work without requesting access):

google/fleurs
mozilla-foundation/common_voice_17_0 (or whichever CV version you use)
collabora/indic-superb
Not needed on HF if you use Vistaar’s benchmark zips: the Vistaar benchmark suite is downloadable directly (no HF dataset access required).

Quick “do I actually have access?” check (doesn’t download everything):

python3 -c "from datasets import get_dataset_config_names; print(get_dataset_config_names('ai4bharat/Kathbath'))"
then try streaming one split: python3 -c "from datasets import load_dataset; ds=load_dataset('ai4bharat/Kathbath','hindi',split='validation',streaming=True); print(next(iter(ds)))"

# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
First, ignore the maya_hf_v0 WER for model decisions — that corpus had duplicate segment_ids (2335 unique for 2400 rows), so refs/audio mismatched; use data/bench/maya_hf_v2.
Next ASR checkpoints to try (all gated=False)

Single multilingual add-on (lowest effort): Vignesh-M/Indic-whisper (pass as --transformers-whisper-model).
Per-language Whisper (best shot at lower WER; ideally via a model-map so you don’t run 12 models per clip):
hin: ARTPARK-IISc/whisper-large-v3-vaani-hindi
tel: ARTPARK-IISc/whisper-large-v3-vaani-telugu
ory: ARTPARK-IISc/whisper-large-v3-vaani-odia
ben: ARTPARK-IISc/whisper-medium-vaani-bengali
kan: ARTPARK-IISc/whisper-medium-vaani-kannada
tam: vasista22/whisper-tamil-large-v2
mal: CXDuncan/whisper-large-v3-malayalam
mar: durgesh10/whisper-large-v3-marathi
guj: vasista22/whisper-gujarati-medium
pan: DrishtiSharma/whisper-large-v2-punjabi
asm: DrishtiSharma/whisper-large-v2-assamese-5k-steps
eng: keep faster-whisper:large-v3 (or openai/whisper-large-v3 via transformers if you want a redundant cross-check)


Summary: auto


**Evaluating benchmarking approach**

The user's request is clear: they want to run benchmarks with a few hundred samples for each language, aiming for about 1,000-2,000 samples from various sources if needed. The main goal is to minimize word error rate (WER). If some languages show higher WER, I'll need to analyze and make adjustments to enhance performance and make the process robust. I'll keep evaluating until we reach a WER of 2.0 or lower, and I see they now have access to specific datasets.


**Clarifying benchmark requirements**

The plan is to create a new benchmark dataset under our Hugging Face ID with a private key from the .env file. This task could involve significant downloads, compute power, and iterative tuning, but I'm determined to make progress. We want to build the dataset from multiple Hugging Face datasets and run evaluations for each language. I'm a bit confused by the WER target; if they mean 2.0%, that makes sense, but achieving 0.02% seems unrealistic for ASR on Indic languages. I'll interpret it as aiming for 2% WER.


**Analyzing WER expectations**

It's clear that achieving a word error rate (WER) as low as 2% is unrealistic with current models since typical WER for large-v3 is around 10-30%. So, we need to set realistic expectations. The user wants aggressive targets, but with a clean dataset and specialized models, we might have some leeway. My immediate tasks include fixing model-map run errors and identifying the root causes, likely related to gating errors. We need to ensure the correct language tokens are being used, focusing on the iso3 and iso639-1 codes while checking our language mappings in `maya_transcribe/language.py`.


**Investigating language routing issues**

The per-language router uses the language passed to the `transcribe` call, which might be the iso3 code (like 'hin') set by the benchmark. I need to determine if we're passing this as iso3 or converting it to iso639-1. Some fine-tuned models might require specific language codes, like "hi." If our code passes 'hin' to `processor.get_decoder_prompt_ids`, it may cause a KeyError, leading to errors. I should inspect the repo, especially the AGENTS instructions, to understand the structure and check for an AGENTS.md file with more guidance.


**Reviewing repository instructions**

We need to stick to the instructions, but there’s no mention of using any skills like skill-creator or skill-installer for our task of benchmarking and dataset building. Since none match, I can proceed without them. The instructions indicate using a skill only if the user names or tasks fit the description, which isn’t the case. I’ll check the repository files, starting with `ls` to scan for AGENTS.md, as it might provide the necessary instructions. I'll use shell commands for this.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 112
drwxrwxr-x  8 ubuntu ubuntu  4096 Jan 13 06:26 .
drwxr-x--- 43 ubuntu ubuntu  4096 Jan 13 15:47 ..
-rw-rw-r--  1 ubuntu ubuntu 41725 Jan 13 04:08 Agents.md
drwxrwxr-x  4 ubuntu ubuntu  4096 Jan 13 07:15 data
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 03:12 docs
-rw-rw-r--  1 ubuntu ubuntu  3704 Jan 13 05:07 .env
-rw-rw-r--  1 ubuntu ubuntu   144 Jan 13 04:16 .gitignore
-rw-rw-r--  1 ubuntu ubuntu  9299 Jan 13 04:09 master.md
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 13 07:11 maya_transcribe
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 04:29 maya_transcribe.egg-info
drwxrwxr-x 12 ubuntu ubuntu  4096 Jan 13 07:31 outputs
-rw-rw-r--  1 ubuntu ubuntu  1090 Jan 13 06:13 Prompt.md
-rw-rw-r--  1 ubuntu ubuntu  1140 Jan 13 07:12 pyproject.toml
-rw-rw-r--  1 ubuntu ubuntu  1280 Jan 13 05:39 README.md
-rw-rw-r--  1 ubuntu ubuntu  2627 Jan 13 02:35 task.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 06:26 .vscode


**Clarifying file names**

It looks like there's an "Agents.md" file open, but I'm noticing that "AGENTS.md" doesn't seem to exist. The user's instructions referenced "AGENTS.md instructions," so I'm wondering if they meant the "Agents.md" file instead. I think I should check for the presence of "AGENTS.md" just to be thorough. It's always good to verify the file names to make sure I’m following the correct instructions. Let’s search for "AGENTS.md" to see what comes up!


find .. -maxdepth 3 -name 'AGENTS.md' -o -name 'Agents.md'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
../maya-transcribe/Agents.md


**Evaluating file instructions**

I need to focus on the difference between Agents.md and AGENTS.md. The developer's instructions specify AGENTS.md, but the only available file is Agents.md. I want to check what's inside to see if it contains any useful information, even if it's not named as specified. I should still consider any relevant guidelines found in Agents.md. Okay, let’s open it and figure out what instructions it holds!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Agents.md — Meta-Prompt + Workflow for “Reference Ensemble” Research (ASR Transcription Quality First)

## Role
You are a **coding + research agent** operating inside a repo/workspace that contains:
- `task.md` (the original problem statement + requirements)
- `Reference1.md` … `Reference6.md` (different LLM-proposed approaches + code snippets)
- Any supporting code/config files already present

Your job is to **read everything**, extract the best ideas, **reject weak or risky suggestions**, and synthesize a **single optimal plan** to produce **near-zero-error transcripts** for multi-lingual audio (offline; compute not a constraint; quality is).

---

## Non-Negotiables
1. **No invention.** Only claim something is “recommended” if it is supported by the references or is a clearly-labeled inference.
2. **Quality > Coverage.** If forced to choose, reject questionable segments rather than poison training data.
3. **Audio-grounded verification required.** Any transcript must be **verified against the audio** before being accepted.
4. **Make decisions.** For every major technique, explicitly decide: **Adopt / Adapt / Reject / Park** and justify.
5. **Keep this file updated.** This file (`Agents.md`) is the living artifact: top section = this prompt; below it = your analysis, TODOs, decisions, and final plan.

---

## Output Contract (What you must produce in THIS file)
By the end, `Agents.md` must contain:
- A crisp understanding of `task.md` (constraints, metrics, languages, tooling assumptions)
- A per-reference analysis for Reference1–Reference6:
  - Summary, unique contributions, strengths, failure modes, missing pieces
  - Concrete actionable items + what to ignore
  - Adopt/Adapt/Reject/Park decisions
- A unified **Master Plan** (pipeline + thresholds + verification + human loop)
- An implementation blueprint (data structures, job orchestration, evaluation harness)
- A validation protocol (gold set, WER/CER reporting, active learning loop)

---

## Step 0 — Read Order (do this first)
1. Read `task.md` fully.
2. Skim all `Reference*.md` quickly to build a map of what exists.
3. Then deep-read each reference, one by one, and complete the per-reference template below.

---

## Step 1 — Extract Requirements from `task.md`
Fill this section after reading `task.md`.

### Task Summary (from task.md)
- **Goal:**
- **Languages:**
- **Audio segment length range:**
- **Compute constraints:**
- **Quality target (WER/CER):**
- **Acceptance philosophy (precision-first vs recall-first):**
- **Any required tools/frameworks:**
- **Any banned tools/constraints:**

### Success Metrics
Define what “good”” means in numbers and gates.
- Primary: (WER / CER / SER)
- Secondary: rejection rate, human edit rate, alignment confidence, hallucination rate
- Must-meet thresholds:
  - **Max allowable WER/CER for accepted samples:**
  - **Max disagreement allowed between systems:**
  - **Minimum alignment confidence:**

### Assumptions (explicit)
List assumptions you will rely on (and mark what needs verification).
- A1:
- A2:
- A3:

---

## Step 2 — Create an Evaluation Rubric (used for every idea)
For each technique/tool/model suggested in references, score it:

**Scoring dimensions (0–5):**
- Expected quality gain
- Robustness across Indic languages + code-switching
- Verification friendliness (supports confidence/alignment scoring)
- Implementation complexity
- Failure risk (hallucinations, bias, brittle dependencies)

**Decision labels:**
- **Adopt** = use as-is
- **Adapt** = use but modify (thresholds, different model, narrowed scope)
- **Reject** = do not use
- **Park** = keep as optional experiment

---

## Step 3 — Per-Reference Processing Template (repeat for each)
For each `ReferenceN.md`, fill:

### ReferenceN — One-paragraph summary
- What is the approach in plain terms?

### Key Contributions (unique ideas worth preserving)
- C1:
- C2:
- C3:

### Strengths (why it could work well)
- S1:
- S2:

### Risks / Failure Modes
Be harsh and specific.
- R1:
- R2:

### Missing Pieces / Gaps
- G1:
- G2:

### Actionable Extract (convert into implementable tasks)
Each item must be concrete.
- [ ] A1:
- [ ] A2:

### Decisions
For each major suggestion:
- Suggestion: …
  - Decision: Adopt / Adapt / Reject / Park
  - Why:
  - What to do instead (if rejected):

### Integration Notes
How this reference fits with others.
- Conflicts with:
- Complements:
- Dependencies:

---

## Step 4 — Synthesis After All References
After finishing Reference1–Reference6, produce:

### Master Plan (single best pipeline)
Must include:
1. **Pre-QC** (audio sanity checks, VAD trimming, clipping/SNR checks)
2. **Language ID routing** (include handling for uncertain/mixed)
3. **ASR ensemble** (per language group; include at least 2 diverse systems)
4. **Normalization layer** (unicode, punctuation, numerals, script normalization)
5. **System combination** (e.g., voting/ROVER or other)
6. **Audio-grounded verification gate** (forced alignment / CTC scoring / confidence)
7. **Reject + re-decode logic** (different decoding params / alternative model)
8. **Human-in-the-loop for rejects** (only the tail; scalable workflow)
9. **Storage schema** (store all hypotheses + scores for auditability)

### Implementation Blueprint (engineering plan)
- Orchestration: batching, GPU scheduling, retries
- Data structures:
  - `segment_id`, `language_probs`, `hypotheses[]`, `rover_output`, `alignment_scores`, `decision`, `audit_log`
- Reproducibility: config-driven pipeline, versioned models, deterministic decoding where possible
- Observability: dashboards for rejection reasons, per-language error patterns

### Validation Protocol (prove it works)
- Build a **gold set** (stratified across languages, speakers, domains, lengths)
- Compute WER/CER by language + conditions
- Threshold tuning approach
- Active learning loop: rejects → human fix → retrain / recalibrate

### Final Deliverable
- A short “Do This Next” checklist that an engineer can execute immediately.

---

## Global TODO Checklist (tick as you complete)
- [x] Read `task.md` and fill **Step 1**
- [x] Create evaluation rubric (**Step 2**) and decision labels
- [x] Complete Reference1 analysis
- [x] Complete Reference2 analysis
- [x] Complete Reference3 analysis
- [x] Complete Reference4 analysis
- [x] Complete Reference5 analysis
- [x] Complete Reference6 analysis
- [x] Write **Master Plan**
- [x] Write **Implementation Blueprint**
- [x] Write **Validation Protocol**
- [x] Produce final “Do This Next” execution checklist

---

# Analysis Workspace (Fill below as you go)
> Everything below this line is your working output. Keep it clean and structured.

## Step 1 — Task Understanding (from task.md)
### Task Summary (from task.md)
- **Goal:** Produce highly accurate, offline transcriptions for cleaned, single-speaker audio segments so they are safe for training ASR/TTS.
- **Languages:** Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English (plus “unknown/unflagged” segments).
- **Audio segment length range:** ~1s up to ~60s max (typical ~30–50s).
- **Compute constraints:** None (can run multiple large models, offline, no latency requirements).
- **Quality target (WER/CER):** “As low as possible” / “can’t afford mistakes” → treat as **precision-first** (accept fewer, near-perfect segments rather than keeping everything with hidden label noise).
- **Acceptance philosophy (precision-first vs recall-first):** **Precision-first**; strict rejection + targeted human correction for the hard tail.
- **Any required tools/frameworks:** Not specified; implied needs: spoken language ID, multi-model ASR, system-combination, and audio-grounded verification (forced alignment / CTC scoring).
- **Any banned tools/constraints:** Not explicit, but the user explicitly worries about **hallucinations** → avoid any step that can invent content (especially text-only LLM “transcription”).

### Success Metrics
Define “good” in numbers and gates (must be tuned on a gold set; initial values are starting heuristics, not guarantees).
- **Primary:** WER by language (space-delimited) + CER by language (robust for Indic scripts), reported on a stratified gold set.
- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).


ls -ლა outputs/bench/maya_hf_v2/per_lang_vaani | head


Output:
Exit code: 0
Wall time: 0 seconds
Output:
ls: invalid option -- 'á'
Try 'ls --help' for more information.


**Revising encoding issues**

I noticed that the command had some weird `-লা` characters due to encoding, and I'm wondering if I typed " -la" but maybe used Arabic or something else, like " -ලා"? It seems like it's mis-encoded, which can happen sometimes. Anyway, I'll go ahead and rerun the command using `ls -la` to get the right output. Let's see if that clears things up!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 5516
drwxrwxr-x 3 ubuntu ubuntu    4096 Jan 13 22:00 .
drwxrwxr-x 7 ubuntu ubuntu    4096 Jan 13 19:20 ..
drwxrwxr-x 2 ubuntu ubuntu  290816 Jan 13 22:00 audio
-rw-rw-r-- 1 ubuntu ubuntu    1753 Jan 13 22:00 eval.json
-rw-rw-r-- 1 ubuntu ubuntu 5340563 Jan 13 22:00 results.jsonl
-rw-rw-r-- 1 ubuntu ubuntu     238 Jan 13 22:00 summary.json


**Inspecting summary.json**

I want to take a quick look at `summary.json` to see what data or insights it holds. It’s crucial to check details like the structure and the content to better understand what’s represented there. I’ll focus on whether it aligns with expectations and serves its intended purpose. Gathering this information will help me ensure that everything is in order and functioning accurately!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "segments_total": 2400,
  "counts": {
    "accept": 172,
    "human": 309,
    "reject": 211,
    "error": 1708
  },
  "avg_total_sec": 4.784006895194627,
  "p50_total_sec": 3.6584825259633362,
  "p90_total_sec": 10.181923252996057
}


**Parsing error messages**

I see there's an error noted as 1708. I need to dig into the `results.jsonl` file and filter for lines that include either "error" or some verification decision that might imply an error. It seems like each line should contain an "error" field. I'll use Python to quickly count the different types of errors, so I can better understand the issue. Time to run a quick script!


python3 - <<'PY'
import json, collections
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/per_lang_vaani/results.jsonl')
errs=collections.Counter(); langs=collections.Counter(); models=collections.Counter(); examples={}
for line in p.open(encoding='utf-8'):
    r=json.loads(line)
    if r.get('error'):
        e=r['error']
        # normalize to first line
        key=e.split('\n',1)[0]
        errs[key]+=1
        lang=r.get('language') or r.get('lang') or r.get('ref_language') or ''
        if lang: langs[lang]+=1
        m=r.get('asr',{}).get('transformers_whisper_model') or r.get('asr',{}).get('model') or ''
        if m: models[m]+=1
        if key not in examples:
            examples[key]=r.get('segment_path') or r.get('segment_id')
print('total lines', sum(1 for _ in p.open(encoding='utf-8')))
print('errors', sum(errs.values()))
print('top error types:')
for k,v in errs.most_common(20):
    print(v, k)
print('top langs (errors):', langs.most_common(20))
print('top models (errors):', models.most_common(10))
print('examples:')
for k in list(examples)[:10]:
    print(k, '->', examples[k])
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Total output lines: 36
Output:
total lines 2400
errors 1708
top error types:
204 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:guj': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 15.04 GiB is allocated by PyTorch, and 472.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}
189 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:kan': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 15.04 GiB is allocated by PyTorch, and 472.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}
180 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:hin': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 7.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.89 GiB is allocated by PyTorch, and 626.21 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}
173 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:mal': 'OSError:CXDuncan/whisper-large-v3-malayalam does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.'}
117 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:pan': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 13.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.92 GiB is allocated by PyTorch, and 594.46 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}
116 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:tam': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 13.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.92 GiB is allocated by PyTorch, and 594.46 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}
115 no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:ory': "ValueError:Unsupported language: or. Language should be one of: ['en', 'zh', 'de', 'es', 'ru', 'ko', 'fr', 'ja', 'pt', 'tr', 'pl', 'ca', 'nl', 'ar', 'sv', 'it', 'id', …13944 chars truncated…_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:guj': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 15.04 GiB is allocated by PyTorch, and 472.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'} -> Kathbath__gujarati__valid__0__844424931508725-889-f.m4a-1b08709a2a42242e
no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:hin': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 7.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.89 GiB is allocated by PyTorch, and 626.21 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'} -> Kathbath__hindi__valid__101__844424931153739-164-f.m4a-2ae6a38a9e2e94ec
no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:pan': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.97 GiB is allocated by PyTorch, and 545.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'} -> Kathbath__hindi__valid__996__844424932834336-727-f.m4a-eaec03309e81d9f6
no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:kan': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 15.04 GiB is allocated by PyTorch, and 472.17 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'} -> Kathbath__kannada__valid__1013__844424930400134-148-f.m4a-655fbc01cff5ac55
no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:tel': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 128.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 11.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.97 GiB is allocated by PyTorch, and 545.07 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'} -> Kathbath__kannada__valid__1203__844424930596382-320-m.m4a-60ef2c3cd92ceb36
no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:mal': 'OSError:CXDuncan/whisper-large-v3-malayalam does not appear to have a file named pytorch_model.bin, model.safetensors, tf_model.h5, model.ckpt or flax_model.msgpack.'} -> Kathbath__malayalam__valid__1012__844424930363179-355-m.m4a-7fac745d732047cc


**Evaluating model loading issues**

I need to check if I'm supposed to use `model.safetensors`, but I'm getting a file not found error. This might mean it’s not a transformers model, possibly needing `ctranslate2` or some other option. I wonder if I should set `trust_remote_code=True` too. The main issue seems to be out-of-memory errors because models aren't offloaded or cached. I'll need to design a router to manage GPU memory effectively. My plan might include loading one transformer model at a time and running sequentially per language.


**Optimizing model loading**

I think our current router caches models for each language in memory. If I keep all of them loaded, it could lead to out-of-memory (OOM) issues. I should modify the setup to limit GPU memory usage by keeping model objects on the CPU and only moving them to the GPU as needed. However, switching large models between CPU and GPU might slow things down. A better approach could be to process language segments in chunks—load a model, run for that language, then unload before moving to the next one.


**Refining model management**

I think I can modify the router to unload the previous GPU model when switching languages, keeping only one active model in the GPU at a time. For benchmarking, running sequentially allows this. My first option is to load a new model and potentially unload the previous one when calling `transcribe` for a new language. I could use `torch.cuda.empty_cache`, but I should be cautious since it doesn't free memory currently occupied by model weights. I will `del model`, run `gc.collect()`, and then call `empty_cache()` to free memory, although there might be some fragmentation issues. Plus, I need to remember that we’re also running the faster-whisper large-v3 model concurrently, which uses GPU memory as well.


**Analyzing memory usage**

I’m pondering whether we actually need to run the faster-whisper base model alongside our transformer per-language models. It might still be useful for providing a diverse second hypothesis. However, if I run the fast-whisper large-v3 along with the transformer large model, I'm concerned about memory usage since it should fit in 80GB. Right now, an error shows that there’s only 11MB free, even though the process uses 60GB, plus an additional 19GB for non-PyTorch memory. I need to clarify why the memory report seems off, particularly with the PyTorch allocation of 15GB compared to the total usage.


**Evaluating memory constraints**

It seems we’ve loaded multiple transformer models, each using about 10-15GB, plus faster-whisper, which needs over 20GB. The large-v3 ctranslate2 model might be around 9GB. When I roughly compute the sizes, the Whisper large-v3 has about 1.55B parameters, which translates to roughly 3GB for half-precision weights and around 5-8GB for intermediate memory. If I load 11 models at once, that’s over 50GB, which definitely leads to OOM issues.

To fix this, I should implement an LRU cache to limit how many models are kept in GPU memory, maybe just one or two. This means I could then move older models to the CPU or unload them altogether. I want to minimize word error rate (WER) and can run multiple transformer models, but running them separately for each language may help reduce memory usage.


find data/bench/maya_hf_v2 -maxdepth 3 -type d | head -n 50


Output:
Exit code: 0
Wall time: 0 seconds
Output:
data/bench/maya_hf_v2
data/bench/maya_hf_v2/audio
data/bench/maya_hf_v2/audio/hin
data/bench/maya_hf_v2/audio/ben
data/bench/maya_hf_v2/audio/eng
data/bench/maya_hf_v2/audio/guj
data/bench/maya_hf_v2/audio/tel
data/bench/maya_hf_v2/audio/kan
data/bench/maya_hf_v2/audio/mar
data/bench/maya_hf_v2/audio/ory
data/bench/maya_hf_v2/audio/pan
data/bench/maya_hf_v2/audio/tam
data/bench/maya_hf_v2/audio/mal
data/bench/maya_hf_v2/audio/asm


Summary: auto


**Evaluating audio processing**

I'm looking into how audio files are organized by iso3 language code directories. It seems like `bench run` processes files found in `data/bench/maya_hf_v2/audio`, potentially including nested directories. The pipeline likely finds audio files recursively, and we can group segments by language. However, they want a larger dataset—around 1000-2000 samples per language instead of just 200. We might need to use the dataset builder to create a bigger corpus from various sources, but first, I’ll address some memory issues before running the models again.


**Inspecting GPU memory and caching**

I’m thinking about how to check the GPU memory with `nvidia-smi`, but I realize that it’s only a temporary state. So, I’ll take a look at the `maya_transcribe/asr.py` router to understand the current caching behavior better. It seems like diving into that relevant file will give me the insights I need to improve things. Let’s see what I can find there!


rg -n "class TransformersWhisperASRRouter" -n maya_transcribe/asr.py && sed -n '1,220p' maya_transcribe/asr.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
235:class TransformersWhisperASRRouter:
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Any

from maya_transcribe.language import to_whisper_lang

@dataclass(frozen=True)
class ASRSegment:
    start: float
    end: float
    text: str
    avg_logprob: float | None = None
    no_speech_prob: float | None = None
    compression_ratio: float | None = None

@dataclass(frozen=True)
class ASRResult:
    model: str
    text: str
    language: str | None
    language_probability: float | None
    segments: list[ASRSegment]
    meta: dict[str, Any]

class FasterWhisperASR:
    def __init__(
        self,
        model_size_or_path: str = "large-v3",
        device: str | None = None,
        compute_type: str | None = None,
    ) -> None:
        try:
            from faster_whisper import WhisperModel
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "faster-whisper is not installed. Install with: pip install -e '.[asr]'"
            ) from e

        if device is None:
            try:
                import torch

                device = "cuda" if torch.cuda.is_available() else "cpu"
            except Exception:
                device = "cpu"

        if compute_type is None:
            compute_type = "float16" if device == "cuda" else "float32"

        self.model_size_or_path = model_size_or_path
        self.device = device
        self.compute_type = compute_type
        self._model = WhisperModel(model_size_or_path, device=device, compute_type=compute_type)

    @property
    def name(self) -> str:
        return f"faster-whisper:{self.model_size_or_path}"

    def transcribe(
        self,
        audio_path: Path,
        language: str | None = None,
        beam_size: int = 5,
        best_of: int = 5,
        temperature: float = 0.0,
        vad_filter: bool = False,
    ) -> ASRResult:
        lang = to_whisper_lang(language)
        if lang is not None:
            # faster-whisper supports a subset of Whisper's language tokens.
            # Avoid hard failures (e.g., Odia "or") by falling back to auto-detect.
            try:
                from faster_whisper.tokenizer import _LANGUAGE_CODES  # type: ignore

                if lang not in _LANGUAGE_CODES:
                    lang = None
            except Exception:
                pass

        segments_iter, info = self._model.transcribe(
            str(audio_path),
            language=lang,
            beam_size=beam_size,
            best_of=best_of,
            temperature=temperature,
            vad_filter=vad_filter,
        )

        segments: list[ASRSegment] = []
        texts: list[str] = []
        for s in segments_iter:
            texts.append(s.text)
            segments.append(
                ASRSegment(
                    start=float(s.start),
                    end=float(s.end),
                    text=str(s.text),
                    avg_logprob=getattr(s, "avg_logprob", None),
                    no_speech_prob=getattr(s, "no_speech_prob", None),
                    compression_ratio=getattr(s, "compression_ratio", None),
                )
            )

        return ASRResult(
            model=self.name,
            text="".join(texts).strip(),
            language=getattr(info, "language", None),
            language_probability=getattr(info, "language_probability", None),
            segments=segments,
            meta={
                "device": self.device,
                "compute_type": self.compute_type,
                "beam_size": beam_size,
                "best_of": best_of,
                "temperature": temperature,
                "vad_filter": vad_filter,
            },
        )

class TransformersWhisperASR:
    """
    Whisper-family ASR using HuggingFace Transformers (useful for fine-tuned checkpoints like IndicWhisper).
    """

    def __init__(
        self,
        model_id: str,
        device: str | None = None,
        dtype: str | None = None,
    ) -> None:
        try:
            import torch
            from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "TransformersWhisperASR requires the optional dependencies: torch + transformers. "
                "Install with: pip install -e '.[torch]'"
            ) from e

        if device is None:
            device = "cuda" if torch.cuda.is_available() else "cpu"
        if dtype is None:
            dtype = "float16" if (device == "cuda") else "float32"

        torch_dtype = torch.float16 if dtype == "float16" else torch.float32

        self.model_id = model_id
        self.device = device
        self.dtype = dtype
        self._torch = torch
        self._processor = AutoProcessor.from_pretrained(model_id)
        self._model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype).to(device)
        self._model.eval()

    @property
    def name(self) -> str:
        return f"transformers:{self.model_id}"

    def transcribe(
        self,
        audio_path: Path,
        language: str | None = None,
        max_new_tokens: int = 256,
        num_beams: int = 5,
    ) -> ASRResult:
        from maya_transcribe.audio import decode_pcm16

        wav, sr = decode_pcm16(audio_path, sample_rate=16000, mono=True)
        if wav.size == 0:
            return ASRResult(
                model=self.name,
                text="",
                language=language,
                language_probability=None,
                segments=[],
                meta={"max_new_tokens": max_new_tokens, "num_beams": num_beams},
            )

        inputs = self._processor(wav, sampling_rate=sr, return_tensors="pt")
        input_features = inputs.get("input_features")
        if input_features is None:
            raise RuntimeError("Processor did not return input_features")
        input_features = input_features.to(self.device)
        model_dtype = getattr(self._model, "dtype", None)
        if model_dtype is not None:
            input_features = input_features.to(dtype=model_dtype)

        forced_decoder_ids = None
        whisper_lang = to_whisper_lang(language)
        if whisper_lang:
            get_ids = getattr(self._processor, "get_decoder_prompt_ids", None)
            if callable(get_ids):
                forced_decoder_ids = get_ids(language=whisper_lang, task="transcribe")

        with self._torch.no_grad():
            gen_kwargs = {
                "max_new_tokens": max_new_tokens,
                "num_beams": num_beams,
            }
            if forced_decoder_ids is not None:
                gen_kwargs["forced_decoder_ids"] = forced_decoder_ids
            try:
                generated_ids = self._model.generate(input_features, **gen_kwargs)
            except ValueError as e:
                # Some fine-tuned checkpoints / transformer versions reject `forced_decoder_ids`.
                msg = str(e)
                if "forced_decoder_ids" in msg and "not used by the model" in msg:
                    gen_kwargs.pop("forced_decoder_ids", None)
                    generated_ids = self._model.generate(input_features, **gen_kwargs)
                else:
                    raise

        text = self._processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
        return ASRResult(


sed -n '220,420p' maya_transcribe/asr.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        return ASRResult(
            model=self.name,
            text=text.strip(),
            language=whisper_lang or language,
            language_probability=None,
            segments=[],
            meta={
                "device": self.device,
                "dtype": self.dtype,
                "max_new_tokens": max_new_tokens,
                "num_beams": num_beams,
            },
        )

class TransformersWhisperASRRouter:
    """
    Lazily load and cache Whisper-family HF checkpoints by language.
    Useful for per-language fine-tuned models without running 10+ models per segment.
    """

    def __init__(
        self,
        *,
        model_map: dict[str, str],
        default_model: str | None = None,
        device: str | None = None,
        dtype: str | None = None,
    ) -> None:
        self.model_map = {str(k).strip().lower(): str(v).strip() for k, v in (model_map or {}).items() if str(v).strip()}
        self.default_model = default_model.strip() if isinstance(default_model, str) and default_model.strip() else None
        self.device = device
        self.dtype = dtype
        self._cache: dict[str, TransformersWhisperASR] = {}

    def _get_model_id(self, language: str | None) -> str | None:
        if language:
            key = language.strip().lower()
            if key in self.model_map:
                return self.model_map[key]
        return self.default_model

    def transcribe(
        self,
        audio_path: Path,
        *,
        language: str | None = None,
        max_new_tokens: int = 256,
        num_beams: int = 5,
    ) -> ASRResult | None:
        model_id = self._get_model_id(language)
        if not model_id:
            return None

        backend = self._cache.get(model_id)
        if backend is None:
            backend = TransformersWhisperASR(model_id=model_id, device=self.device, dtype=self.dtype)
            self._cache[model_id] = backend

        return backend.transcribe(
            audio_path=audio_path,
            language=language,
            max_new_tokens=max_new_tokens,
            num_beams=num_beams,
        )


**Considering model caching**

I'm realizing that the router caching models indefinitely is leading to out-of-memory issues. I need to implement a solution like LRU or set a limit on loaded models. A simple fix might be to add a `max_loaded` parameter to the router, with a default of maybe 1, but I want to preserve prior behavior. For benchmarking, we should limit it, but for general use, keeping 2 models might be ideal. Still, using a safe default of 1 to avoid OOM seems wise. I also need to consider segment loading times and possibly suggest running by language to mitigate performance hits.


**Inspecting pipeline code**

I’m planning to check out the `maya_transcribe/pipeline.py` file since it constructs the ASR backend and handles segment looping. This is important because understanding how these components work together will help me manage aspects like model loading and performance. I want to be sure I'm seeing the flow clearly and taking note of any potential issues. Diving into this code could clarify how to optimize for better resource management!


rg -n "transformers_whisper_model_map" -n maya_transcribe/pipeline.py && sed -n '1,220p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
60:    transformers_whisper_model_map: dict[str, str] | None = None,
112:    if transformers_whisper_model_map:
114:            model_map=transformers_whisper_model_map, default_model=transformers_whisper_default_model
from __future__ import annotations

import json
import time
from dataclasses import asdict
from pathlib import Path
from typing import Any

from rich.progress import track

from maya_transcribe.asr import ASRResult, FasterWhisperASR, TransformersWhisperASR, TransformersWhisperASRRouter
from maya_transcribe.audio import compute_qc, ffprobe, normalize_audio
from maya_transcribe.ids import segment_identity
from maya_transcribe.language import LANG_CODE_TO_NAME
from maya_transcribe.lid import LIDResult, MMSLID
from maya_transcribe.rover import rover_combine
from maya_transcribe.textnorm import normalize_text
from maya_transcribe.verify import verify_segment

def _iter_audio_files(segments_dir: Path) -> list[Path]:
    audio_files: list[Path] = []
    for ext in ("*.wav", "*.mp3", "*.m4a", "*.flac", "*.ogg", "*.opus"):
        audio_files.extend(sorted(segments_dir.rglob(ext)))
    return sorted(set(audio_files))

def _asr_summary(result: ASRResult) -> dict[str, Any]:
    avg_logprobs = [s.avg_logprob for s in result.segments if s.avg_logprob is not None]
    no_speech_probs = [s.no_speech_prob for s in result.segments if s.no_speech_prob is not None]
    compression_ratios = [s.compression_ratio for s in result.segments if s.compression_ratio is not None]

    def mean(xs: list[float]) -> float | None:
        return (sum(xs) / len(xs)) if xs else None

    return {
        "language": result.language,
        "language_probability": result.language_probability,
        "avg_logprob_mean": mean([float(x) for x in avg_logprobs]),
        "no_speech_prob_mean": mean([float(x) for x in no_speech_probs]),
        "compression_ratio_mean": mean([float(x) for x in compression_ratios]),
    }

def _infer_lang_from_relpath(relpath: str) -> str | None:
    parts = [p.strip().lower() for p in (relpath or "").split("/") if p.strip()]
    for p in parts[:-1]:
        if p in LANG_CODE_TO_NAME:
            return p
    return None

def run_pipeline(
    *,
    segments_dir: Path,
    out_dir: Path,
    lid_model_id: str = "facebook/mms-lid-256",
    whisper_model: str = "large-v3",
    extra_transformers_whisper_models: list[str] | None = None,
    transformers_whisper_model_map: dict[str, str] | None = None,
    transformers_whisper_default_model: str | None = None,
    gemini_model: str | None = None,
    gemini_prompt_path: Path | None = None,
    gemini_max_output_tokens: int = 512,
    gemini_hard_only: bool = True,
    ctc_model: str | None = None,
    ctc_model_map: dict[str, str] | None = None,
    require_lid_backend: bool = True,
    trim_silence: bool = True,
    silence_threshold_db: float = -40.0,
    min_silence_sec: float = 0.15,
    limit: int | None = None,
    thresholds: dict[str, Any] | None = None,
) -> Path:
    """
    End-to-end run: preprocess -> LID -> ASR ensemble -> normalize -> ROVER -> verify.
    Writes `out_dir/results.jsonl` and returns that path.
    """
    thresholds = thresholds or {}
    if (ctc_model or ctc_model_map) and "require_ctc" not in thresholds:
        thresholds["require_ctc"] = True
    if (ctc_model or ctc_model_map) and "min_ctc_margin" not in thresholds:
        thresholds["min_ctc_margin"] = 0.05
    out_dir.mkdir(parents=True, exist_ok=True)

    audio_files = _iter_audio_files(segments_dir)
    if not audio_files:
        raise RuntimeError(f"No audio files found under: {segments_dir}")
    if limit is not None:
        audio_files = audio_files[: max(0, int(limit))]

    # Instantiate models once.
    lid_backend: MMSLID | None
    try:
        lid_backend = MMSLID(model_id=lid_model_id)
    except Exception as e:
        if require_lid_backend:
            raise RuntimeError(
                "Failed to initialize MMS-LID. Install torch/transformers deps (pip install -e '.[torch]') "
                "or run with require_lid_backend=False."
            ) from e
        lid_backend = None

    whisper_backend = FasterWhisperASR(model_size_or_path=whisper_model)

    extra_transformers_whisper_models = extra_transformers_whisper_models or []
    transformers_backends: list[TransformersWhisperASR] = []
    for model_id in extra_transformers_whisper_models:
        transformers_backends.append(TransformersWhisperASR(model_id=model_id))

    transformers_router = None
    if transformers_whisper_model_map:
        transformers_router = TransformersWhisperASRRouter(
            model_map=transformers_whisper_model_map, default_model=transformers_whisper_default_model
        )

    ctc_scorer = None
    rank_by_ctc = None
    if ctc_model or ctc_model_map:
        from maya_transcribe.ctc_fit import CTCFitScorer, rank_candidates_by_ctc

        ctc_scorer = CTCFitScorer(model_map=ctc_model_map, default_model=ctc_model)
        rank_by_ctc = rank_candidates_by_ctc

    gemini_backend = None
    if gemini_model:
        try:
            from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template

            prompt = load_prompt_template(gemini_prompt_path or Path("Prompt.md"))
            gemini_backend = GeminiASR(
                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
                prompt_template=prompt,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e

    results_path = out_dir / "results.jsonl"
    summary_path = out_dir / "summary.json"

    counts = {"accept": 0, "human": 0, "reject": 0, "error": 0}
    timings_total: list[float] = []

    with results_path.open("w", encoding="utf-8") as f:
        for src in track(audio_files, description="Transcribing"):
            ident = segment_identity(src, root=segments_dir)
            segment_id = ident.segment_id
            t0 = time.perf_counter()

            try:
                # 1) Preprocess
                t_pre0 = time.perf_counter()
                src_info = ffprobe(src)
                wav_path = out_dir / "audio" / f"{segment_id}.wav"
                normalize_audio(
                    src,
                    wav_path,
                    trim_silence=trim_silence,
                    silence_threshold_db=silence_threshold_db,
                    min_silence_sec=min_silence_sec,
                )
                info = ffprobe(wav_path)
                qc = compute_qc(wav_path)
                t_pre1 = time.perf_counter()

                # 2) LID
                t_lid0 = time.perf_counter()
                lid: LIDResult | None = None
                if lid_backend is not None:
                    lid = lid_backend.predict(wav_path, top_k=5)
                t_lid1 = time.perf_counter()

                # Route language hint (only if confident).
                route_lang = None
                route_lang_source = None
                if lid and lid.predictions:
                    top1 = lid.predictions[0]
                    top2p = lid.predictions[1].prob if len(lid.predictions) > 1 else 0.0
                    if float(top1.prob) >= float(thresholds.get("lid_min_prob", 0.90)) and (
                        float(top1.prob) - float(top2p)
                    ) >= float(thresholds.get("lid_min_margin", 0.20)):
                        route_lang = top1.code
                        route_lang_source = "lid"

                if route_lang is None and bool(thresholds.get("route_lang_from_path", False)):
                    inferred = _infer_lang_from_relpath(ident.relpath)
                    if inferred:
                        route_lang = inferred
                        route_lang_source = "path"

                # 3) ASR ensemble
                t_asr0 = time.perf_counter()
                asr_results: dict[str, ASRResult] = {}
                asr_errors: dict[str, str] = {}
                primary_asr_key = None
                det = None

                # Whisper: deterministic + a small sampling diversity pass.
                try:
                    det = whisper_backend.transcribe(
                        wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
                    )
                    primary_asr_key = f"{det.model}|det"
                    asr_results[primary_asr_key] = det
                except Exception as e:
                    asr_errors[f"{whisper_backend.name}|det"] = f"{type(e).__name__}:{e}"

                try:
                    samp = whisper_backend.transcribe(wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2)
                    asr_results[f"{samp.model}|samp_t0.2"] = samp
                except Exception as e:
                    asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"

                for tf_backend in transformers_backends:
                    try:
                        res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
                        asr_results[res.model] = res
                    except Exception as e:
                        asr_errors[tf_backend.name] = f"{type(e).__name__}:{e}"


sed -n '220,460p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:

                if transformers_router is not None:
                    try:
                        routed = transformers_router.transcribe(
                            wav_path,
                            language=route_lang,
                            num_beams=5,
                            max_new_tokens=256,
                        )
                        if routed is not None:
                            asr_results[routed.model] = routed
                    except Exception as e:
                        asr_errors[f"transformers_router:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if primary_asr_key is None and asr_results:
                    primary_asr_key = sorted(asr_results.keys())[0]
                if not asr_results:
                    raise RuntimeError(f"no_asr_hypotheses:{asr_errors or 'unknown'}")

                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
                if gemini_backend is not None:
                    run_gemini = True
                    if gemini_hard_only:
                        base_for_heur = det or (next(iter(asr_results.values())) if asr_results else None)
                        det_sum = _asr_summary(base_for_heur) if base_for_heur is not None else {}
                        avg_lp = det_sum.get("avg_logprob_mean")
                        comp = det_sum.get("compression_ratio_mean")
                        no_speech = det_sum.get("no_speech_prob_mean")
                        run_gemini = bool(
                            route_lang is None
                            or (no_speech is not None and float(no_speech) > 0.5)
                            or (avg_lp is not None and float(avg_lp) < float(thresholds.get("min_avg_logprob_mean", -1.0)))
                            or (
                                comp is not None
                                and float(comp) > float(thresholds.get("max_compression_ratio_mean", 2.4))
                            )
                        )

                    if run_gemini:
                        try:
                            g = gemini_backend.transcribe(wav_path, language=route_lang)
                            asr_results[g.model] = g
                        except Exception:
                            pass

                t_asr1 = time.perf_counter()

                ensemble_models = sorted({v.model for v in asr_results.values()})
                ensemble_backends = sorted({v.model.split(":", 1)[0] for v in asr_results.values()})
                ensemble_warnings: list[str] = []
                if len(ensemble_models) < 2:
                    ensemble_warnings.append("low_model_diversity")

                # 4) Normalize hypotheses
                t_norm0 = time.perf_counter()
                norm_hyps: dict[str, str] = {}
                for k, res in asr_results.items():
                    lang_hint = res.language or route_lang
                    norm_hyps[k] = normalize_text(res.text, lang=lang_hint)
                t_norm1 = time.perf_counter()

                # 5) ROVER
                t_rover0 = time.perf_counter()
                rover = rover_combine(norm_hyps)
                t_rover1 = time.perf_counter()

                # 5.5) Audio-grounded fit scoring (optional): choose best candidate by CTC loss.
                t_ctc0 = time.perf_counter()
                ctc_info = None
                final_text = rover.text
                final_source = "rover"
                if ctc_scorer is not None and rank_by_ctc is not None:
                    candidates = {"rover": rover.text, **norm_hyps}
                    ctc_info = rank_by_ctc(ctc_scorer, wav_path, candidates=candidates, lang=route_lang)
                    best_key = ctc_info.get("best_key") if isinstance(ctc_info, dict) else None
                    if best_key and best_key in candidates:
                        final_text = candidates[best_key]
                        final_source = "rover" if best_key == "rover" else f"asr:{best_key}"
                t_ctc1 = time.perf_counter()

                # 6) Verify
                t_ver0 = time.perf_counter()
                asr_summaries = {k: _asr_summary(v) for k, v in asr_results.items()}
                verification = verify_segment(
                    qc=asdict(qc),
                    lid=lid,
                    final_text=final_text,
                    rover={
                        "text": rover.text,
                        "disagreement_rate": rover.disagreement_rate,
                        "skeleton_key": rover.skeleton_key,
                    },
                    asr_summaries=asr_summaries,
                    primary_asr_key=primary_asr_key,
                    ctc=ctc_info,
                    thresholds=thresholds,
                )
                t_ver1 = time.perf_counter()

                record = {
                    "schema_version": 1,
                    "segment_id": segment_id,
                    "base_id": ident.base_id,
                    "relpath": ident.relpath,
                    "uid": ident.uid,
                    "src_path": str(src),
                    "wav_path": str(wav_path),
                    "src_ffprobe": asdict(src_info),
                    "ffprobe": asdict(info),
                    "qc": asdict(qc),
                    "lid": (
                        None
                        if lid is None
                        else {
                            "model": lid.model,
                            "predictions": [{"code": p.code, "prob": p.prob} for p in lid.predictions],
                        }
                    ),
                    "route": {"language_hint": route_lang, "language_hint_source": route_lang_source},
                    "asr": {
                        k: {
                            "model": v.model,
                            "text": v.text,
                            "language": v.language,
                            "language_probability": v.language_probability,
                            "segments": [asdict(s) for s in v.segments],
                            "meta": v.meta,
                        }
                        for k, v in asr_results.items()
                    },
                    "asr_summary": asr_summaries,
                    "asr_errors": asr_errors,
                    "ensemble": {
                        "unique_models": ensemble_models,
                        "unique_backends": ensemble_backends,
                        "warnings": ensemble_warnings,
                    },
                    "normalized": norm_hyps,
                    "rover": {
                        "text": rover.text,
                        "skeleton_key": rover.skeleton_key,
                        "disagreement_rate": rover.disagreement_rate,
                        "disagreements": [
                            {"kind": d.kind, "index": d.index, "chosen": d.chosen, "options": d.options}
                            for d in rover.disagreements
                        ],
                    },
                    "ctc": ctc_info,
                    "final": {"text": final_text, "source": final_source},
                    "verification": {
                        "decision": verification.decision,
                        "reasons": verification.reasons,
                        "scores": verification.scores,
                    },
                    "config": {
                        "trim_silence": trim_silence,
                        "silence_threshold_db": silence_threshold_db,
                        "min_silence_sec": min_silence_sec,
                        "gemini_model": gemini_model,
                        "gemini_hard_only": gemini_hard_only,
                        "ctc_model": ctc_model,
                    },
                    "timing": {
                        "preprocess_sec": t_pre1 - t_pre0,
                        "lid_sec": t_lid1 - t_lid0,
                        "asr_sec": t_asr1 - t_asr0,
                        "normalize_sec": t_norm1 - t_norm0,
                        "rover_sec": t_rover1 - t_rover0,
                        "ctc_sec": t_ctc1 - t_ctc0,
                        "verify_sec": t_ver1 - t_ver0,
                        "total_sec": time.perf_counter() - t0,
                    },
                }

                f.write(json.dumps(record, ensure_ascii=False) + "\n")
                decision = verification.decision
                counts[decision] = counts.get(decision, 0) + 1
                timings_total.append(record["timing"]["total_sec"])

            except Exception as e:  # pragma: no cover
                counts["error"] += 1
                f.write(
                    json.dumps(
                        {"segment_id": segment_id, "src_path": str(src), "error": str(e)}, ensure_ascii=False
                    )
                    + "\n"
                )

    summary = {
        "segments_total": len(audio_files),
        "counts": counts,
        "avg_total_sec": (sum(timings_total) / len(timings_total)) if timings_total else None,
        "p50_total_sec": _percentile(timings_total, 0.50),
        "p90_total_sec": _percentile(timings_total, 0.90),
    }
    summary_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
    return results_path

def _percentile(xs: list[float], q: float) -> float | None:
    if not xs:
        return None
    xs_sorted = sorted(xs)
    idx = int(round((len(xs_sorted) - 1) * q))
    idx = max(0, min(len(xs_sorted) - 1, idx))
    return float(xs_sorted[idx])


**Managing memory in pipelines**

I'm thinking about how the pipeline instantiates the `whisper_backend` which runs deterministic and sampling passes. It seems that while using both FasterWhisper and transformers, out-of-memory (OOM) errors can occur due to GPU memory saturation, especially when keeping all transformer models loaded. To prevent these OOM issues, I should consider managing memory more effectively. Modifying the router to limit cache size and unloading models when switching could help, along with experimenting with different device settings for efficiency.


**Addressing GPU and language issues**

I need to utilize the GPU while handling concurrency with faster-whisper and ctranslate2. If I keep only one transformer model loaded at a time, memory use should be manageable, but adding faster-whisper complicates things. There's also an error related to the Odia language code. It looks like when using `processor.get_decoder_prompt_ids`, the processor tries to convert the code incorrectly. I think "or" should be supported by Whisper for Odia, but I need to confirm how the language mappings work.


**Handling language support errors**

It seems the processor's mapping might be missing the language code 'or' for Odia, possibly because of older transformer versions. The error indicates 'or' is unsupported, which could come from the `WhisperTokenizer.get_decoder_prompt_ids` that uses a language-to-ID mapping. I need to ensure that if 'or' isn't supported, it shouldn't crash the system. Instead, we can ignore forced decoder IDs and fall back on defaults. I will catch the ValueError from `get_decoder_prompt_ids` to prevent these errors in the future.


**Addressing transformer issues**

It looks like the transformer forced decoder IDs might still cause crashes if we don't handle them properly. In `TransformersWhisperASR.transcribe`, we should catch exceptions from `get_decoder_prompt_ids`. Adding a try/except block will allow us to manage ValueError or KeyError situations indicating unsupported languages, setting `forced_decoder_ids` to None to avoid errors. The OOM errors are still a priority, and it seems there's also an issue with loading model files for some language models. I need to verify the model structure and accessibility to resolve this.


**Verifying model compatibility and memory management**

I need to verify model IDs for each language, especially since the user mentioned `CXDuncan/whisper-large-v3-malayalam`, which might not have the necessary weights. If that's the case, I could adapt the mapping to a different model like `vasista22/whisper-malayalam-large-v2`. We should confirm that each model is compatible and not gated. There are memory management concerns, particularly for OOM errors. If we implement an LRU cache with a maximum model count, we can keep memory usage optimal. We'll also need to handle the missing weights and make adjustments if necessary.


**Assessing model compatibility and dataset building**

I need to check if we can list files from the `huggingface_hub` to identify whether `CXDuncan/whisper-large-v3-malayalam` is a proper model. It might not be a valid HF model but could be a ctranslate2 conversion. If that's the case, our Transformers backend might not load it, but I could use faster-whisper instead. I should also search for alternatives like `vasista22/whisper-malayalam-large-v2`, which is known to have weights. Plus, we need to integrate datasets from HF and create a bench dataset with samples for our project. I'll inspect the current builder implementation.


sed -n '1,260p' maya_transcribe/bench.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
import os
import random
import re
import wave
from dataclasses import dataclass
from pathlib import Path
from typing import Any

import numpy as np

from maya_transcribe.ids import segment_identity
from maya_transcribe.io import read_jsonl, write_jsonl

@dataclass(frozen=True)
class HFSource:
    dataset: str
    config: str | None
    split: str
    audio_key: str
    text_key: str
    id_key: str | None = None
    trust_remote_code: bool = False
    name: str | None = None

_SAFE_NAME_RE = re.compile(r"[^a-zA-Z0-9._-]+")

def _safe_name(value: str) -> str:
    value = (value or "").strip()
    value = value.replace("/", "_")
    value = value.replace("\\", "_")
    value = _SAFE_NAME_RE.sub("_", value)
    value = value.strip("._-")
    return value or "x"

def _ensure_hf_token_env() -> None:
    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
    if token and not os.getenv("HUGGINGFACE_HUB_TOKEN"):
        os.environ["HUGGINGFACE_HUB_TOKEN"] = token

def _write_wav_mono16(path: Path, audio: np.ndarray, sampling_rate: int) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)

    wav = np.asarray(audio, dtype=np.float32)
    if wav.ndim == 2:
        wav = wav.mean(axis=1)
    wav = np.clip(wav, -1.0, 1.0)
    pcm16 = (wav * 32767.0).astype(np.int16)

    with wave.open(str(path), "wb") as wf:
        wf.setnchannels(1)
        wf.setsampwidth(2)
        wf.setframerate(int(sampling_rate))
        wf.writeframes(pcm16.tobytes())

def _extract_audio(example: dict[str, Any], audio_key: str) -> tuple[np.ndarray, int] | None:
    audio = example.get(audio_key)
    if not isinstance(audio, dict):
        return None
    arr = audio.get("array")
    sr = audio.get("sampling_rate")
    if arr is None or sr is None:
        return None
    try:
        sr_i = int(sr)
    except (TypeError, ValueError):
        return None
    return np.asarray(arr, dtype=np.float32), sr_i

def _extract_text(example: dict[str, Any], text_key: str) -> str | None:
    text = example.get(text_key)
    if not isinstance(text, str):
        return None
    text = text.strip()
    return text or None

def default_sources() -> dict[str, list[HFSource]]:
    """
    Default benchmark sources for the 12 target languages:
    Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English.
    """
    kathbath = "ai4bharat/Kathbath"
    fleurs = "google/fleurs"

    # Canonical ISO-639-3 codes aligned to MMS-LID output (and language.py mappings).
    return {
        "hin": [HFSource(dataset=kathbath, config="hindi", split="valid", audio_key="audio_filepath", text_key="text")],
        "tam": [HFSource(dataset=kathbath, config="tamil", split="valid", audio_key="audio_filepath", text_key="text")],
        "tel": [HFSource(dataset=kathbath, config="telugu", split="valid", audio_key="audio_filepath", text_key="text")],
        "kan": [HFSource(dataset=kathbath, config="kannada", split="valid", audio_key="audio_filepath", text_key="text")],
        "mal": [
            HFSource(dataset=kathbath, config="malayalam", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "ben": [HFSource(dataset=kathbath, config="bengali", split="valid", audio_key="audio_filepath", text_key="text")],
        "guj": [
            HFSource(dataset=kathbath, config="gujarati", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "pan": [
            HFSource(dataset=kathbath, config="punjabi", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "mar": [
            HFSource(dataset=kathbath, config="marathi", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "ory": [HFSource(dataset=kathbath, config="odia", split="valid", audio_key="audio_filepath", text_key="text")],
        # Kathbath doesn't include Assamese/English; use FLEURS (requires trust_remote_code=True).
        "asm": [
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="test",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="validation",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="train",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
        ],
        "eng": [
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="test",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="validation",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="train",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
        ],
    }

def build_benchmark_corpus(
    *,
    out_dir: Path,
    samples_per_lang: int = 200,
    seed: int = 0,
    shuffle: bool = True,
    langs: list[str] | None = None,
    min_duration_sec: float = 1.0,
    max_duration_sec: float = 30.0,
    force: bool = False,
) -> dict[str, Any]:
    """
    Materialize a local benchmark corpus:
      - `out_dir/audio/<lang>/*.wav`
      - `out_dir/refs.jsonl` with `segment_id`, `text`, `lang`, `audio_relpath`, `source_*`
      - `out_dir/manifest.json` with build config + counts
    """
    _ensure_hf_token_env()

    if out_dir.exists():
        existing = [p for p in out_dir.iterdir()]
        if existing and not force:
            raise RuntimeError(f"Refusing to overwrite non-empty directory: {out_dir} (pass force=True)")
        if force:
            for p in sorted(out_dir.rglob("*"), reverse=True):
                if p.is_file() or p.is_symlink():
                    p.unlink()
                elif p.is_dir():
                    p.rmdir()

    out_dir.mkdir(parents=True, exist_ok=True)
    audio_root = out_dir / "audio"

    sources_by_lang = default_sources()
    target_langs = langs or sorted(sources_by_lang.keys())
    target_langs = [l.strip().lower() for l in target_langs if l.strip()]

    try:
        from datasets import load_dataset
    except Exception as e:  # pragma: no cover
        raise RuntimeError("Benchmark corpus build requires `datasets`. Install with: pip install -e '.[bench]'") from e

    rows: list[dict[str, Any]] = []
    counts: dict[str, int] = {}
    skipped: dict[str, int] = {}

    for lang in target_langs:
        sources = sources_by_lang.get(lang)
        if not sources:
            continue
        remaining = int(samples_per_lang)
        written = 0
        skipped_lang = 0

        for src in sources:
            if remaining <= 0:
                break

            ds = load_dataset(
                src.dataset,
                src.config,
                split=src.split,
                trust_remote_code=bool(src.trust_remote_code),
            )
            idxs = list(range(len(ds)))
            if shuffle:
                # Stable per-source shuffling (seeded by global seed + source identity).
                s = f"{seed}|{lang}|{src.dataset}|{src.config}|{src.split}"
                srng = random.Random(int.from_bytes(s.encode("utf-8"), "little") % (2**32))
                srng.shuffle(idxs)

            for i in idxs:
                if remaining <= 0:
                    break

                ex = ds[int(i)]
                if not isinstance(ex, dict):
                    continue

                text = _extract_text(ex, src.text_key)
                if not text:
                    skipped_lang += 1
                    continue

                audio_ex = _extract_audio(ex, src.audio_key)
                if audio_ex is None:
                    skipped_lang += 1
                    continue
                audio_arr, sr = audio_ex
                dur = float(audio_arr.shape[0]) / float(sr) if sr else 0.0


sed -n '260,520p' maya_transcribe/bench.py