# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <shell>bash</shell>
</environment_context>

plan was [master.md](master.md)

Full Benchmark Analysis Summary

  Run Comparison
  ┌─────────────┬────────────────────┬──────────────────┬───────────────────┐
  │   Metric    │ Baseline (FW only) │ + Vaani (strict) │ + Vaani (relaxed) │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ WER (final) │ 30.6%              │ 23.7%            │ 23.7%             │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Accept rate │ 0/200              │ 11/200           │ 81/200            │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Human rate  │ 200/200            │ 189/200          │ 119/200           │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Oracle best │ 29.7%              │ 9.8%             │ 9.8%              │
  └─────────────┴────────────────────┴──────────────────┴───────────────────┘
  Decision Quality with Relaxed Threshold
  ┌──────────┬───────┬──────────┬───────────────┐
  │ Decision │ Count │ WER Mean │ Disagree Rate │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Accept   │ 81    │ 18.6%    │ 8.5%          │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Human    │ 119   │ 27.1%    │ 13.2%         │
  └──────────┴───────┴──────────┴───────────────┘
  WER by Disagreement Rate Bucket

  - 0-5%: n=35, WER=6.0% (excellent)
  - 5-10%: n=47, WER=19.6%
  - 10-15%: n=68, WER=24.6%
  - 15-25%: n=44, WER=36.9%
  - 25-100%: n=6, WER=50.9%

  Remaining Human Decision Reasons

  - lid_uncertain: 63 (LID confused between Hindi/Urdu/related languages)
  - asr_disagreement: 50 (still above 15% threshold)
  - ctc_ambiguous: 49 (CTC margin too low)

  LID Analysis

  - 82% correctly identified as Hindi
  - 31.5% flagged uncertain due to low margin (Hindi vs Urdu/Gujarati/Bhojpuri)
  - This is expected: MMS-LID struggles with closely related Indic languages

  Final Selection Distribution

  - ROVER: 64.5% (129/200)
  - Vaani Hindi: 34.0% (68/200)
  - FW large-v3: 1.5% (3/200)

  Key Insights

  1. Oracle potential is real: 9.8% WER is achievable with perfect selection
  2. Current selection gap: 13.9pp left on the table due to suboptimal hypothesis selection
  3. Gating works directionally: Accept segments (18.6% WER) are better than Human segments (27.1% WER)
  4. Disagreement correlates with quality: Strong correlation between disagreement rate and WER
  5. LID is the new bottleneck: 31.5% flagged uncertain due to Hindi/Urdu confusion

  Recommendations

  1. Consider LID-Urdu equivalence for Hindi: Treat Hindi and Urdu as equivalent for routing since they share script (when using Devanagari transcription)
  2. Better CTC selection: Current CTC selection isn't capturing the oracle improvement - the selection_gap of 13.9pp suggests the CTC model isn't reliably picking the best hypothesis
  3. Two-stage approach: Accept high-confidence segments (low disagreement), use LLM-based selection for ambiguous cases

✻ Sautéed for 32m 59s

● Background command "Run benchmark with relaxed disagreement threshold" completed (exit code 0).

● I've already analyzed those results above. The benchmark with relaxed threshold (15% disagreement rate) completed successfully:

  Key result: Accept rate improved from 11 → 81 segments (40.5%) while maintaining the same 23.7% WER.

  The analysis shows the pipeline is working - accepted segments have lower WER (18.6%) than segments routed to human review (27.1%). The main remaining bottlenecks are:
  - LID uncertainty (Hindi/Urdu confusion) - 63 segments
  - Disagreement still above threshold - 50 segments
  - CTC ambiguity - 49 segments

  The oracle potential (9.8% WER) is significantly better than what we're achieving (23.7%), indicating room for improved hypothesis selection.

But these WER scores are not what i'm looking for. i need <5% WER scores for all languages consistently. For that to happen, i can run any LLMs like Gemma3n, voxtral.

Can we try these LLMs for audio transcription. can you test and give the performance scores or WER scores when compared to our recent ones above. download these models from huggingface and integrate them and do inference on transcription tasks and get full scores. both models wont fit at once, onload one, then another. you should basically not run transformers based infernece. use vLLM or any other inference engine for this to run the gemma3n models or voxtral models.

i've cleared the other task which is using GPU. you now have 80GB GPU. use more utilization for GPU.

# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Can you now check why this gemma 3n model is at such high WER rate ? take it up personally and compare the actual results the model is providing. if after everything the performance is from model's capability, then i agree we can't do much here. but if there is something wrong in our setup or prompt or whatever, we can fix it. dig deeper. Voxtral-small's performance is actually really good. can you note it down in the readme or somewhere for us to note the results. keep this in a file called results.md and explain what the experiemnt and it's results are, so we know which experiement gave best WER results. once you are done tweaking gemma model, let's note those results as well.

# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
can you summarize our findings on ASR with LLMs sofar.

# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
How about we do an ensemble of both ?

Run these next (ordered by highest expected WER drop for your exact setup).

## 1) Make sure Voxtral-Small is running in **true transcription mode**

You may already be close, but Voxtral has a dedicated “pure speech transcription mode” + explicit decoding guidance: **temperature=0.0 for transcription**. ([Hugging Face][1])
**Experiment**

* Use the OpenAI-compatible **audio.transcriptions** path (TranscriptionRequest) instead of chat prompting.
* Force `language="hi"` for your Hindi bench (removes one failure mode vs auto-detect). Voxtral supports Hindi explicitly. ([Hugging Face][1])
  **Why**: people often benchmark Voxtral via chat-completions + a prompt, which is measurably worse than its transcription API/mode.

## 2) Add a **2-pass “proofread” loop using the same audio**

This is the biggest “LLM-style” trick that often moves the needle without swapping models.

**Pass 1 (ASR):** normal transcription (temp 0).
**Pass 2 (repair):** feed *audio + pass1 transcript* and ask:

* “Correct transcription errors using the audio. Keep the same language/script. Do not add new content. Output only the corrected transcript.”

This specifically attacks your remaining Voxtral-Small 14.9% WER gap, which is usually dominated by (a) proper nouns, (b) word boundary errors, (c) dropped short words.

## 3) Stop optimizing Gemma-3n before you fix its known constraints

Gemma-3n’s audio encoder was (at launch) implemented for **up to ~30s clips**; longer segments tend to degrade or behave oddly unless you do long-form handling. ([Google Developers Blog][2])
If your Hindi bench has segments near 30–60s, cap/split to ≤20–25s and retry Gemma-3n *only after* that.

## 4) Add 1–2 more audio-LLM candidates worth testing for Indic

### Shuka v1 (Sarvam) — audio-native + Indic focus

Shuka is explicitly built for **audio in Indic languages** and reports testing on multiple Indic languages (including Hindi, Telugu, Tamil, etc.). ([Hugging Face][3])
**Run:** same Hindi bench with a strict “transcribe” prompt (it’s QA-oriented, but still worth a pass because it’s Indic-targeted).

### Qwen2-Audio / Qwen2.5-Omni

Worth testing, but be aware: vLLM users have hit prompt-format gotchas (e.g., requiring an explicit `<|AUDIO|>` tag depending on the serving path). ([GitHub][4])
**Run:** if you do it in vLLM, copy the exact audio prompt/template requirements.

## 5) If the real goal is **<5% WER**, add an Indic-ASR “ringer” (open-weight)

Even if your current focus is “LLM-based ASR,” the fastest way to hit the target is to include at least one strong Hindi-specialized ASR model and use it as:

* a fallback for hard segments, or
* a teacher / tie-breaker in an ensemble.

Two high-ROI options:

* **Whisper-Hindi v2 (Collabora)** claims ~**5% WER on google/fleurs** for Hindi. ([collabora.com][5])
* **AI4Bharat IndicConformer / IndicASR**: explicitly targets all 22 scheduled Indian languages (good for your “other language benches” plan). ([ai4bharat.iitm.ac.in][6])

## 6) Don’t waste cycles on models that won’t help Hindi

* **Nemotron Speech ASR** is optimized for low-latency streaming and (as described) is **English** transcription oriented; not a Hindi WER play. ([Hugging Face][7])
* **NVIDIA Canary** coverage is largely European languages (not Hindi). ([Hugging Face][8])

## 7) One eval tweak that can “free WER” without cheating

Before you chase model changes, confirm your WER pipeline matches Hindi realities:

* normalize punctuation consistently (or strip it)
* normalize digits (०/0), nukta variants, diacritics normalization
* consistent tokenization rules for compound words

This often shaves a couple points on Hindi WER when the model is already close.

---

### Minimal “next run set” (fast)

1. Voxtral-Small in **TranscriptionRequest** mode, `language=hi`, `temperature=0.0`. ([Hugging Face][1])
2. Voxtral-Small **2-pass repair** (audio + pass1 transcript).
3. Shuka v1 on same bench. ([Hugging Face][3])
4. Qwen2-Audio-7B-Instruct (template-correct). ([GitHub][4])

If you run only one thing: do (1)+(2). That’s the most likely to move 14.9% → materially lower without leaving the “audio LLM” lane.

Ignore the suggestions about the ASR based transformers models. they dont work. we've already tested them. see if the other suggestions help reduce further WER.

# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Can you explain how did we do do the experiments in this run. how did we choose the ensemble transcription. who chose the final output from the ensemble models. and what was the criteria. Also explain about the prompt we used to transcribe these models. explain this and let me come with a better strategy than the ones we tried sofar. you can actually add this information in the results.md so i have all info  at one place.

# Context from my IDE setup:

## Active file: Prompt.md

## Open tabs:
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
can you do one more run with all previous experiments with voxtral, gemma, ensemble and whatever experiments we did with this new prompt instruct at [Prompt.md](Prompt.md) and update it in results as new prompt experiments in a new section or add new point in same section as previous experiment, or maybe little organized like a tabulated but more verbosic way.

# Context from my IDE setup:

## Active file: .env

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Can you also evaluate

1. https://huggingface.co/MiniMaxAI/MiniMax-M2.1
2. https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507
3. https://huggingface.co/Qwen/Qwen3-235B-A22B-Thinking-2507

also, do a thorough test on all languages this time. please also evaluate Voxtral-small as well on all languages. put your results in results.md and also what happened to our ROVER method, can we use Gemini-3-pro as a judge of evaluating right sentence for the benchmark ? see if it helps before doing the above experiments, because i dont want to waste credits if we dont.  Gemini key is in .env and i've tested all these models via their API platform and all of them worked for transcriptions. i now need to evaluate performance.

# Context from my IDE setup:

## Active file: maya_transcribe/asr.py

## Open tabs:
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:

Fine, i thought these models can be run on audio transcriptions. such a dumbass models. it's a pity to not be able to use them.

Anyways, one last experiment i want you to do is induvidual first.

1. https://huggingface.co/ai4bharat/indic-conformer-600m-multilingual
2. https://huggingface.co/openai/whisper-large-v3
3. Voxtral
4. Gemma

Over 100 samples from each language. and store them in a Json first. we will do a thorough thinking and analyse how we can get the lowest transcription possible with induvidual or ensembled or a judge model involved -- all these we will analyse once we have the transcriptions. So, get those done from each of the above models. all languages in the bench with 100 samples each.

Some reference code for double checking indicConformer.

from transformers import AutoModel
import torch, torchaudio

# Load the model
model = AutoModel.from_pretrained("ai4bharat/indic-conformer-600m-multilingual", trust_remote_code=True)

# Load an audio file
wav, sr = torchaudio.load("audio.flac")
wav = torch.mean(wav, dim=0, keepdim=True)

target_sample_rate = 16000  # Expected sample rate
if sr != target_sample_rate:
    resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=target_sample_rate)
    wav = resampler(wav)

# Perform ASR with CTC decoding
transcription_ctc = model(wav, "hi", "ctc")
print("CTC Transcription:", transcription_ctc)

# Perform ASR with RNNT decoding
transcription_rnnt = model(wav, "hi", "rnnt")
print("RNNT Transcription:", transcription_rnnt)

# Context from my IDE setup:

## Active file: results.md

## Active selection of the file:
## Multilingual 100-sample Runs (Transcription Dumps)
Bench subset: `data/bench/maya_hf_v2_100` (12 languages x 100 samples). These runs are for JSON dumps first; eval.json was generated but not analyzed yet.

- IndicConformer 600M multilingual (HF): `outputs/bench/maya_hf_v2_100/indicconformer_600m_hf/results.jsonl`
  - Note: English segments failed with `KeyError:'joint_post_net_en'` (100 error rows).
- Whisper large-v3: `outputs/bench/maya_hf_v2_100/whisper_large_v3/results.jsonl`
- Voxtral-Small-24B-2507: `outputs/bench/maya_hf_v2_100/voxtral_small_vllm/results.jsonl`
  - Note: Voxtral only natively supports a small language set (hi/en/...) so non-supported languages likely degrade.
- Gemma-3n-E2B-it: `outputs/bench/maya_hf_v2_100/gemma3n_e2b_vllm/results.jsonl`

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Analyse and add metrics into results.md for this new section as well. create a table for per lang and model wise performance. and we will analyse the samples later. for now add this new section.

## Multilingual 100-sample Runs (Transcription Dumps)
Bench subset: `data/bench/maya_hf_v2_100` (12 languages x 100 samples). These runs are for JSON dumps first; eval.json was generated but not analyzed yet.

- IndicConformer 600M multilingual (HF): `outputs/bench/maya_hf_v2_100/indicconformer_600m_hf/results.jsonl`
  - Note: English segments failed with `KeyError:'joint_post_net_en'` (100 error rows).
- Whisper large-v3: `outputs/bench/maya_hf_v2_100/whisper_large_v3/results.jsonl`
- Voxtral-Small-24B-2507: `outputs/bench/maya_hf_v2_100/voxtral_small_vllm/results.jsonl`
  - Note: Voxtral only natively supports a small language set (hi/en/...) so non-supported languages likely degrade.
- Gemma-3n-E2B-it: `outputs/bench/maya_hf_v2_100/gemma3n_e2b_vllm/results.jsonl`

# Context from my IDE setup:

## Active file: results.md

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Can you now, cleanup the repo. keep the data, results, and all setup. just clean the code files where we didnt had to use them at all. Only required files are files that we used to generate transcriptions for the recent run which are needed for the voxtral, gemma, indicconformer, whisper-large v3 only. everything else is not needed. cleanup the files that we no longer need. just keep files and code that we needed to reproduce the recent run.

# Context from my IDE setup:

## Active file: results.md

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
keep the gemini_asr.py, r2.py, dataset.py and also keep the egg-info , keep the optional flags we'll decide on what to do about them.

# Context from my IDE setup:

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
fine. forget about the cleanup. can you also do a run few meta's ASR model ?

1. https://huggingface.co/facebook/seamless-m4t-v2-large
2. https://github.com/facebookresearch/omnilingual-asr - try the bigger model to see full performance.

github repo says,
Checkpoints of improved accuracy (CER) for the CTC and LLM-ASR models compared to our existing LLM-ASR model (omniASR_{CTC,LLM}_{300M,1B,3B,7B}_v2).

from omnilingual_asr.models.wav2vec2_llama.lang_ids import supported_langs

# Print all supported languages
print(f"Total supported languages: {len(supported_langs)}")
print(supported_langs)

# Check if a specific language is supported
if "eng_Latn" in supported_langs:
    print("English (Latin script) is supported!")

get the repo locally to a new folder locally, and check the readme, extract required code to run these evaluations. also only try the largest model for max intelligence.

Once these models are evaluated, add these results into results.md into the last table, if possible arrange results also to make them seem together with older model's runs.

# Context from my IDE setup:

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
fine create a new venv and run with it if you feel like it breaks the global env. else proceed with evals.
Create an isolated venv for omnilingual-asr + Seamless M4T, run those there, and keep the base env intact for vLLM.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Checking existing venv and disk space**

I'm listing the repo directory to confirm if the specific virtual environment exists and checking disk space to address the prior OSError related to no available space.


**Running directory listing and disk space check in parallel**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
.
..
Agents.md
.claude
configs
data
docs
.env
external
.gitignore
master.md
maya_transcribe
maya_transcribe.egg-info
outputs
Prompt.md
pyproject.toml
README.md
results.md
task.md
.venv_omni
.vscode


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       984G  932G   11G  99% /


Summary: auto


**Assessing disk constraints before proceeding**

Storage is limited (11 GB free), blocking large model downloads and experiments; I'll check cache sizes and ask the user to approve selective cleanup or alternatives before continuing.


**Planning safe cache size checks**

I'll run disk usage commands on key cache directories to gather sizes without changing anything, preparing to present cleanup options to the user for approval.


du -sh /home/ubuntu/.cache/huggingface /home/ubuntu/.cache/vllm /home/ubuntu/.cache/pip /home/ubuntu/.cache/torch /home/ubuntu/.cache/fairseq2 /home/ubuntu/.cache/whisper 2>/dev/null || true


Output:
Exit code: 0
Wall time: 1.1 seconds
Output:
519G	/home/ubuntu/.cache/huggingface
605M	/home/ubuntu/.cache/vllm
13G	/home/ubuntu/.cache/pip
115M	/home/ubuntu/.cache/torch
8.0K	/home/ubuntu/.cache/fairseq2


**Recommending Huggingface cache cleanup**

The huggingface cache is using 519GB, so freeing space there offers the best immediate relief. I’ll propose pruning old cache entries with `huggingface-cli` commands or moving the cache to external storage if available, while noting the base environment is untouched. I'll ask the user for permission before deleting or relocating cache data.


**Delaying plan and requesting disk cleanup approval**

I’m holding off on creating a detailed plan since disk space is insufficient to proceed. The existing `.venv_omni` is ready, but 11GB free won’t support OmniASR 7B (30GB) and Seamless M4T large models (>10GB). I’ll ask permission to prune the 519GB huggingface cache—proposing to keep only key models—or to relocate the cache to another path with more space. I’ll also suggest deleting old outputs if needed and request guidance on which approach to take.


Blocked on disk space before I can run OmniASR + Seamless M4T in the isolated venv. The root volume has only ~11 GB free, while OmniASR 7B alone needs ~30 GB plus temp download space; Seamless M4T large will add more. The biggest consumer is `/home/ubuntu/.cache/huggingface` at ~519 GB (pip cache is ~13 GB; vllm cache ~605 MB). The venv `.venv_omni` is already present, and I’ll keep the base env untouched as requested.

To proceed, I need your go‑ahead on freeing space. Pick one:
1. Approve pruning the HF cache (I can keep only models we need: Gemma‑3n, Voxtral, Whisper‑large‑v3, IndicConformer).
2. Give me an alternate path with free space for `HF_HOME`/`TRANSFORMERS_CACHE` in the venv.
3. Tell me which large outputs/data can be removed.


# Context from my IDE setup:

## Open tabs:
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
check if there are any files outside this folder with massive storage and delete them. make sure to not clear the code files in it. but just clear the massive data for us to make space for these models. you have full access to delete them. go ahead. dont worry.


**Planning large cache cleanup outside repo**

I'm targeting large cache directories outside the repo like huggingface, pip, and torch caches under /home/ubuntu/.cache, with user approval, to free space while avoiding code files and active model data. Next, I'll identify and selectively remove these caches.


**Mapping top-level disk usage in home**

I'm planning to run disk usage summaries on top-level and hidden directories in /home/ubuntu to identify large caches or outputs outside the repo, then selectively remove those with user approval while avoiding code directories.


du -sh /home/ubuntu/.[!.]* /home/ubuntu/.??* 2>/dev/null || true


Output:
Exit code: 0
Wall time: 4.4 seconds
Output:
220M	/home/ubuntu/cherry
24K	/home/ubuntu/denoise_audio.py
70M	/home/ubuntu/denoised_audio
12K	/home/ubuntu/fetch_and_denoise.py
102M	/home/ubuntu/FullSubNet-plus
119M	/home/ubuntu/inworld
4.0K	/home/ubuntu/languages.txt
9.9G	/home/ubuntu/maya1-fastapi
1.4G	/home/ubuntu/maya3
13G	/home/ubuntu/maya3data
4.8G	/home/ubuntu/maya_data
12G	/home/ubuntu/maya-transcribe
15M	/home/ubuntu/nltk_data
313M	/home/ubuntu/panns_data
4.0K	/home/ubuntu/papers
1.6G	/home/ubuntu/pipecat
11G	/home/ubuntu/qwen_image
512K	/home/ubuntu/raw_audio
100M	/home/ubuntu/research
1.6M	/home/ubuntu/runpod
4.0K	/home/ubuntu/sample.py
4.0K	/home/ubuntu/samples
20G	/home/ubuntu/spark
24M	/home/ubuntu/test_denoised
24M	/home/ubuntu/test_denoised_v2
108K	/home/ubuntu/test_s07
396K	/home/ubuntu/test_strength_07
4.0K	/home/ubuntu/test_strength.py
144K	/home/ubuntu/test_tts_16k.wav
252K	/home/ubuntu/test_tts.wav
119M	/home/ubuntu/tts
4.0K	/home/ubuntu/Untitled


Output:
Exit code: 0
Wall time: 2.7 seconds
Output:
8.0K	/home/ubuntu/.ansible
248K	/home/ubuntu/.azure
16K	/home/ubuntu/.bash_history
4.0K	/home/ubuntu/.bash_logout
4.0K	/home/ubuntu/.bashrc
543G	/home/ubuntu/.cache
50M	/home/ubuntu/.claude
12K	/home/ubuntu/.claude.json
12K	/home/ubuntu/.claude.json.backup
37M	/home/ubuntu/.codex
30M	/home/ubuntu/.config
302M	/home/ubuntu/.cursor
2.2G	/home/ubuntu/.cursor-server
24K	/home/ubuntu/.docker
4.0K	/home/ubuntu/.gitconfig
8.8M	/home/ubuntu/.keras
4.0K	/home/ubuntu/.lesshst
20K	/home/ubuntu/.lightning
17G	/home/ubuntu/.local
4.0K	/home/ubuntu/.modal.toml
12K	/home/ubuntu/.modelscope
880M	/home/ubuntu/.npm
912K	/home/ubuntu/.nv
4.0K	/home/ubuntu/.profile
44K	/home/ubuntu/.ssh
0	/home/ubuntu/.sudo_as_admin_successful
436K	/home/ubuntu/.triton
4.0K	/home/ubuntu/.wget-hsts


**Planning safe space cleanup**

The user allows deleting large data directories and caches outside the repo to free space, explicitly permitting removal beyond code files. I'll clear caches and big data dirs cautiously, avoiding known code but prioritizing substantial space gains.


**Finalizing cache deletion plan**

I'll delete the massive cache directories like huggingface, pip, vllm, torch, and npm caches outside the repo to maximize space freed while avoiding possible code repos. This targets safe large data without risking code removal.


du -sh /home/ubuntu/.cache/* 2>/dev/null | sort -h


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
0	/home/ubuntu/.cache/motd.legal-displayed
8.0K	/home/ubuntu/.cache/claude
8.0K	/home/ubuntu/.cache/fairseq2
8.0K	/home/ubuntu/.cache/gdown
8.0K	/home/ubuntu/.cache/git
12K	/home/ubuntu/.cache/flashinfer
12K	/home/ubuntu/.cache/Microsoft
16K	/home/ubuntu/.cache/com.vercel.cli
16K	/home/ubuntu/.cache/vastai
68K	/home/ubuntu/.cache/matplotlib
196K	/home/ubuntu/.cache/tvm-ffi
632K	/home/ubuntu/.cache/claude-cli-nodejs
8.0M	/home/ubuntu/.cache/typescript
8.4M	/home/ubuntu/.cache/DeepFilterNet
56M	/home/ubuntu/.cache/modelscope
115M	/home/ubuntu/.cache/torch
605M	/home/ubuntu/.cache/vllm
11G	/home/ubuntu/.cache/uv
13G	/home/ubuntu/.cache/pip
519G	/home/ubuntu/.cache/huggingface


**Planning large cache removals**

Preparing to delete major cache directories under the user's home, including huggingface, pip, uv, vllm, torch, and modelscope caches, totaling about 544GB, plus some smaller caches like npm and cursor-server, to free significant disk space safely.


rm -rf /home/ubuntu/.cache/huggingface /home/ubuntu/.cache/pip /home/ubuntu/.cache/uv /home/ubuntu/.cache/vllm /home/ubuntu/.cache/torch /home/ubuntu/.cache/modelscope /home/ubuntu/.npm /home/ubuntu/.cursor-server


Output: `/bin/bash -lc 'rm -rf /home/ubuntu/.cache/huggingface /home/ubuntu/.cache/pip /home/ubuntu/.cache/uv /home/ubuntu/.cache/vllm /home/ubuntu/.cache/torch /home/ubuntu/.cache/modelscope /home/ubuntu/.npm /home/ubuntu/.cursor-server'` rejected: blocked by policy


**Planning elevated permissions for deletion**

Deletion commands outside the repo are likely blocked without escalation, so I'll request `sandbox_permissions: "require_escalated"` with justification and proceed with smaller, stepwise removals to comply with policy.


User explicitly requested deleting large caches outside the repo to free disk space for model downloads.


Output: approval policy is Never; reject command — you should not ask for escalated permissions if the approval policy is Never


Summary: auto


**Confirming deletion restrictions**

The sandbox policy forbids deleting files outside the repo and disallows escalated permissions, so deletion must be limited to inside the repo if at all; I'll need to inform the user about these constraints and suggest manual deletion or cache relocation for outside paths.


**Preparing detailed user guidance**

I identified large cache directories outside the repo that can’t be deleted due to sandbox limits, so I’ll provide their sizes along with numbered cleanup suggestions and ask the user to run deletion commands or adjust cache locations before proceeding.


I’m blocked from deleting anything outside `/home/ubuntu/maya-transcribe` by the sandbox policy. I tried removing the big caches, but the command was rejected. I did scan for large dirs so you can delete them manually.

Largest outside the repo:
- `/home/ubuntu/.cache` ≈ 543G (Hugging Face cache alone ≈ 519G)
- `/home/ubuntu/.local` ≈ 17G
- `/home/ubuntu/spark` ≈ 20G
- `/home/ubuntu/maya3data` ≈ 13G
- `/home/ubuntu/qwen_image` ≈ 11G
- `/home/ubuntu/maya1-fastapi` ≈ 9.9G

If you want the fastest space back, delete the HF cache first:
```bash
rm -rf ~/.cache/huggingface ~/.cache/pip ~/.cache/uv ~/.cache/vllm ~/.cache/torch ~/.cache/modelscope ~/.npm ~/.cursor-server
```

Once you’ve cleared space, I can resume OmniASR + Seamless M4T in the isolated venv.

Next steps (pick one):
1. Run the cleanup commands above and tell me when disk is free.
2. Tell me which big directories I should delete (and you’ll do it on your side).
3. Provide a different cache path/volume and I’ll point `HF_HOME`/`TRANSFORMERS_CACHE` there.


# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json
- results.md: results.md
- asr.py: maya_transcribe/asr.py
- .env: .env
- Prompt.md: Prompt.md

## My request for Codex:
done


**Checking disk space before proceeding**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       984G  397G  546G  43% /


**Reviewing ASR repo and dataset setup**

I'm confirming the existing omnilingual ASR repo and virtual environment state, recalling the multilingual 100-sample dataset, and preparing to inspect repo scripts to implement inference and evaluation for OmniASR and Seamless M4T models.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 1040
drwxrwxr-x 8 ubuntu ubuntu   4096 Jan 15 21:27 .
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 15 21:27 ..
-rw-rw-r-- 1 ubuntu ubuntu     48 Jan 15 21:27 CHANGELOG.md
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 15 21:27 ci
-rw-rw-r-- 1 ubuntu ubuntu   3537 Jan 15 21:27 CODE_OF_CONDUCT.md
-rw-rw-r-- 1 ubuntu ubuntu   2636 Jan 15 21:27 CONTRIBUTING.md
drwxrwxr-x 8 ubuntu ubuntu   4096 Jan 15 21:28 .git
drwxrwxr-x 4 ubuntu ubuntu   4096 Jan 15 21:27 .github
-rw-rw-r-- 1 ubuntu ubuntu    673 Jan 15 21:27 .gitignore
-rw-rw-r-- 1 ubuntu ubuntu    581 Jan 15 21:27 LICENSE
-rw-rw-r-- 1 ubuntu ubuntu    137 Jan 15 21:27 mypy.ini
-rw-rw-r-- 1 ubuntu ubuntu 441477 Jan 15 21:27 omniASR_header.jpg
-rw-rw-r-- 1 ubuntu ubuntu  30360 Jan 15 21:27 per_language_results_table_7B_llm_asr.csv
-rw-rw-r-- 1 ubuntu ubuntu   1162 Jan 15 21:27 .pre-commit-config.yaml
-rw-rw-r-- 1 ubuntu ubuntu   2522 Jan 15 21:27 pyproject.toml
-rw-rw-r-- 1 ubuntu ubuntu  13928 Jan 15 21:27 README.md
-rw-rw-r-- 1 ubuntu ubuntu 507738 Jan 15 21:27 result_table.png
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 15 21:27 src
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 15 21:27 tests
drwxrwxr-x 4 ubuntu ubuntu   4096 Jan 15 21:27 workflows


**Checking README for instructions**


rg -n "ASRInferencePipeline|inference|usage" external/omnilingual-asr/README.md


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: rg: command not found


**Using grep instead of rg**


grep -n "ASRInferencePipeline\|inference\|usage" external/omnilingual-asr/README.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
30:- **[Inference Pipeline](src/omnilingual_asr/models/inference/README.md)** - Comprehensive transcription guide with batch processing, language conditioning, and context examples
47:The models were developed using [fairseq2](https://github.com/facebookresearch/fairseq2), a research-focused sequence modeling toolkit. While we provide a **reference** inference pipeline that works across platforms, audio support requires [libsndfile](https://github.com/facebookresearch/fairseq2?tab=readme-ov-file#system-dependencies) (Mac: `brew install libsndfile`; Windows may need an additional [setup](https://github.com/facebookresearch/fairseq2?tab=readme-ov-file#installing-on-windows)).
60:from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
62:pipeline = ASRInferencePipeline(model_card="omniASR_LLM_Unlimited_7B_v2")
68:More details on running specific models can be found in the [src/omnilingual_asr/models/inference](/src/omnilingual_asr/models/inference/README.md) directory.
70:> **⚠ Important:** Currently only audio files shorter than 40 seconds are accepted for inference on CTC and LLM model suites.
93:This dataset can be directly used with our inference pipeline for evaluation or testing:
101:from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
111:# Run inference
112:pipeline = ASRInferencePipeline(model_card="omniASR_LLM_7B_v2")
164:- **Automatic Download**: Models are automatically downloaded on first use during training or inference


**Opening inference README**


sed -n '1,220p' external/omnilingual-asr/src/omnilingual_asr/models/inference/README.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Wav2Vec2 Inference Pipeline

Quick start guide for transcribing audio with our multilingual ASR models.

---

## 1. Quick Start

```python
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline
pipeline = ASRInferencePipeline(model_card="omniASR_CTC_1B_v2")
transcriptions = pipeline.transcribe(["/path/to/audio1.flac"], batch_size=1)
print(transcriptions[0])
```

---

## 2. Basic Usage

### 2.1 Audio Input Formats

Our pipeline accepts multiple input formats (`AudioInput`):

1. **File paths**: `["/path/to/audio.wav", "/path/to/audio.flac"]`
2. **Encoded audio binary data in memory**: `[open("audio.wav", "rb").read()]` or `[numpy_audio_array]` (int8)
3. **Decoded audio dicts**: `[{"waveform": tensor, "sample_rate": 16000}]`

The audio data is optionally decoded (`.wav/.flac`), resampled to 16kHz, converted to a mono-channel, and normalized before being ingested by the model.

> [!TIP]
> The models were mostly trained on audio durations of 30 seconds or less. Therefore, we do not recommend transcribing sequences longer than 30 seconds in a single sample.
> **⚠ Important:** Currently only audio files shorter than 40 seconds are accepted for inference!

> [!TIP]
> We recommend keeping this preprocessing pipeline similar when integrating the model in downstream applications.

### 2.2 Batch Processing

```python
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

pipeline = ASRInferencePipeline(model_card="omniASR_CTC_1B_v2")

audio_files = ["/path/to/audio1.flac", "/path/to/audio2.wav"]
transcriptions = pipeline.transcribe(audio_files, batch_size=2)

for file, trans in zip(audio_files, transcriptions):
    print(f"{file}: {trans}")
```

---

## 3. Model Types

### 3.1 Parallel Generation with CTC Models

The `omniASR_CTC_{300M,1B,3B,7B}_v2` models are most useful for their parallel generation capabilities, resulting in faster throughput. Note that they don't support language conditioning or context examples.

```python
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

pipeline = ASRInferencePipeline(model_card="omniASR_CTC_3B_v2", device=None)

audio_files = ["/path/to/audio1.flac", "/path/to/audio2.wav"]
transcriptions = pipeline.transcribe(audio_files, batch_size=2)

for file_path, text in zip(audio_files, transcriptions):
    print(f"CTC transcription - {file_path}: {text}")
```

### 3.2 Autoregressive Generation with Language-conditioned LLM Models

The `omniASR_LLM_{300M,1B,3B,7B}_v2` and `omniASR_LLM_Unlimited_{300M,1B,3B,7B}_v2` models take an optional language code as argument to guide transcription towards the target language and script.
The language codes can be found in [lang_ids.py](/src/omnilingual_asr/models/wav2vec2_llama/lang_ids.py) and the language to language-id mapping is described in [our paper (Appendix A)](https://ai.meta.com/research/publications/omnilingual-asr-open-source-multilingual-speech-recognition-for-1600-languages/).

> [!TIP]
> We recommend providing the language code when possible as it helps with transcription quality.

```python
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

pipeline = Wav2Vec2InferencePipeline(model_card="omniASR_LLM_Unlimited_1B_v2")

audio_files = ["/path/to/russian_audio.wav", "/path/to/english_audio.flac", "/path/to/german_audio.wav"]

transcriptions = pipeline.transcribe(audio_files, lang=["rus_Cyrl", "eng_Latn", "deu_Latn"], batch_size=3)
```

### 3.3 Zero-Shot Generation

The `omniASR_LLM_7B_ZS` model is trained to accept in-context audio/transcription pairs to perform zero-shot inference on unseen languages via in-context learning. You can provide anywhere **from one to ten examples**, with more examples generally leading to better performance. Internally, the model uses exactly ten context slots: if fewer than ten examples are provided, samples are duplicated sequentially to fill all slots (and cropped to ten if more are provided).

> [!TIP]
> Similar to the other models, the zero-shot model has been trained with context samples of up to 30 seconds length and will likely perform suboptimally given longer samples (maximum audio length is 60 seconds).

```python
from omnilingual_asr.models.inference.pipeline import (
    ASRInferencePipeline,
    ContextExample
)

pipeline = ASRInferencePipeline(model_card="omniASR_LLM_7B_ZS")

context_examples = [
    ContextExample("/path/to/context_audio1.wav", "Hello world"),
    ContextExample("/path/to/context_audio2.wav", "How are you today"),
    ContextExample("/path/to/context_audio3.flac", "Nice to meet you")
]

transcriptions = pipeline.transcribe_with_context(
    ["/path/to/test_audio.wav"],
    context_examples=[context_examples],
    batch_size=1
)

print(f"Transcription: {transcriptions[0]}")
```

---

## 4. Developer Reference

### 4.1 Parquet Binary Audio Data Input
Our training parquet datasets follow a special format conversion (see [data preparation guide](/workflows/dataprep/README.md#21-schema) for schema details). We can use them in the inference pipeline as follows:

```python
import pyarrow.parquet as pq
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

ds = pq.ParquetDataset("/path/to/dataset/")
batch_data = ds._dataset.head(10).to_pandas()  # taking only few first samples
audio_bytes = batch_data["audio_bytes"].tolist()

pipeline = ASRInferencePipeline(model_card="omniASR_LLM_1B_v2")
transcriptions = pipeline.transcribe(audio_bytes, batch_size=4)

for i, text in enumerate(transcriptions):
    print(f"Sample {i+1}: {text}")
```

### 4.2 HuggingFace Datasets

```python
from datasets import load_dataset
from omnilingual_asr.models.inference.pipeline import ASRInferencePipeline

# Load dataset
omni_dataset = load_dataset("facebook/omnilingual-asr-corpus", "lij_Latn", split="train", streaming=True)
batch = next(omni_dataset.iter(5))

# Renaming columns to match the pipeline's expected input
audio_data = [{"waveform": x["array"], "sample_rate": x["sampling_rate"]}
              for x in batch["audio"]]

pipeline = ASRInferencePipeline(model_card="omniASR_LLM_1B_v2")
transcriptions = pipeline.transcribe(audio_data, batch_size=2)

for i, text in enumerate(transcriptions):
    print(f"Sample {i+1}: {text}")
```

### 4.3 Model Input Format Specification

This section contains the description of the `Wav2Vec2LlamaModel` input batch format. It is primarily for users who aim to integrate the model in their own applications while using the fairseq2 `Seq2SeqBatch` interface.

> [!WARNING]
> The behavior of `.forward()` can vary depending on the structure of the batch. In particular, the presence of extra fields in `batch.example` may cause the model to interpret the batch differently.

1. **Basic ASR**
```
  batch = Seq2SeqBatch(
      source_seqs=audio_tensor,           # [BS, T_audio, D_audio] - target audio
      source_seq_lens=audio_lengths,      # [BS] - actual audio lengths
      target_seqs=text_tensor,            # [BS, T_text] - target text tokens
      target_seq_lens=text_lengths,       # [BS] - actual text lengths
      example={}                          # Empty dict - no special fields needed
  )
```

2. **Language-Aware ASR**
```
  batch = Seq2SeqBatch(
      source_seqs=audio_tensor,           # [BS, T_audio, D_audio] - target audio
      source_seq_lens=audio_lengths,      # [BS] - actual audio lengths
      target_seqs=text_tensor,            # [BS, T_text] - target text tokens
      target_seq_lens=text_lengths,       # [BS] - actual text lengths
      example={
          "lang": ['mxs_Latn', ...]  # [BS] - language codes per sample
      }
  )
```
Our batch has some `lang_tokens` as part of the `batch.example` parameter, but they are only used for S2TT (kept for reference), not for LID conditioning, and can be ignored. The `lang` parameter should only consist of strings from [lang_ids.py](/src/omnilingual_asr/models/wav2vec2_llama/lang_ids.py).

3. **Zero-Shot Context**
```
  batch = Seq2SeqBatch(
      source_seqs=audio_tensor,           # [BS, T_audio, D_audio] - target audio
      source_seq_lens=audio_lengths,      # [BS] - actual audio lengths
      target_seqs=text_tensor,            # [BS, T_text] - target text tokens
      target_seq_lens=text_lengths,       # [BS] - actual text lengths
      example={
          "context_audio": [
              {"seqs": context_audio_1, "seq_lens": [audio_len_1]},  # Context audio 1 # List[Dict] - N context examples
              ...
              {"seqs": context_audio_BS, "seq_lens": [audio_len_BS]},  # BS Context audio
          ],
          "context_text": [               # List[Dict] - N context text examples
              {"seqs": context_text_1, "seq_lens": [text_len_1]},
              ...
              {"seqs": context_text_BS, "seq_lens": [text_len_BS]},   # BS Context text
          ]
      }
  )
```

### 4.4 Punctuation and Capitalization

Our models are trained to output transcripts in spoken form without any punctuation or capitalization. If you would like transcripts in written form, we recommend passing our model's outputs through a third-party library to add punctuation, such as [this one](https://github.com/oliverguhr/deepmultilingualpunctuation). Note, however, that most punctuation libraries only cover a small subset of the 1600+ languages supported by our model.


**Inspecting existing pipeline for run script**

I’m checking the `maya_transcribe` pipeline to find a pattern for running models and scripting output formatting to help create the new OmniASR run script producing results.jsonl.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 240
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 14 01:03 .
drwxrwxr-x 12 ubuntu ubuntu  4096 Jan 15 21:34 ..
-rw-rw-r--  1 ubuntu ubuntu  2181 Jan 13 04:24 align.py
-rw-rw-r--  1 ubuntu ubuntu 36086 Jan 15 18:47 asr.py
-rw-rw-r--  1 ubuntu ubuntu  4147 Jan 13 04:18 audio.py
-rw-rw-r--  1 ubuntu ubuntu 14114 Jan 13 09:41 bench.py
-rw-rw-r--  1 ubuntu ubuntu 26339 Jan 15 18:48 cli.py
-rw-rw-r--  1 ubuntu ubuntu  1258 Jan 13 04:16 config.py
-rw-rw-r--  1 ubuntu ubuntu 13909 Jan 13 07:09 ctc_fit.py
-rw-rw-r--  1 ubuntu ubuntu   387 Jan 13 04:17 dataset.py
-rw-rw-r--  1 ubuntu ubuntu  1508 Jan 13 04:37 db.py
-rw-rw-r--  1 ubuntu ubuntu 11240 Jan 14 00:40 eval.py
-rw-rw-r--  1 ubuntu ubuntu  9033 Jan 13 06:06 gemini_asr.py
-rw-rw-r--  1 ubuntu ubuntu   992 Jan 13 06:16 ids.py
-rw-rw-r--  1 ubuntu ubuntu    50 Jan 13 04:16 __init__.py
-rw-rw-r--  1 ubuntu ubuntu   667 Jan 13 04:19 io.py
-rw-rw-r--  1 ubuntu ubuntu  4782 Jan 13 23:24 language.py
-rw-rw-r--  1 ubuntu ubuntu  2272 Jan 13 04:35 lid.py
-rw-rw-r--  1 ubuntu ubuntu 29267 Jan 15 18:48 pipeline.py
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 15 18:49 __pycache__
-rw-rw-r--  1 ubuntu ubuntu  3887 Jan 13 04:32 r2.py
-rw-rw-r--  1 ubuntu ubuntu  6414 Jan 13 04:37 rover.py
-rw-rw-r--  1 ubuntu ubuntu  1120 Jan 13 04:17 tar.py
-rw-rw-r--  1 ubuntu ubuntu  1448 Jan 13 04:25 textnorm.py
-rw-rw-r--  1 ubuntu ubuntu  6613 Jan 13 06:08 verify.py


**Planning OmniASR integration**

I'm considering implementing a separate script to run the OmniASR pipeline within its own virtual environment for compatibility and to produce results.jsonl in the correct format by examining existing outputs for evaluation.


python - <<'PY'
import json
from pathlib import Path
path = Path('outputs/bench/maya_hf_v2_100/voxtral_small_vllm/results.jsonl')
with path.open() as f:
    for i in range(3):
        print(f.readline().strip())
PY


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


**Correcting Python command usage**


python3 - <<'PY'
from pathlib import Path
path = Path('outputs/bench/maya_hf_v2_100/voxtral_small_vllm/results.jsonl')
with path.open() as f:
    for i in range(3):
        print(f.readline().strip())
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{"schema_version": 1, "segment_id": "fleurs__as_in__test__105__1875-fc7f2738561720cf", "base_id": "fleurs__as_in__test__105__1875", "relpath": "asm/fleurs__as_in__test__105__1875.wav", "uid": "fc7f2738561720cf", "src_path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__105__1875.wav", "wav_path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__105__1875-fc7f2738561720cf.wav", "src_ffprobe": {"path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__105__1875.wav", "duration_sec": 9.0, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__105__1875-fc7f2738561720cf.wav", "duration_sec": 9.0, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 9.0, "rms_dbfs": -21.883280209057816, "peak_dbfs": -4.625800291220733, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "asm", "prob": 0.9999120235443115}, {"code": "ben", "prob": 2.1966910935589112e-05}, {"code": "ory", "prob": 5.017155217501568e-06}, {"code": "guj", "prob": 3.7184656775934855e-06}, {"code": "pcm", "prob": 2.9838331556675257e-06}]}, "route": {"language_hint": "asm", "language_hint_source": "path"}, "asr": {"vllm:mistralai/Voxtral-Small-24B-2507": {"model": "vllm:mistralai/Voxtral-Small-24B-2507", "text": "হাগরত পর্বত নদীহমুহর বিশালতংখ পানি এমাজন পর্বত নদীহমুহর।", "language": "as", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:mistralai/Voxtral-Small-24B-2507": {"language": "as", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:mistralai/Voxtral-Small-24B-2507"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:mistralai/Voxtral-Small-24B-2507": "হাগরত পর্বত নদীহমুহর বিশালতংখ পানি এমাজন পর্বত নদীহমুহর"}, "rover": {"text": "হাগরত পর্বত নদীহমুহর বিশালতংখ পানি এমাজন পর্বত নদীহমুহর", "skeleton_key": "vllm:mistralai/Voxtral-Small-24B-2507", "disagreement_rate": 0.0, "inputs": ["vllm:mistralai/Voxtral-Small-24B-2507"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "হাগরত পর্বত নদীহমুহর বিশালতংখ পানি এমাজন পর্বত নদীহমুহর", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 9.0, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "asm", "top1_prob": 0.9999120235443115, "margin": 0.9998900566333759, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:mistralai/Voxtral-Small-24B-2507", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "hf_indicconformer_model": null, "hf_indicconformer_decode": "rnnt", "vllm_asr_model": "mistralai/Voxtral-Small-24B-2507", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "vllm_transcribe_prompt": null, "vllm_proofread": false, "vllm_proofread_prompt": null, "skip_whisper": true}, "timing": {"preprocess_sec": 0.2101431591436267, "lid_sec": 0.5126403518952429, "asr_sec": 2.645038617774844, "normalize_sec": 7.093371823430061e-05, "rover_sec": 6.827898323535919e-05, "ctc_sec": 1.193024218082428e-06, "verify_sec": 7.80872069299221e-05, "total_sec": 3.3681430839933455}}
{"schema_version": 1, "segment_id": "fleurs__as_in__test__128__1900-f706b548dac4d420", "base_id": "fleurs__as_in__test__128__1900", "relpath": "asm/fleurs__as_in__test__128__1900.wav", "uid": "f706b548dac4d420", "src_path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__128__1900.wav", "wav_path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__128__1900-f706b548dac4d420.wav", "src_ffprobe": {"path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__128__1900.wav", "duration_sec": 15.0, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__128__1900-f706b548dac4d420.wav", "duration_sec": 15.0, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 15.0, "rms_dbfs": -22.348078206046246, "peak_dbfs": -5.832851947338778, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "asm", "prob": 0.9999334812164307}, {"code": "ben", "prob": 1.3221640074334573e-05}, {"code": "ory", "prob": 3.2946936698863283e-06}, {"code": "guj", "prob": 3.0147664347168757e-06}, {"code": "pcm", "prob": 2.4582166133768624e-06}]}, "route": {"language_hint": "asm", "language_hint_source": "path"}, "asr": {"vllm:mistralai/Voxtral-Small-24B-2507": {"model": "vllm:mistralai/Voxtral-Small-24B-2507", "text": "असल में, यह अस्तित्व में नहीं है, यह एक विशाल भ्रम है। गुफा के भीतर प्रवेश करने के बाद, हम पूर्ण रूप से पृथक हो जाते हैं।", "language": "as", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:mistralai/Voxtral-Small-24B-2507": {"language": "as", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:mistralai/Voxtral-Small-24B-2507"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:mistralai/Voxtral-Small-24B-2507": "असल में यह अस्तित्व में नहीं है यह एक विशाल भ्रम है गुफा के भीतर प्रवेश करने के बाद हम पूर्ण रूप से पृथक हो जाते हैं"}, "rover": {"text": "असल में यह अस्तित्व में नहीं है यह एक विशाल भ्रम है गुफा के भीतर प्रवेश करने के बाद हम पूर्ण रूप से पृथक हो जाते हैं", "skeleton_key": "vllm:mistralai/Voxtral-Small-24B-2507", "disagreement_rate": 0.0, "inputs": ["vllm:mistralai/Voxtral-Small-24B-2507"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "असल में यह अस्तित्व में नहीं है यह एक विशाल भ्रम है गुफा के भीतर प्रवेश करने के बाद हम पूर्ण रूप से पृथक हो जाते हैं", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 15.0, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "asm", "top1_prob": 0.9999334812164307, "margin": 0.9999202595763563, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:mistralai/Voxtral-Small-24B-2507", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "hf_indicconformer_model": null, "hf_indicconformer_decode": "rnnt", "vllm_asr_model": "mistralai/Voxtral-Small-24B-2507", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "vllm_transcribe_prompt": null, "vllm_proofread": false, "vllm_proofread_prompt": null, "skip_whisper": true}, "timing": {"preprocess_sec": 0.22859176201745868, "lid_sec": 0.18826681608334184, "asr_sec": 1.496032873634249, "normalize_sec": 7.916800677776337e-05, "rover_sec": 0.00010321428999304771, "ctc_sec": 1.3927929103374481e-06, "verify_sec": 6.323866546154022e-05, "total_sec": 1.9132462097331882}}
{"schema_version": 1, "segment_id": "fleurs__as_in__test__12__1728-f479ba43ea37790d", "base_id": "fleurs__as_in__test__12__1728", "relpath": "asm/fleurs__as_in__test__12__1728.wav", "uid": "f479ba43ea37790d", "src_path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__12__1728.wav", "wav_path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__12__1728-f479ba43ea37790d.wav", "src_ffprobe": {"path": "data/bench/maya_hf_v2_100/audio/asm/fleurs__as_in__test__12__1728.wav", "duration_sec": 8.1, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_v2_100/voxtral_small_vllm/audio/fleurs__as_in__test__12__1728-f479ba43ea37790d.wav", "duration_sec": 8.1, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 8.1, "rms_dbfs": -31.181398760098006, "peak_dbfs": -16.369645884313897, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "asm", "prob": 0.9999278783798218}, {"code": "ben", "prob": 6.0592115005420055e-06}, {"code": "ory", "prob": 4.989031822333345e-06}, {"code": "npi", "prob": 4.680767233367078e-06}, {"code": "guj", "prob": 3.6243295653548557e-06}]}, "route": {"language_hint": "asm", "language_hint_source": "path"}, "asr": {"vllm:mistralai/Voxtral-Small-24B-2507": {"model": "vllm:mistralai/Voxtral-Small-24B-2507", "text": "रखाइनेकोटोर अम्लता ना हो या खोरोखियो तर एलकेलाइनर ऊपर निर्भर करे बंधाको भी रंग परिवर्तन हुन्छ।", "language": "as", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:mistralai/Voxtral-Small-24B-2507": {"language": "as", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:mistralai/Voxtral-Small-24B-2507"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:mistralai/Voxtral-Small-24B-2507": "रखाइनेकोटोर अम्लता ना हो या खोरोखियो तर एलकेलाइनर ऊपर निर्भर करे बंधाको भी रंग परिवर्तन हुन्छ"}, "rover": {"text": "रखाइनेकोटोर अम्लता ना हो या खोरोखियो तर एलकेलाइनर ऊपर निर्भर करे बंधाको भी रंग परिवर्तन हुन्छ", "skeleton_key": "vllm:mistralai/Voxtral-Small-24B-2507", "disagreement_rate": 0.0, "inputs": ["vllm:mistralai/Voxtral-Small-24B-2507"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "रखाइनेकोटोर अम्लता ना हो या खोरोखियो तर एलकेलाइनर ऊपर निर्भर करे बंधाको भी रंग परिवर्तन हुन्छ", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 8.1, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "asm", "top1_prob": 0.9999278783798218, "margin": 0.9999218191683212, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:mistralai/Voxtral-Small-24B-2507", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "hf_indicconformer_model": null, "hf_indicconformer_decode": "rnnt", "vllm_asr_model": "mistralai/Voxtral-Small-24B-2507", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "vllm_transcribe_prompt": null, "vllm_proofread": false, "vllm_proofread_prompt": null, "skip_whisper": true}, "timing": {"preprocess_sec": 0.22724887914955616, "lid_sec": 0.1448286809027195, "asr_sec": 1.463312078267336, "normalize_sec": 7.266690954566002e-05, "rover_sec": 8.863676339387894e-05, "ctc_sec": 1.3532117009162903e-06, "verify_sec": 7.668603211641312e-05, "total_sec": 1.8357429159805179}}


Summary: auto


**Inspecting evaluation script expectations**

I'm checking the eval.py and evaluation pipeline to determine the required fields for results.jsonl, ensuring they include all keys needed to compute WER correctly.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from jiwer import cer, wer

from maya_transcribe.io import read_jsonl
from maya_transcribe.textnorm import normalize_text

@dataclass(frozen=True)
class EvalRow:
    segment_id: str
    language: str | None
    decision: str | None
    duration_sec: float | None
    duration_bucket: str | None
    ref: str
    hyp: str
    wer: float
    cer: float
    oracle_best_wer: float | None = None
    oracle_best_cer: float | None = None
    oracle_best_key: str | None = None
    selection_gap_wer: float | None = None

def load_refs(refs_path: Path) -> dict[str, dict[str, Any]]:
    """
    Read a JSONL file of references.
    Expected keys (any one): `text`, `transcript`, `reference`.
    Required key: `segment_id` (or `id`).
    """
    refs: dict[str, dict[str, Any]] = {}
    for row in read_jsonl(refs_path):
        segment_id = str(row.get("segment_id") or row.get("id") or "").strip()
        if not segment_id:
            continue
        refs[segment_id] = row
    return refs

def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    for k in keys:
        v = row.get(k)
        if isinstance(v, str) and v.strip():
            return v
    return None

def evaluate(
    results_path: Path,
    refs_path: Path,
    out_path: Path | None = None,
    hypothesis: str = "final",
    oracle: bool = True,
    language_source: str = "route",
    decision_filter: list[str] | None = None,
) -> dict[str, Any]:
    refs = load_refs(refs_path)
    results = read_jsonl(results_path)

    decision_filter_set = None
    if decision_filter:
        decision_filter_set = {str(d).strip().lower() for d in decision_filter if str(d).strip()}

    rows: list[EvalRow] = []
    for r in results:
        segment_id = str(r.get("segment_id") or "").strip()
        if not segment_id or segment_id not in refs:
            continue

        ref_row = refs[segment_id]
        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
        if ref_text is None:
            continue

        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
        if hyp_text is None:
            continue

        route_lang = None
        route = r.get("route") or {}
        if isinstance(route, dict):
            route_lang = route.get("language_hint")

        ref_lang = None
        if language_source == "refs":
            v = ref_row.get("lang") or ref_row.get("language")
            if isinstance(v, str) and v.strip():
                ref_lang = v.strip()
        elif language_source != "route":
            raise ValueError("language_source must be 'route' or 'refs'")

        lang = ref_lang or route_lang

        decision = None
        ver = r.get("verification") or {}
        if isinstance(ver, dict):
            decision = ver.get("decision")
        if decision_filter_set is not None:
            dec_norm = str(decision or "").strip().lower()
            if dec_norm not in decision_filter_set:
                continue

        duration_sec = None
        qc = r.get("qc") or {}
        if isinstance(qc, dict):
            try:
                duration_sec = float(qc.get("duration_sec")) if qc.get("duration_sec") is not None else None
            except (TypeError, ValueError):
                duration_sec = None
        bucket = duration_bucket(duration_sec)

        ref_norm = normalize_text(ref_text, lang=lang)
        hyp_norm = normalize_text(hyp_text, lang=lang)
        w = float(wer(ref_norm, hyp_norm))
        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))

        oracle_best_wer = None
        oracle_best_cer = None
        oracle_best_key = None
        selection_gap_wer = None

        if oracle:
            cand_wer: dict[str, float] = {}
            cand_cer: dict[str, float] = {}
            for key, text in extract_all_candidates(result_row=r).items():
                ref_n = ref_norm
                hyp_n = normalize_text(text, lang=lang)
                cand_wer[key] = float(wer(ref_n, hyp_n))
                cand_cer[key] = float(cer(ref_n.replace(" ", ""), hyp_n.replace(" ", "")))

            if cand_wer:
                oracle_best_key, oracle_best_wer = min(cand_wer.items(), key=lambda kv: kv[1])
                oracle_best_cer = cand_cer.get(oracle_best_key)
                selection_gap_wer = float(w) - float(oracle_best_wer)

        rows.append(
            EvalRow(
                segment_id=segment_id,
                language=lang,
                decision=str(decision) if decision is not None else None,
                duration_sec=duration_sec,
                duration_bucket=bucket,
                ref=ref_text,
                hyp=hyp_text,
                wer=w,
                cer=c,
                oracle_best_wer=oracle_best_wer,
                oracle_best_cer=oracle_best_cer,
                oracle_best_key=oracle_best_key,
                selection_gap_wer=selection_gap_wer,
            )
        )

    by_lang: dict[str, list[float]] = {}
    by_dec: dict[str, list[float]] = {}
    by_bucket: dict[str, list[float]] = {}
    by_lang_cer: dict[str, list[float]] = {}
    by_dec_cer: dict[str, list[float]] = {}
    by_bucket_cer: dict[str, list[float]] = {}
    selection_gaps: list[float] = []
    oracle_best_wers: list[float] = []
    for row in rows:
        if row.language:
            by_lang.setdefault(row.language, []).append(row.wer)
            by_lang_cer.setdefault(row.language, []).append(row.cer)
        if row.decision:
            by_dec.setdefault(row.decision, []).append(row.wer)
            by_dec_cer.setdefault(row.decision, []).append(row.cer)
        if row.duration_bucket:
            by_bucket.setdefault(row.duration_bucket, []).append(row.wer)
            by_bucket_cer.setdefault(row.duration_bucket, []).append(row.cer)
        if row.selection_gap_wer is not None:
            selection_gaps.append(float(row.selection_gap_wer))
        if row.oracle_best_wer is not None:
            oracle_best_wers.append(float(row.oracle_best_wer))

    summary = {
        "hypothesis": hypothesis,
        "pairs_evaluated": len(rows),
        "wer_mean": (sum(r.wer for r in rows) / len(rows)) if rows else None,
        "cer_mean": (sum(r.cer for r in rows) / len(rows)) if rows else None,
        "wer_p50": _percentile([r.wer for r in rows], 0.50),
        "wer_p90": _percentile([r.wer for r in rows], 0.90),
        "wer_p95": _percentile([r.wer for r in rows], 0.95),
        "cer_p50": _percentile([r.cer for r in rows], 0.50),
        "cer_p90": _percentile([r.cer for r in rows], 0.90),
        "cer_p95": _percentile([r.cer for r in rows], 0.95),
        "segment_error_rate": (sum(1 for r in rows if r.wer > 0.0) / len(rows)) if rows else None,
        "pct_wer_gt_5pct": (sum(1 for r in rows if r.wer > 0.05) / len(rows)) if rows else None,
        "pct_wer_gt_10pct": (sum(1 for r in rows if r.wer > 0.10) / len(rows)) if rows else None,
        "wer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang.items())},
        "wer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec.items())},
        "wer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket.items())},
        "cer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang_cer.items())},
        "cer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec_cer.items())},
        "cer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket_cer.items())},
        "oracle": {
            "enabled": bool(oracle),
            "oracle_best_wer_mean": (sum(oracle_best_wers) / len(oracle_best_wers)) if oracle_best_wers else None,
            "selection_gap_wer_mean": (sum(selection_gaps) / len(selection_gaps)) if selection_gaps else None,
            "selection_gap_wer_p90": _percentile(selection_gaps, 0.90),
        },
    }

    if out_path is not None:
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")

    return summary

def extract_hypothesis_text(result_row: dict[str, Any], hypothesis: str) -> str | None:
    """
    hypothesis:
      - "final": uses result_row["final"]["text"] (falls back to rover if missing)
      - "rover" (default): uses result_row["rover"]["text"]
      - "asr:<key>": uses result_row["asr"][<key>]["text"]
    """
    hypothesis = (hypothesis or "rover").strip()
    if hypothesis == "final":
        final = result_row.get("final") or {}
        if isinstance(final, dict) and isinstance(final.get("text"), str):
            return final["text"]
        hypothesis = "rover"
    if hypothesis == "rover":
        rover = result_row.get("rover") or {}
        if isinstance(rover, dict) and isinstance(rover.get("text"), str):
            return rover["text"]
        return None

    if hypothesis.startswith("asr:"):
        key = hypothesis[len("asr:") :].strip()
        asr = result_row.get("asr") or {}
        if isinstance(asr, dict):


**Planning minimal results.jsonl format**

The evaluation expects specific keys, so I'll produce results.jsonl with the minimal required fields—setting language_hint, text, and duration from audio metadata while ignoring others. I'll also prepare to link segments by ID and source path for evaluation and check bench references for integration.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 712
drwxrwxr-x  3 ubuntu ubuntu   4096 Jan 15 18:49 .
drwxrwxr-x  8 ubuntu ubuntu   4096 Jan 15 18:49 ..
drwxrwxr-x 14 ubuntu ubuntu   4096 Jan 15 18:49 audio
-rw-rw-r--  1 ubuntu ubuntu   5164 Jan 15 18:49 manifest.json
-rw-rw-r--  1 ubuntu ubuntu 705946 Jan 15 18:49 refs.jsonl


**Identifying refs.jsonl structure**

refs.jsonl likely contains fields such as segment_id, text, lang, and possibly path; next, I'll examine its contents to confirm.


python3 - <<'PY'
from pathlib import Path
path = Path('data/bench/maya_hf_v2_100/refs.jsonl')
with path.open() as f:
    for i in range(3):
        print(f.readline().strip())
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{"segment_id": "fleurs__as_in__test__285__1764-5006634a7372640e", "text": "উপস্থিত থকা লোকৰ সংখ্যা ইমানেই বেছি আছিল যে ছেইণ্ট পিটাৰৰ স্কুৱেৰত অন্তিম সংস্কাৰলৈ সকলো লোক যোৱাটো সম্ভৱপৰ নাছিল", "lang": "asm", "audio_relpath": "asm/fleurs__as_in__test__285__1764.wav", "duration_sec": 7.74, "source_dataset": "google/fleurs", "source_config": "as_in", "source_split": "test", "source_id": "1764", "source_index": 285}
{"segment_id": "fleurs__as_in__test__204__1790-3d591a345e393aec", "text": "তেওঁৰ কাম এনে স্বীকৃত মানদণ্ড আৰু সুক্ষ্ম আছিল যে তেওঁ ডাক টিকট সংগ্ৰহকাৰীসকলৰ মাজৰ অতি কমসংখ্যক পৰিচিত নামসমূহৰ অন্যতম আছিল কিছুমানে কেৱল তেওঁৰ কৰ্মৰাজীৰ সংগ্ৰহতেই বিশেষজ্ঞ আছিল", "lang": "asm", "audio_relpath": "asm/fleurs__as_in__test__204__1790.wav", "duration_sec": 15.78, "source_dataset": "google/fleurs", "source_config": "as_in", "source_split": "test", "source_id": "1790", "source_index": 204}
{"segment_id": "fleurs__as_in__test__950__1936-0672af7c54333693", "text": "\"পেৰীয়ে ঘোষণা কৰিছিল যে তেওঁ \"টেক্সাছলৈ ঘূৰি যাম দলৰ কমিটিৰ ৰাতিৰ বৈঠকৰ ফলাফল নিৰূপণ  কৰিবলৈ এই প্ৰতিযোগিতাত আগুৱাই যাবলৈ মোৰ বাবে পথ আছেনে নাই থিৰাং কৰিবলৈ\" কিন্তু পিছত ক'লে যে তেওঁ প্ৰতিযোগিতাত থাকিব আৰু ২১তম দক্ষিণ কেৰোলিনা প্ৰাইমেৰীত প্ৰতিদ্বন্দ্বিতা কৰিব।\"\"", "lang": "asm", "audio_relpath": "asm/fleurs__as_in__test__950__1936.wav", "duration_sec": 25.8, "source_dataset": "google/fleurs", "source_config": "as_in", "source_split": "test", "source_id": "1936", "source_index": 950}


**Clarifying output data requirements**

The refs.jsonl has audio path info for producing outputs, while existing runs copy audio data to outputs with extra metadata. For new runs, generating a minimal results.jsonl with segment IDs, language hints, and transcripts may suffice without durations, since WER is the key metric and duration isn’t critical. Next, I'll check the eval usage in cli.py to confirm how WER per language is computed.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from pathlib import Path

import typer
from dotenv import load_dotenv

from maya_transcribe.audio import compute_qc, ffprobe, normalize_audio
from maya_transcribe.config import Settings
from maya_transcribe.dataset import find_segments_dir
from maya_transcribe.ids import segment_identity
from maya_transcribe.io import write_jsonl
from maya_transcribe.r2 import R2Client
from maya_transcribe.tar import safe_extract_tar

load_dotenv(dotenv_path=Path(".env"), override=False)

app = typer.Typer(add_completion=False, no_args_is_help=True)
bench_app = typer.Typer(add_completion=False, no_args_is_help=True, help="Benchmarking utilities.")
app.add_typer(bench_app, name="bench")

@app.command()
def download(
    video_id: str = typer.Option(..., help="YouTube video id (used to locate the tar in R2)."),
    out: Path = typer.Option(Path("data"), help="Output directory."),
    r2_key: str | None = typer.Option(None, help="Exact R2 object key (skips search)."),
    bucket: str | None = typer.Option(None, help="R2 bucket override (defaults to env R2_BUCKET)."),
    prefix: str | None = typer.Option(None, help="Optional R2 prefix override."),
    use_db: bool = typer.Option(False, help="Fetch `r2_tar_key` from Supabase Postgres (DATABASE_URL)."),
    dry_run: bool = typer.Option(False, help="Only show matched key(s); do not download."),
) -> None:
    """
    Download the cleaned/segmented tar for a video from R2 and extract it.
    """
    settings = Settings.load()
    r2 = R2Client.from_settings(settings)
    if bucket:
        r2 = R2Client(
            endpoint_url=r2.endpoint_url,
            access_key_id=r2.access_key_id,
            secret_access_key=r2.secret_access_key,
            bucket=bucket,
        )

    out.mkdir(parents=True, exist_ok=True)
    r2_prefix = prefix if prefix is not None else settings.r2_prefix

    matches: list[str]
    if use_db and not r2_key:
        from maya_transcribe.db import fetch_video_meta

        meta = fetch_video_meta(video_id)
        if meta and meta.r2_tar_key:
            r2_key = meta.r2_tar_key

    if r2_key:
        matches = [r2_key]
    else:
        best = r2.find_video_tar_key(video_id=video_id, prefix=r2_prefix)
        if not best:
            typer.echo("No matching objects found in R2.")
            raise typer.Exit(code=2)
        matches = [best]

    if dry_run:
        for key in matches:
            typer.echo(key)
        return

    key = matches[0]
    tar_path = r2.download_to(key=key, out_dir=out / video_id)
    extract_dir = out / video_id
    safe_extract_tar(tar_path, extract_dir=extract_dir)
    segments_dir = find_segments_dir(extract_dir)
    typer.echo(f"Downloaded: {tar_path}")
    typer.echo(f"Extracted to: {extract_dir}")
    if segments_dir:
        typer.echo(f"Segments dir: {segments_dir}")

@app.command()
def preprocess(
    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
    out: Path = typer.Option(Path("outputs/preprocessed"), help="Output directory for normalized audio + QC."),
    sample_rate: int = typer.Option(16000, help="Target sample rate."),
    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
) -> None:
    """
    Normalize audio to 16kHz mono WAV, optionally trim silence, and write QC metrics.
    """
    out.mkdir(parents=True, exist_ok=True)

    audio_files: list[Path] = []
    for ext in ("*.wav", "*.mp3", "*.m4a", "*.flac", "*.ogg", "*.opus"):
        audio_files.extend(sorted(segments_dir.rglob(ext)))
    audio_files = sorted(set(audio_files))

    if not audio_files:
        typer.echo(f"No audio files found under: {segments_dir}")
        raise typer.Exit(code=2)

    rows = []
    norm_dir = out / "audio"
    for src in audio_files:
        ident = segment_identity(src, root=segments_dir)
        segment_id = ident.segment_id
        dst = norm_dir / f"{segment_id}.wav"
        normalize_audio(
            src,
            dst,
            sample_rate=sample_rate,
            mono=True,
            trim_silence=trim_silence,
            silence_threshold_db=silence_threshold_db,
            min_silence_sec=min_silence_sec,
        )
        info = ffprobe(dst)
        qc = compute_qc(dst, sample_rate=sample_rate)
        rows.append(
            {
                "segment_id": segment_id,
                "base_id": ident.base_id,
                "relpath": ident.relpath,
                "uid": ident.uid,
                "src_path": str(src),
                "wav_path": str(dst),
                "ffprobe": {
                    "duration_sec": info.duration_sec,
                    "sample_rate": info.sample_rate,
                    "channels": info.channels,
                },
                "qc": {
                    "duration_sec": qc.duration_sec,
                    "rms_dbfs": qc.rms_dbfs,
                    "peak_dbfs": qc.peak_dbfs,
                    "clipping_fraction": qc.clipping_fraction,
                },
            }
        )

    write_jsonl(out / "qc.jsonl", rows)
    typer.echo(f"Wrote: {out / 'qc.jsonl'}")
    typer.echo(f"Normalized audio: {norm_dir}")

@app.command()
def lid(
    audio: Path = typer.Option(..., exists=True, dir_okay=False, help="Audio file path."),
    model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace model id."),
    top_k: int = typer.Option(5, help="Return top-K predictions."),
) -> None:
    """
    Spoken language identification (MMS-LID).
    """
    from maya_transcribe.lid import MMSLID

    model = MMSLID(model_id=model_id)
    res = model.predict(audio_path=audio, top_k=top_k)
    for p in res.predictions:
        typer.echo(f"{p.code}\t{p.prob:.4f}")

@app.command()
def asr(
    audio: Path = typer.Option(..., exists=True, dir_okay=False, help="Audio file path."),
    language: str | None = typer.Option(None, help="Language hint (ISO-639-1 or ISO-639-3)."),
    model: str = typer.Option("large-v3", help="Whisper model size or path for faster-whisper."),
    beam_size: int = typer.Option(5, help="Beam size."),
    best_of: int = typer.Option(5, help="Best-of (sampling candidates)."),
    temperature: float = typer.Option(0.0, help="Temperature."),
) -> None:
    """
    Transcribe a single file with faster-whisper.
    """
    from maya_transcribe.asr import FasterWhisperASR

    asr_backend = FasterWhisperASR(model_size_or_path=model)
    res = asr_backend.transcribe(
        audio_path=audio, language=language, beam_size=beam_size, best_of=best_of, temperature=temperature
    )
    typer.echo(res.text)

@app.command()
def gemini(
    audio: Path = typer.Option(..., exists=True, dir_okay=False, help="Audio file path."),
    language: str | None = typer.Option(None, help="Language hint (e.g., hin/tam/eng)."),
    model: str = typer.Option("gemini-2.0-flash", help="Gemini model name."),
    prompt_file: Path | None = typer.Option(Path("Prompt.md"), help="Prompt file (defaults to Prompt.md)."),
    max_output_tokens: int = typer.Option(512, help="Max output tokens."),
) -> None:
    """
    Transcribe a single file with Gemini (audio-native LLM). Output is parsed/validated strictly.
    """
    from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template

    prompt = load_prompt_template(prompt_file)
    backend = GeminiASR(config=GeminiConfig(model=model, max_output_tokens=max_output_tokens), prompt_template=prompt)
    res = backend.transcribe(audio_path=audio, language=language)
    typer.echo(res.text)

@app.command()
def run(
    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
    out: Path = typer.Option(Path("outputs/run"), help="Output directory."),
    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
    skip_whisper: bool = typer.Option(False, help="Skip faster-whisper decoding (use only other ASR backends)."),
    vllm_asr_model: str | None = typer.Option(
        None, help="vLLM multimodal ASR model id (e.g., mistralai/Voxtral-Mini-3B-2507)."
    ),
    vllm_gpu_memory_utilization: float | None = typer.Option(
        None, help="vLLM GPU memory utilization fraction (e.g., 0.15 for low free memory)."
    ),
    vllm_max_model_len: int | None = typer.Option(None, help="Override vLLM max_model_len to reduce KV cache size."),
    vllm_transcribe_prompt_file: Path | None = typer.Option(None, help="Text prompt file for vLLM transcription."),


**Planning evaluation integration**

The existing `bench.py` likely runs model evaluations via pipeline, but crafting a custom results.jsonl matching eval's input will be simpler for new models, including extra metadata. I'll also update the results table to add columns for Omnilingual and Seamless and run WER computations per language using the current eval script.


sed -n '1,220p' maya_transcribe/bench.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
import os
import random
import re
import wave
from dataclasses import dataclass
from pathlib import Path
from typing import Any

import numpy as np

from maya_transcribe.ids import segment_identity
from maya_transcribe.io import read_jsonl, write_jsonl

@dataclass(frozen=True)
class HFSource:
    dataset: str
    config: str | None
    split: str
    audio_key: str
    text_key: str
    id_key: str | None = None
    trust_remote_code: bool = False
    name: str | None = None

_SAFE_NAME_RE = re.compile(r"[^a-zA-Z0-9._-]+")

def _safe_name(value: str) -> str:
    value = (value or "").strip()
    value = value.replace("/", "_")
    value = value.replace("\\", "_")
    value = _SAFE_NAME_RE.sub("_", value)
    value = value.strip("._-")
    return value or "x"

def _ensure_hf_token_env() -> None:
    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
    if token and not os.getenv("HUGGINGFACE_HUB_TOKEN"):
        os.environ["HUGGINGFACE_HUB_TOKEN"] = token

def _write_wav_mono16(path: Path, audio: np.ndarray, sampling_rate: int) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)

    wav = np.asarray(audio, dtype=np.float32)
    if wav.ndim == 2:
        wav = wav.mean(axis=1)
    wav = np.clip(wav, -1.0, 1.0)
    pcm16 = (wav * 32767.0).astype(np.int16)

    with wave.open(str(path), "wb") as wf:
        wf.setnchannels(1)
        wf.setsampwidth(2)
        wf.setframerate(int(sampling_rate))
        wf.writeframes(pcm16.tobytes())

def _extract_audio(example: dict[str, Any], audio_key: str) -> tuple[np.ndarray, int] | None:
    audio = example.get(audio_key)
    if not isinstance(audio, dict):
        return None
    arr = audio.get("array")
    sr = audio.get("sampling_rate")
    if arr is None or sr is None:
        return None
    try:
        sr_i = int(sr)
    except (TypeError, ValueError):
        return None
    return np.asarray(arr, dtype=np.float32), sr_i

def _extract_text(example: dict[str, Any], text_key: str) -> str | None:
    text = example.get(text_key)
    if not isinstance(text, str):
        return None
    text = text.strip()
    return text or None

def default_sources() -> dict[str, list[HFSource]]:
    """
    Default benchmark sources for the 12 target languages:
    Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English.
    """
    kathbath = "ai4bharat/Kathbath"
    fleurs = "google/fleurs"

    # Canonical ISO-639-3 codes aligned to MMS-LID output (and language.py mappings).
    return {
        "hin": [HFSource(dataset=kathbath, config="hindi", split="valid", audio_key="audio_filepath", text_key="text")],
        "tam": [HFSource(dataset=kathbath, config="tamil", split="valid", audio_key="audio_filepath", text_key="text")],
        "tel": [HFSource(dataset=kathbath, config="telugu", split="valid", audio_key="audio_filepath", text_key="text")],
        "kan": [HFSource(dataset=kathbath, config="kannada", split="valid", audio_key="audio_filepath", text_key="text")],
        "mal": [
            HFSource(dataset=kathbath, config="malayalam", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "ben": [HFSource(dataset=kathbath, config="bengali", split="valid", audio_key="audio_filepath", text_key="text")],
        "guj": [
            HFSource(dataset=kathbath, config="gujarati", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "pan": [
            HFSource(dataset=kathbath, config="punjabi", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "mar": [
            HFSource(dataset=kathbath, config="marathi", split="valid", audio_key="audio_filepath", text_key="text")
        ],
        "ory": [HFSource(dataset=kathbath, config="odia", split="valid", audio_key="audio_filepath", text_key="text")],
        # Kathbath doesn't include Assamese/English; use FLEURS (requires trust_remote_code=True).
        "asm": [
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="test",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="validation",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="as_in",
                split="train",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
        ],
        "eng": [
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="test",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="validation",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
            HFSource(
                dataset=fleurs,
                config="en_us",
                split="train",
                audio_key="audio",
                text_key="transcription",
                trust_remote_code=True,
            ),
        ],
    }

def build_benchmark_corpus(
    *,
    out_dir: Path,
    samples_per_lang: int = 200,
    seed: int = 0,
    shuffle: bool = True,
    langs: list[str] | None = None,
    min_duration_sec: float = 1.0,
    max_duration_sec: float = 30.0,
    force: bool = False,
) -> dict[str, Any]:
    """
    Materialize a local benchmark corpus:
      - `out_dir/audio/<lang>/*.wav`
      - `out_dir/refs.jsonl` with `segment_id`, `text`, `lang`, `audio_relpath`, `source_*`
      - `out_dir/manifest.json` with build config + counts
    """
    _ensure_hf_token_env()

    if out_dir.exists():
        existing = [p for p in out_dir.iterdir()]
        if existing and not force:
            raise RuntimeError(f"Refusing to overwrite non-empty directory: {out_dir} (pass force=True)")
        if force:
            for p in sorted(out_dir.rglob("*"), reverse=True):
                if p.is_file() or p.is_symlink():
                    p.unlink()
                elif p.is_dir():
                    p.rmdir()

    out_dir.mkdir(parents=True, exist_ok=True)
    audio_root = out_dir / "audio"

    sources_by_lang = default_sources()
    target_langs = langs or sorted(sources_by_lang.keys())
    target_langs = [l.strip().lower() for l in target_langs if l.strip()]

    try:
        from datasets import load_dataset
    except Exception as e:  # pragma: no cover
        raise RuntimeError("Benchmark corpus build requires `datasets`. Install with: pip install -e '.[bench]'") from e

    rows: list[dict[str, Any]] = []
    counts: dict[str, int] = {}
    skipped: dict[str, int] = {}

    for lang in target_langs:
        sources = sources_by_lang.get(lang)
        if not sources:
            continue


**Searching for evaluation command**

I'll look for a CLI option like `maya_transcribe eval` to confirm if there's a built-in way to evaluate using results.jsonl before creating custom scripts.


grep -n "def eval" -n maya_transcribe/cli.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
337:def eval(


sed -n '320,440p' maya_transcribe/cli.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        hf_indicconformer_decode=hf_indicconformer_decode,
        gemini_model=gemini_model,
        gemini_prompt_path=gemini_prompt_file,
        gemini_hard_only=gemini_hard_only,
        ctc_model=ctc_model,
        ctc_model_map=ctc_map,
        trim_silence=trim_silence,
        silence_threshold_db=silence_threshold_db,
        min_silence_sec=min_silence_sec,
        limit=limit,
        thresholds=thresholds,
    )
    typer.echo(f"Wrote: {results_path}")
    typer.echo(f"Wrote: {out / 'summary.json'}")

@app.command()
def eval(
    results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
    refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
    out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
    hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
    oracle: bool = typer.Option(True, help="Compute oracle-best and selection-gap stats across all candidates."),
    language_source: str = typer.Option("route", help="Use language from: 'route' (LID) or 'refs' (refs.jsonl)."),
    decision: list[str] = typer.Option([], help="Filter to verification decision(s): accept/human/reject (repeatable)."),
) -> None:
    """
    Compute WER/CER for a hypothesis against a reference JSONL keyed by `segment_id`.
    """
    from maya_transcribe.eval import evaluate

    summary = evaluate(
        results_path=results,
        refs_path=refs,
        out_path=out,
        hypothesis=hypothesis,
        oracle=oracle,
        language_source=language_source,
        decision_filter=decision or None,
    )
    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))

@bench_app.command("build")
def bench_build(
    out: Path = typer.Option(Path("data/bench/maya_hf_v0"), help="Output benchmark corpus directory."),
    samples_per_lang: int = typer.Option(200, help="Target samples per language."),
    seed: int = typer.Option(0, help="Shuffle seed."),
    shuffle: bool = typer.Option(True, help="Shuffle within each source before selecting."),
    lang: list[str] = typer.Option([], help="Limit to ISO-639-3 language codes (repeatable)."),
    min_duration_sec: float = typer.Option(1.0, help="Min audio duration."),
    max_duration_sec: float = typer.Option(30.0, help="Max audio duration."),
    force: bool = typer.Option(False, help="Delete and rebuild output directory if it exists."),
) -> None:
    """
    Build a local benchmark corpus from Hugging Face datasets.
    """
    from maya_transcribe.bench import build_benchmark_corpus

    manifest = build_benchmark_corpus(
        out_dir=out,
        samples_per_lang=samples_per_lang,
        seed=seed,
        shuffle=shuffle,
        langs=lang or None,
        min_duration_sec=min_duration_sec,
        max_duration_sec=max_duration_sec,
        force=force,
    )
    typer.echo(json.dumps({"wrote": str(out), "total_rows": manifest.get("total_rows"), "counts": manifest.get("counts")}, indent=2))

@bench_app.command("run")
def bench_run(
    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
    out: Path | None = typer.Option(None, help="Output directory (defaults under outputs/bench/)."),
    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
    skip_whisper: bool = typer.Option(False, help="Skip faster-whisper decoding (use only other ASR backends)."),
    vllm_asr_model: str | None = typer.Option(
        None, help="vLLM multimodal ASR model id (e.g., mistralai/Voxtral-Mini-3B-2507)."
    ),
    vllm_gpu_memory_utilization: float | None = typer.Option(
        None, help="vLLM GPU memory utilization fraction (e.g., 0.15 for low free memory)."
    ),
    vllm_max_model_len: int | None = typer.Option(None, help="Override vLLM max_model_len to reduce KV cache size."),
    vllm_transcribe_prompt_file: Path | None = typer.Option(None, help="Text prompt file for vLLM transcription."),
    vllm_transcribe_prompt: str | None = typer.Option(None, help="Inline prompt text for vLLM transcription."),
    vllm_proofread: bool = typer.Option(False, help="Run vLLM proofread pass when supported (audio + transcript)."),
    vllm_proofread_prompt: str | None = typer.Option(None, help="Override the proofread prompt template."),
    lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
    require_lid_backend: bool = typer.Option(True, help="Fail if MMS-LID backend can't initialize."),
    transformers_whisper_model: list[str] = typer.Option(
        [], help="Extra whisper-family HF models to run via transformers (repeatable)."
    ),
    transformers_whisper_model_map: Path | None = typer.Option(
        None, help="JSON map {lang_code: hf_model_id} for per-language Whisper-family HF models (transformers)."
    ),
    # MMS ASR for CTC-based model diversity (per master.md plan).
    mms_asr_model: str | None = typer.Option(
        None, help="Optional MMS ASR model id (e.g., facebook/mms-1b-all) for CTC-based diversity."
    ),
    # IndicConformer for Conformer architecture diversity (per master.md plan).
    indicconformer_model_map: Path | None = typer.Option(
        None, help="JSON map {lang_code: nemo_model_name} for per-language IndicConformer models."
    ),
    indicconformer_default_model: str | None = typer.Option(
        None, help="Default IndicConformer NeMo model name for architecture diversity."
    ),
    hf_indicconformer_model: str | None = typer.Option(
        None, help="HF IndicConformer model id (e.g., ai4bharat/indic-conformer-600m-multilingual)."
    ),
    hf_indicconformer_decode: str = typer.Option("rnnt", help="HF IndicConformer decode mode (rnnt or ctc)."),
    ctc_model: str | None = typer.Option(None, help="Optional HF CTC model id for audio-grounded fit scoring."),
    ctc_model_map: Path | None = typer.Option(None, help="JSON map {lang_code: hf_model_id} for CTC scoring."),
    require_ctc: bool = typer.Option(True, help="When CTC is enabled, require a CTC score to accept."),
    min_ctc_margin: float = typer.Option(0.05, help="Min (second-best - best) CTC loss margin to auto-accept."),
    trim_silence: bool = typer.Option(
        False, help="Trim leading/trailing silence (ffmpeg silenceremove). Off by default for benchmarking."
    ),
    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),


**Deciding script placement**

Considering creating a dedicated evaluation script for OmniASR and Seamless M4T, likely under a new `scripts/` folder for clarity and reproducibility, since inline Python would be too long; this aligns with the user's acceptance of adding scripts despite prior cleanup thoughts.


**Planning isolated evaluation scripts**

Decided to create reproducible evaluation scripts under a new `scripts/` folder, adding them via patch; confirmed isolated venv usage with necessary dependencies like `transformers`, `soundfile`, and `torchaudio` for OmniASR and Seamless M4T per user instructions.


source .venv_omni/bin/activate && python - <<'PY'
import sys
print(sys.executable)
try:
    import torch
    print('torch', torch.__version__)
except Exception as e:
    print('torch import failed', e)
try:
    import transformers
    print('transformers', transformers.__version__)
except Exception as e:
    print('transformers import failed', e)
try:
    import soundfile as sf
    print('soundfile', sf.__version__)
except Exception as e:
    print('soundfile import failed', e)
PY


Output:
Exit code: 0
Wall time: 2.2 seconds
Output:
/home/ubuntu/maya-transcribe/.venv_omni/bin/python
torch 2.8.0+cu128
transformers 4.57.5
soundfile 0.13.1


**Verifying OmniASR package install**


source .venv_omni/bin/activate && python - <<'PY'
try:
    import omnilingual_asr
    print('omnilingual_asr', omnilingual_asr.__file__)
except Exception as e:
    print('omnilingual_asr import failed', e)
PY


Output:
Exit code: 0
Wall time: 4.3 seconds
Output:
omnilingual_asr /home/ubuntu/maya-transcribe/.venv_omni/lib/python3.10/site-packages/omnilingual_asr/__init__.py