Output:
Exit code: 0
Wall time: 0 seconds
Output:
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424930357016-618-f.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424930422043-883-f.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424931137843-955-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424931195916-955-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424930744004-955-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424932415771-97-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424931155214-955-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424930506078-883-f.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424931387082-955-m.m4a.wav
data/bench/maya_hf_v0/audio_only_ory/ory/Kathbath__844424931524866-824-m.m4a.wav


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


maya-transcribe run --segments-dir data/bench/maya_hf_v0/audio_only_ory --out outputs/bench/maya_hf_v0/odia_only_largev3 --whisper-model large-v3 --limit 200


Output:
Exit code: 0
Wall time: 643.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 09:26:22.943479: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:10:31
Wrote: outputs/bench/maya_hf_v0/odia_only_largev3/results.jsonl
Wrote: outputs/bench/maya_hf_v0/odia_only_largev3/summary.json


Summary: auto


maya-transcribe eval --results outputs/bench/maya_hf_v0/odia_only_largev3/results.jsonl --refs data/bench/maya_hf_v0/refs.jsonl --out outputs/bench/maya_hf_v0/odia_only_largev3/eval.json


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 1.415626998001998,
  "cer_mean": 1.2236990777808503,
  "wer_p50": 1.0,
  "wer_p90": 1.8,
  "wer_p95": 2.5,
  "cer_p50": 1.0,
  "cer_p90": 1.671641791044776,
  "cer_p95": 2.090909090909091,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 1.0,
  "wer_by_language_mean": {
    "asm": 1.8958333333333335,
    "ben": 1.191919191919192,
    "guj": 1.0,
    "npi": 1.4472902097902098,
    "ory": 1.3614551338327567,
    "pus": 1.0
  },
  "wer_by_decision_mean": {
    "accept": 1.0900450312215018,
    "human": 1.4694737832142415,
    "reject": 1.9462243003909674
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 1.575255069005069,
    "3-10s": 1.3258362080627708
  },
  "cer_by_language_mean": {
    "asm": 1.123582995951417,
    "ben": 1.1057692307692308,
    "guj": 1.0,
    "npi": 1.4034170597985465,
    "ory": 1.2045097493052348,
    "pus": 1.0
  },
  "cer_by_decision_mean": {
    "accept": 1.0179624061943804,
    "human": 1.2833448284894593,
    "reject": 1.3725311282298642
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 1.2536468453466363,
    "3-10s": 1.206853458525096
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 1.1152319624819627,
    "selection_gap_wer_mean": 0.3003950355200356,
    "selection_gap_wer_p90": 0.40000000000000036
  }
}


**Evaluating Odia ASR Models**

It looks like the WER for Odia is really high at 1.415 because the current whisper model doesn't support it properly. To improve this, I should consider using a better model like IndicWhisper or some form of transliteration. It’s good that the mapping is working without errors. I'll need to update the evaluation process so it doesn't fail. We might also need to integrate `kavyamanohar/AI4B-Indicwhisper-or` into the pipeline for better results with Odia.


**Improving Model Selection in Pipeline**

I’m realizing that the pipeline can run multiple transformer models for each segment, but running a model for every language would be inefficient. I need to create a per-language model selection system that allows routing based on language, which sounds a bit complex. But I can implement a transformer whisper model map to only run specific models when the language matches. This would enhance the pipeline's flexibility and robustness, though it will involve significant code changes for mapping and lazy loading models when needed.


**Implementing Language-Specific Model Routing**

I’m considering whether we should load multiple models in the pipeline since there’s no compute constraint, but we still want a robust setup. Running separate pipelines for each language works, but it’s not ideal for mixed-language datasets. Instead, I could create mappings like `whisper_model_map` for language codes to model sizes and `transformers_whisper_model_map` for languages to model IDs. This way, we can select the best models for each segment based on language, aligning with the goal of minimal WER. It seems like a significant improvement, but I’m wondering if I have enough time to implement it.


**Considering Pipeline Adjustments**

I’m evaluating how the current pipeline works with a single FasterWhisperASR backend and multiple TransformersWhisperASR backends, which doesn't support per-language models. To change this, I could keep the existing whisper backend but add an optional argument `whisper_model_map` in the `run_pipeline`, mapping language codes to specific model paths. When processing each segment, I'd determine the `route_lang` from LID and select the appropriate whisper backend based on that. Using caching for models could streamline this process and enhance language support.


**Planning Model Implementation and Evaluation**

I need to ensure we run at least two hypotheses—deterministic and sample—for the chosen whisper backend. To manage transformer models, I can create a `transformers_whisper_model_map` that links languages to model IDs, plus an `extra_transformers_whisper_models` list for models that should always run.

For each segment, I’ll determine which models to instantiate, allowing user input for model selection. Since improving performance is crucial, I should focus on adding better models per language. However, it’s also essential to create a proper HF dataset and run benchmarks, though this could be time-intensive. For now, I can aim to run an improved model across all languages to see WER improvements. We'll work on updating the evaluation process and implementing this efficiently.


Input:
*** Begin Patch
*** Update File: maya_transcribe/eval.py
@@
 def evaluate(
     results_path: Path,
     refs_path: Path,
     out_path: Path | None = None,
     hypothesis: str = "final",
     oracle: bool = True,
+    language_source: str = "route",
 ) -> dict[str, Any]:
@@
-        lang = None
+        route_lang = None
         route = r.get("route") or {}
         if isinstance(route, dict):
-            lang = route.get("language_hint")
+            route_lang = route.get("language_hint")
+
+        ref_lang = None
+        if language_source == "refs":
+            v = ref_row.get("lang") or ref_row.get("language")
+            if isinstance(v, str) and v.strip():
+                ref_lang = v.strip()
+        elif language_source != "route":
+            raise ValueError("language_source must be 'route' or 'refs'")
+
+        lang = ref_lang or route_lang
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/eval.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
 def eval(
     results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
     refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
     out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
     hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
     oracle: bool = typer.Option(True, help="Compute oracle-best and selection-gap stats across all candidates."),
+    language_source: str = typer.Option("route", help="Use language from: 'route' (LID) or 'refs' (refs.jsonl)."),
 ) -> None:
@@
-    summary = evaluate(results_path=results, refs_path=refs, out_path=out, hypothesis=hypothesis, oracle=oracle)
+    summary = evaluate(
+        results_path=results,
+        refs_path=refs,
+        out_path=out,
+        hypothesis=hypothesis,
+        oracle=oracle,
+        language_source=language_source,
+    )
     typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
-    summary = evaluate(results_path=results_path, refs_path=refs_path, out_path=eval_path, hypothesis=hypothesis, oracle=True)
+    summary = evaluate(
+        results_path=results_path,
+        refs_path=refs_path,
+        out_path=eval_path,
+        hypothesis=hypothesis,
+        oracle=True,
+        language_source="refs",
+    )
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating code execution process**

I'm thinking about updating the `bench_run` output to reference the language, which seems like a great idea! Next, I need to run `python3 -m py_compile` for both `eval.py` and `cli.py` after the modifications. This way, I can ensure everything is compiled properly and ready to go. It's important to keep track of these steps to avoid any potential issues later!


python3 -m py_compile maya_transcribe/eval.py maya_transcribe/cli.py maya_transcribe/language.py


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:


maya-transcribe eval --results outputs/bench/maya_hf_v0/baseline/results.jsonl --refs data/bench/maya_hf_v0/refs.jsonl --out outputs/bench/maya_hf_v0/baseline/eval_refs.json --language-source refs


Output:
Exit code: 0
Wall time: 2.1 seconds
Output:
{
  "hypothesis": "final",
  "pairs_evaluated": 2188,
  "wer_mean": 0.958137176179309,
  "cer_mean": 0.6892070914668146,
  "wer_p50": 1.0,
  "wer_p90": 1.2,
  "wer_p95": 1.6666666666666667,
  "cer_p50": 0.7303370786516854,
  "cer_p90": 1.0,
  "cer_p95": 1.1979166666666667,
  "segment_error_rate": 0.9972577696526508,
  "pct_wer_gt_5pct": 0.9972577696526508,
  "pct_wer_gt_10pct": 0.9894881170018281,
  "wer_by_language_mean": {
    "asm": 1.2136130359678567,
    "ben": 0.9608822300148822,
    "eng": 0.9179840475746643,
    "guj": 0.6482344183594183,
    "hin": 0.6929420024420025,
    "kan": 0.9317456016206016,
    "mal": 1.3426665695415694,
    "mar": 1.0868887640137634,
    "ory": 1.5957731352468192,
    "pan": 0.8963747224997225,
    "tam": 0.6000177461427461,
    "tel": 1.1066519036519034
  },
  "wer_by_decision_mean": {
    "accept": 0.6600290144894885,
    "human": 1.068130605528151,
    "reject": 1.1809687125866046
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 1.1180530532878201,
    "10-30s": 1.3224812060525328,
    "3-10s": 0.8010562896560971
  },
  "cer_by_language_mean": {
    "asm": 1.0646248925218766,
    "ben": 0.6384613879852299,
    "eng": 0.9111160893139855,
    "guj": 0.3767729143532668,
    "hin": 0.5793067943604345,
    "kan": 0.6515604959195067,
    "mal": 1.0034571606808869,
    "mar": 0.6329671243519063,
    "ory": 1.2871967869778045,
    "pan": 0.5977032412878359,
    "tam": 0.29953127066920104,
    "tel": 0.7540866662783096
  },
  "cer_by_decision_mean": {
    "accept": 0.3739248809149772,
    "human": 0.7492229748256926,
    "reject": 1.0849652909765737
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.9660921611102244,
    "10-30s": 1.0225491434559282,
    "3-10s": 0.42673370170984454
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.8459395614677641,
    "selection_gap_wer_mean": 0.11219761471154517,
    "selection_gap_wer_p90": 0.11111111111111116
  }
}


**Reviewing evaluation processes**

I’m considering running benchmarks with a limit of 600 segments and checking the thresholds. This might take about 25 minutes, which seems manageable. I want to cover languages like asm, ben, and eng.

I’ve sorted out the audio files, but I need to review the structure. The first 600 segments should include some asm, but there’s a mismatch in how many were processed due to errors. Time to clarify why the numbers don’t seem to add up!


**Clarifying asm segment counts**

I’m trying to figure out why the Python script computed 164 for asm when it seems like there should be 200 rows. I noticed that the evaluation pairs for asm show 164, but there were no errors on asm.

So, I’m questioning why the counts don't match up. If the benchmark corpus counts show 200 for asm, but the evaluation indicates only 164, there might be an issue in the refs file causing this discrepancy with segment IDs. Time to dig deeper!


**Exploring segment mismatches**

I’m trying to understand why the evaluation script found 164 pairs for asm and 171 for eng. It seems like some segments might be skipped due to missing final text.

Earlier, the error count indicated there should be 171 for eng and 164 for asm. That suggests the pipeline did not process 200 segments for these languages. I expect there to be 200 audio files per language under the audio directory. So, why aren’t all segments matching up with the references? Time to investigate further!


**Examining dataset challenges**

I need to figure out why the build_benchmark_corpus for eng and asm used the FLEURS dataset. It involved writing audio file names, and the segment_id is created using the relative path linked to audio_root.

I saw that src_id extraction relies on id, but the src_name seems like it’s just "fleurs." This means filenames like "fleurs__1663.wav" appear frequently. I’m wondering if the id from the FLEURS might overlap across different data splits. It’s important to determine if these ids are indeed unique or not across splits to clarify the confusion!


**Investigating filename collisions**

I'm considering the possibility that if the ids overlap, file names could collide, leading to some segments being lost as newer writes overwrite earlier ones. This would effectively reduce the number of unique audio files and references.

Even though we built 200 rows per language, if file paths collide, we'd still get duplicate entries in refs.jsonl. The segment_id would remain the same for those colliding files, so I'll check the duplicates in refs.jsonl by counting unique segment_ids for each language. This might clarify why there are only 164 entries for asm instead of 200.


python3 - <<'PY'
import json
from collections import defaultdict

langs=defaultdict(set)
counts=defaultdict(int)
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    lang=r.get('lang')
    seg=r.get('segment_id')
    counts[lang]+=1
    langs[lang].add(seg)

for lang in sorted(counts):
    print(lang, 'rows', counts[lang], 'unique_ids', len(langs[lang]), 'dupes', counts[lang]-len(langs[lang]))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
asm rows 200 unique_ids 164 dupes 36
ben rows 200 unique_ids 200 dupes 0
eng rows 200 unique_ids 171 dupes 29
guj rows 200 unique_ids 200 dupes 0
hin rows 200 unique_ids 200 dupes 0
kan rows 200 unique_ids 200 dupes 0
mal rows 200 unique_ids 200 dupes 0
mar rows 200 unique_ids 200 dupes 0
ory rows 200 unique_ids 200 dupes 0
pan rows 200 unique_ids 200 dupes 0
tam rows 200 unique_ids 200 dupes 0
tel rows 200 unique_ids 200 dupes 0


**Patching for collisions**

I'm considering adjusting the build to avoid collisions and create a new corpus with the correct data in `data/bench/maya_hf_v1`. That seems better than keeping the old version. Since we want to ensure robustness, it makes sense to fix things now. I'll patch bench.py and update how we derive the source names to ensure uniqueness with the provided logic. Also, we should ensure we don't accidentally overwrite existing files. Let’s get on with the patch!


Input:
*** Begin Patch
*** Update File: maya_transcribe/bench.py
@@
                 if src.id_key and isinstance(ex.get(src.id_key), (str, int)):
                     src_id = str(ex.get(src.id_key))
                 elif isinstance(ex.get("id"), (str, int)):
                     src_id = str(ex.get("id"))
                 elif isinstance(ex.get("fname"), str):
@@
                 elif isinstance(ex.get("path"), str):
                     src_id = ex.get("path")

-                src_name = src.name or _safe_name(src.dataset.split("/")[-1])
+                src_name = src.name or _safe_name(src.dataset.split("/")[-1])
+                src_key = _safe_name("__".join([src_name, src.config or "default", src.split]))
                 lang_dir = audio_root / lang
-                fname = _safe_name(f"{src_name}__{src_id or i}")
+                fname = _safe_name(f"{src_key}__{src_id or i}")
                 wav_path = lang_dir / f"{fname}.wav"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/bench.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Updating segment stability**

I need to make sure duplicates are eliminated across splits. It’s also important to keep the segment_id stable during the rebuild process. Including a new relpath is fine. Additionally, I should think about updating the manifest counts to show unique segment IDs instead of just the ones that have been written. Lastly, I’ll compile bench.py again to finalize these updates. It feels like a bit of a process, but I’ll manage!


python3 -m py_compile maya_transcribe/bench.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Rebuilding benchmark corpus**

I need to rebuild the benchmark corpus since the current one has duplicates. The user requested a few hundred samples per language, but duplicates have reduced English to 171 and Assamese to 164; still, that's above 100. However, they might prefer 200 each. If we rebuild, we'll also need to rerun the pipeline on the new corpus, which is a big task. Yet, we could continue using the old baseline for approximate metrics while confirming unique counts.


**Considering dataset rebuilding**

So, we’re looking at around 171 and 164 samples respectively, which isn’t quite enough. I think we can rebuild the corpus, especially since our dataset caches are still warm; this could take less than 15 minutes. Rerunning the pipeline would take longer, around 1 hour and 44 minutes, but we might not need to do that because the current results are still roughly accurate. The user eventually wants 1000-2000 samples per language, so fixing duplicates is important. However, if we rebuild now and run the pipeline on it with the improved model, that's a big effort!


**Planning dataset improvements**

I think the best approach is to deliver the infrastructure, initial baseline results, and key fixes, including the Odia mapping, MMS CTC target language, and the duplicates bug. We should also outline guidance for next improvements like better ASR models and LID thresholds. Since the user wants a new benchmark dataset under their Hugging Face ID, I'll rebuild the corpus under a new path to avoid duplicates. This process should take around 5-10 minutes, and then we can push the dataset manifest to HF. However, we need to remember that our previous pipeline results were for v0, and we haven't processed v1 yet.


**Updating benchmark plans**

We could run the pipeline on a smaller subset or just for Odia, but considering the user asked for benchmarks, it makes sense to stick with the v0 results for now. Step 4 was about generating the sample corpus, and since we fixed a bug, I’m wondering if we should regenerate it. However, the plan is already in progress with step 5, and we discovered and fixed a bug. We might need to update the plan to include the rebuild and rerun, but it wasn't specifically requested. I think we’ll incorporate the running of the pipeline on the improved corpus into step 5.


**Revising the plan**

I think we should adjust our plan by adding a sub-step to "Rebuild corpus without duplicates" in step 4. Since we discovered a duplication bug, we should update the plan to reflect this. We could treat the rebuild as part of step 5 or separate it but need to avoid a stale plan. It’s okay to have a new step; we'll add "Rebuild corpus without duplicates" after step 4 and mark it as in progress. Also, we should mark step 5 as completed since we ran the pipeline and evaluate results, paving the way for step 6 to push the private Hugging Face dataset.


Explanation: Baseline benchmark run revealed two blockers: Odia language-code error in faster-whisper and duplicate IDs in FLEURS-derived samples. I fixed both and will rebuild the corpus before pushing to the Hub.


Output: Plan updated


**Rebuilding corpus without duplicates**

I need to rebuild the corpus without duplicates, aiming for a new set in `data/bench/maya_hf_v1`, keeping around 200 unique samples per language. The code changes mean file names will include splits to eliminate duplicates. I'll execute the command `maya-transcribe bench build --out data/bench/maya_hf_v1 --samples-per-lang 200 --force`. While I expect the build to take some time, I hope it will be faster since the dataset is already downloaded and prepared. Still, the process might require a timeout of about 30 minutes. Let's proceed!


maya-transcribe bench build --out data/bench/maya_hf_v1 --samples-per-lang 200 --seed 0 --shuffle


Output:
Exit code: 0
Wall time: 27.3 seconds
Output:
{
  "wrote": "data/bench/maya_hf_v1",
  "total_rows": 2400,
  "counts": {
    "asm": 200,
    "ben": 200,
    "eng": 200,
    "guj": 200,
    "hin": 200,
    "kan": 200,
    "mal": 200,
    "mar": 200,
    "ory": 200,
    "pan": 200,
    "tam": 200,
    "tel": 200
  }
}


**Verifying processing speed**

That's cool! It built in just 27 seconds, possibly due to how quickly the caches and audio arrays loaded. Now, I should verify that any duplicates have been eliminated. I'll quickly check the unique counts to ensure everything looks correct. It's always good to double-check these things to avoid any issues later on!


python3 - <<'PY'
import json
from collections import defaultdict

langs=defaultdict(set)
counts=defaultdict(int)
for line in open('data/bench/maya_hf_v1/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    lang=r.get('lang')
    seg=r.get('segment_id')
    counts[lang]+=1
    langs[lang].add(seg)

for lang in sorted(counts):
    print(lang, 'rows', counts[lang], 'unique_ids', len(langs[lang]), 'dupes', counts[lang]-len(langs[lang]))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
asm rows 200 unique_ids 164 dupes 36
ben rows 200 unique_ids 200 dupes 0
eng rows 200 unique_ids 171 dupes 29
guj rows 200 unique_ids 200 dupes 0
hin rows 200 unique_ids 200 dupes 0
kan rows 200 unique_ids 200 dupes 0
mal rows 200 unique_ids 200 dupes 0
mar rows 200 unique_ids 200 dupes 0
ory rows 200 unique_ids 200 dupes 0
pan rows 200 unique_ids 200 dupes 0
tam rows 200 unique_ids 200 dupes 0
tel rows 200 unique_ids 200 dupes 0


**Analyzing duplicate segment IDs**

I’m noticing that duplicates for asm and eng are still present! That suggests our file naming change didn’t resolve the issue. It seems the problem might come from the `segment_identity base_id`. The `relpath` hashes are colliding, which means the duplicates likely stem from similar file paths. Even with the changes we've made, duplicates remain, indicating that there’s still a collision happening. I should check a duplicate segment_id in the refs file to investigate further. It’s pretty complex!


**Investigating wav_path collisions**

I’m pondering whether we still have a collision issue with the wav_path despite changing it to include the split. If `src_id` contains slashes, it could lead to the same safe_name. Maybe the `_safe_name` method is too aggressive and results in different IDs collapsing into the same sanitized string. It seems like duplicates persist, suggesting we might be pulling the same example multiple times from our sources. We sample from test, validation, and train data, but we only need 200 entries. I need to double-check how this sampling is structured!


**Troubleshooting segment_id duplicates**

I’m noticing duplicates again, which suggests we might have repeated segment_ids among the 200 examples. This could happen if we wrote to the same wav_path due to repeated src_id values in the same split. Although it's unlikely, I’m considering if the FLEURS 'id' could repeat since it’s an integer. That could lead to duplicates. Alternatively, there might be issues with `segment_identity` even with different relpaths. If the safe_name function causes collisions, that could be a concern too. I need to clarify how we’re generating these names!


**Investigating src_id duplicates**

I’m wondering how we still have duplicates if the src_id is unique, like "1663," which should lead to unique file names. I need to check if we rebuilt v1 using the new code — yes, we did. However, the duplicate counts are still present. This might happen if the calculation of unique_ids is incorrect because it's based on the segment_id set. I should look for actual duplicate segment_id values in refs.jsonl, especially for asm, and note their audio_relpath. It's puzzling!


python3 - <<'PY'
import json
from collections import defaultdict

rows=[]
for line in open('data/bench/maya_hf_v1/refs.jsonl',encoding='utf-8'):
    rows.append(json.loads(line))

by_seg=defaultdict(list)
for r in rows:
    if r.get('lang')=='asm':
        by_seg[r['segment_id']].append(r)

dupes=[(k,v) for k,v in by_seg.items() if len(v)>1]
print('asm dupes', len(dupes))
if dupes:
    k,v=dupes[0]
    print('example segment_id', k, 'count', len(v))
    for rr in v:
        print('  audio_relpath', rr.get('audio_relpath'), 'source_split', rr.get('source_split'), 'source_id', rr.get('source_id'), 'source_index', rr.get('source_index'))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
asm dupes 33
example segment_id fleurs__as_in__test__1900-74ecb97a86c5c6aa count 2
  audio_relpath asm/fleurs__as_in__test__1900.wav source_split test source_id 1900 source_index 790
  audio_relpath asm/fleurs__as_in__test__1900.wav source_split test source_id 1900 source_index 128


Summary: auto


**Resolving duplicate identifier issues**

I discovered duplicates because the `audio_relpath` is the same for two rows, while the `source_index` differs. This reveals that the `src_id` field is not unique within the same split. We should consider using the dataset's `path` field as a unique identifier instead of just the `id` field, which can cause collisions. For the FLEURS sources, I'll set the `id_key` to 'path' to address this issue, ensuring each identifier is unique.


**Implementing unique filenames**

I’m considering including the `source_index` (i) in the filename to ensure uniqueness, since it is already included in the row. By using a format like `fname = f"{src_key}__{i}__{src_id}"`, we can avoid duplicate names effectively. This adjustment will keep the filenames unique across any dataset. I’ll modify `bench.py` to replace the existing method with this new format, ensuring it handles extensions as well. This should greatly reduce the chances of collisions.


Input:
*** Begin Patch
*** Update File: maya_transcribe/bench.py
@@
                 src_name = src.name or _safe_name(src.dataset.split("/")[-1])
                 src_key = _safe_name("__".join([src_name, src.config or "default", src.split]))
                 lang_dir = audio_root / lang
-                fname = _safe_name(f"{src_key}__{src_id or i}")
+                fname = _safe_name(f"{src_key}__{i}__{src_id or 'x'}")
                 wav_path = lang_dir / f"{fname}.wav"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/bench.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Output:
Exit code: 0
Wall time: 0 seconds
Output:


maya-transcribe bench build --out data/bench/maya_hf_v2 --samples-per-lang 200 --seed 0 --shuffle


Output:
Exit code: 0
Wall time: 21.7 seconds
Output:
{
  "wrote": "data/bench/maya_hf_v2",
  "total_rows": 2400,
  "counts": {
    "asm": 200,
    "ben": 200,
    "eng": 200,
    "guj": 200,
    "hin": 200,
    "kan": 200,
    "mal": 200,
    "mar": 200,
    "ory": 200,
    "pan": 200,
    "tam": 200,
    "tel": 200
  }
}


python3 - <<'PY'
import json
from collections import defaultdict

langs=defaultdict(set)
counts=defaultdict(int)
for line in open('data/bench/maya_hf_v2/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    lang=r.get('lang')
    seg=r.get('segment_id')
    counts[lang]+=1
    langs[lang].add(seg)

for lang in sorted(counts):
    print(lang, 'rows', counts[lang], 'unique_ids', len(langs[lang]), 'dupes', counts[lang]-len(langs[lang]))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
asm rows 200 unique_ids 200 dupes 0
ben rows 200 unique_ids 200 dupes 0
eng rows 200 unique_ids 200 dupes 0
guj rows 200 unique_ids 200 dupes 0
hin rows 200 unique_ids 200 dupes 0
kan rows 200 unique_ids 200 dupes 0
mal rows 200 unique_ids 200 dupes 0
mar rows 200 unique_ids 200 dupes 0
ory rows 200 unique_ids 200 dupes 0
pan rows 200 unique_ids 200 dupes 0
tam rows 200 unique_ids 200 dupes 0
tel rows 200 unique_ids 200 dupes 0


**Updating progress plan**

I'm excited that I fixed the duplicates in version 2! Now I can consider the step "Rebuild corpus without dupes" completed since we built v2. I'll make sure to update the plan accordingly. The next step is to create a private HF benchmark dataset, which I’ll get started on now. Additionally, it seems like I might want to use v2 for pushing as well. Let’s get everything marked and moving forward!


Output: Plan updated


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
from huggingface_hub import HfApi

token=os.getenv('HF_TOKEN') or os.getenv('HUGGINGFACE_HUB_TOKEN')
if not token:
    raise SystemExit('missing token')
api=HfApi(token=token)
info=api.whoami()
print(info.get('name'))
print('orgs', [o['name'] for o in info.get('orgs',[])])
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
bharathkumarK
orgs ['maya-research']


maya-transcribe bench push --bench-dir data/bench/maya_hf_v2 --repo-id maya-research/maya-transcribe-bench-v2 --private


Output:
Exit code: 0
Wall time: 4.3 seconds
Output:

Uploading the dataset shards:   0%|          | 0/1 [00:00<?, ?it/s]

Creating parquet from Arrow format:   0%|          | 0/3 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 3/3 [00:00<00:00, 583.35ba/s]
Uploading files as a binary IO buffer is not supported by Xet Storage. Falling back to HTTP upload.

Uploading the dataset shards: 100%|██████████| 1/1 [00:00<00:00,  1.20it/s]
Uploading the dataset shards: 100%|██████████| 1/1 [00:00<00:00,  1.20it/s]
Pushed: maya-research/maya-transcribe-bench-v2


**Confirming dataset privacy**

I pushed the dataset to `maya-research/maya-transcribe-bench-v2`, which is great! Now, I want to confirm that it’s set to private as intended. I remember setting the private option, but I should double-check to be sure. Once I’ve confirmed that, I can mark step 7 as completed in my plan. It's satisfying to tick things off the list! Let's keep moving forward!


Output: Plan updated


**Summarizing final response**

All plan steps are completed, and I need to summarize what we did, including file references and line numbers using `path:line` format. I should compute these line numbers for modifications.

The user previously asked to run benchmarks with hundreds of samples for each language to analyze issues, like the extremely high WER from the baseline on v0. We identified problems with the odia mapping, fixed duplicates in the corpus, and created a dataset manifest. However, we didn't rerun the full benchmark with the fixes, so maybe we should run it with the v2 dataset to avoid those odia errors.


**Considering benchmarking decisions**

I could rerun the benchmark using the v2 corpus with the odia mapping fix, although it would take quite a long time again. It might be necessary to provide updated results since the user is looking for minimal WER.

I could at least consider running a quick benchmark on the v2 corpus with the improved mapping to confirm there are no errors. The user wanted "a few hundreds of samples for each language," and we have 200 samples per language in v2. While a full run would take about an hour and forty-four minutes, I'm wondering if that's feasible right now.


**Evaluating benchmark requests**

The user asked if I can run the benchmarks. We did run them on v0, but since we changed the corpus and pipeline, the results have changed. The baseline still has value, though. I might consider running a benchmark on the v2 corpus with a smaller limit, maybe 2400 samples.

That could take another hour and forty-four minutes, which might not be ideal for a final answer. The user wants robust evaluation until we hit extremely low WER, but that feels a bit unrealistic. Still, we should persist until the task is fully handled as instructed. Implementing infrastructure and running a baseline seems like a good approach for such a large request.


**Considering an improvement run**

I’m thinking we might attempt a quick improvement run with a better model, like `Superleap/faster_indic_whisper_nodcil`, using a manageable subset of around 600 segments to check if WER improves. The user wants results for all languages, but this could at least help. Running the full v2 dataset with the better model could take longer, but it might be worth it.

Still, the user is insisting on continuous evaluation until WER is below 2%, which seems unrealistic. I should communicate that we can’t guarantee such low WER but can provide tools to iterate. Also, they want dataset sample sizes of 1000-2000 per language, yet we only have 200 per language.


**Organizing the benchmark update**

I should note that using `--samples-per-lang 1500` will create a bigger dataset, and we might consider scaling to 1000 samples per language, totaling 12,000 segments. However, running the pipeline could take around 9 hours, which is a long time. The user mentioned compute isn't a constraint since we have A100 available, but still, it's a huge commitment.

We need to deliver the infrastructure and initial baseline, highlighting that Whisper's large-v3 baseline WER is around 0.96, suggesting a need for better ASR models. Also, Odia isn't supported by Whisper, so we’ll need IndicWhisper. I’ll mention our fixes, including updating the language code error and adding mappings for Urdu/Sanskrit.

We can provide the necessary file references and commands for running benchmarks while being cautious about sharing sensitive details like tokens. To gather line numbers for changes, I’ll use `multi_tool_use.parallel` with `nl -ba` and `sed -n` to collect key lines efficiently.


nl -ba maya_transcribe/ctc_fit.py | sed -n '1,140p'


nl -ba maya_transcribe/ctc_fit.py | sed -n '140,320p'


nl -ba maya_transcribe/ctc_fit.py | sed -n '320,520p'


nl -ba maya_transcribe/cli.py | sed -n '1,120p'


nl -ba maya_transcribe/cli.py | sed -n '200,320p'


nl -ba maya_transcribe/cli.py | sed -n '320,520p'


nl -ba maya_transcribe/language.py | sed -n '1,120p'


nl -ba maya_transcribe/eval.py | sed -n '40,120p'


nl -ba maya_transcribe/bench.py | sed -n '1,260p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	from dataclasses import dataclass
     4	from pathlib import Path
     5	from typing import Any
     6
     7	from maya_transcribe.audio import decode_pcm16
     8	from maya_transcribe.language import ISO3_TO_WHISPER_LANG
     9	from maya_transcribe.textnorm import normalize_text
    10
    11
    12	@dataclass(frozen=True)
    13	class CTCScore:
    14	    model_id: str
    15	    loss: float
    16	    text: str
    17	    unk_ratio: float | None
    18	    target_len: int
    19	    token_score_mean: float | None
    20	    token_score_min: float | None
    21	    token_coverage: float | None
    22	    token_logp_threshold: float | None
    23	    span_count: int | None
    24	    span_mismatch: bool | None
    25
    26
    27	_WHISPER_TO_ISO3: dict[str, str] = {v: k for k, v in ISO3_TO_WHISPER_LANG.items()}
    28
    29
    30	def _to_iso3(lang: str | None) -> str | None:
    31	    if not lang:
    32	        return None
    33	    code = lang.strip().lower()
    34	    if not code:
    35	        return None
    36	    # Handle common BCP-47 forms (e.g., hi_IN / hi-IN).
    37	    code = code.replace("-", "_")
    38	    if "_" in code:
    39	        code = code.split("_", 1)[0]
    40	    if len(code) == 3:
    41	        return code
    42	    if len(code) == 2:
    43	        return _WHISPER_TO_ISO3.get(code, None)
    44	    return code
    45
    46
    47	class _CTCModel:
    48	    def __init__(self, model_id: str, device: str | None = None) -> None:
    49	        try:
    50	            import torch
    51	            from transformers import AutoModelForCTC, AutoProcessor
    52	        except Exception as e:  # pragma: no cover
    53	            raise RuntimeError(
    54	                "CTC scoring requires torch + transformers. Install with: pip install -e '.[torch]'"
    55	            ) from e
    56
    57	        if device is None:
    58	            device = "cuda" if torch.cuda.is_available() else "cpu"
    59
    60	        self.model_id = model_id
    61	        self.device = device
    62	        self._torch = torch
    63	        self._processor = AutoProcessor.from_pretrained(model_id)
    64	        self._model = AutoModelForCTC.from_pretrained(model_id).to(device)
    65	        self._model.eval()
    66
    67	        tokenizer = getattr(self._processor, "tokenizer", None)
    68	        self._force_uppercase = False
    69	        if tokenizer is not None and hasattr(tokenizer, "get_vocab"):
    70	            vocab = tokenizer.get_vocab()
    71	            # Common for wav2vec2 CTC: vocab contains A-Z but not a-z; transcripts should be uppercased.
    72	            self._force_uppercase = ("A" in vocab) and ("a" not in vocab)
    73
    74	        blank = getattr(self._model.config, "pad_token_id", None)
    75	        if blank is None and getattr(self._processor, "tokenizer", None) is not None:
    76	            blank = getattr(self._processor.tokenizer, "pad_token_id", None)
    77	        self.blank_id = int(blank) if blank is not None else 0
    78
    79	        self._target_lang: str | None = None
    80
    81	    def _maybe_set_target_lang(self, lang: str | None) -> None:
    82	        tokenizer = getattr(self._processor, "tokenizer", None)
    83	        set_lang = getattr(tokenizer, "set_target_lang", None) if tokenizer is not None else None
    84	        if not callable(set_lang):
    85	            return
    86
    87	        iso3 = _to_iso3(lang)
    88	        if not iso3:
    89	            return
    90	        if self._target_lang == iso3:
    91	            return
    92	        try:
    93	            set_lang(iso3)
    94	        except Exception as e:
    95	            raise RuntimeError(f"set_target_lang_failed:{iso3}:{e}") from e
    96	        self._target_lang = iso3
    97
    98	    def _compute_log_probs(self, audio_path: Path):
    99	        wav, sr = decode_pcm16(audio_path, sample_rate=16000, mono=True)
   100	        if wav.size == 0:
   101	            raise ValueError("empty_audio")
   102
   103	        inputs = self._processor(wav, sampling_rate=sr, return_tensors="pt")
   104
   105	        # Common key names: input_values (wav2vec2) or input_features (some processors).
   106	        if "input_values" in inputs:
   107	            model_inputs = {"input_values": inputs["input_values"].to(self.device)}
   108	        elif "input_features" in inputs:
   109	            model_inputs = {"input_features": inputs["input_features"].to(self.device)}
   110	        else:
   111	            # Best-effort: move all tensors to device.
   112	            model_inputs = {k: v.to(self.device) for k, v in inputs.items() if hasattr(v, "to")}
   113
   114	        with self._torch.no_grad():
   115	            logits = self._model(**model_inputs).logits  # (B,T,C)
   116	            log_probs = logits.log_softmax(dim=-1)  # (B,T,C)
   117
   118	        input_lengths = self._torch.tensor([log_probs.size(1)], dtype=self._torch.long, device=log_probs.device)
   119	        return log_probs, input_lengths
   120
   121	    def _score_from_log_probs(
   122	        self,
   123	        *,
   124	        log_probs,
   125	        input_lengths,
   126	        text: str,
   127	        lang: str | None,
   128	        max_unk_ratio: float,
   129	        token_logp_threshold: float,
   130	    ) -> CTCScore:
   131	        import torch.nn.functional as F
   132	        import warnings
   133
   134	        if not text:
   135	            raise ValueError("empty_text")
   136	        if "<unk>" in text or "<inaudible>" in text:
   137	            raise ValueError("contains_unk")
   138
   139	        # Conservative normalization helps tokenizer compatibility.
   140	        text_norm = normalize_text(text, lang=lang)


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   140	        text_norm = normalize_text(text, lang=lang)
   141	        if not text_norm:
   142	            raise ValueError("empty_text")
   143	        if self._force_uppercase:
   144	            text_norm = text_norm.upper()
   145
   146	        # Tokenize targets (MMS multilingual tokenizers require set_target_lang()).
   147	        tokenizer = getattr(self._processor, "tokenizer", None)
   148	        if tokenizer is None:
   149	            raise ValueError("missing_tokenizer")
   150	        self._maybe_set_target_lang(lang)
   151	        tokenized = tokenizer(text_norm, add_special_tokens=False)
   152	        target_ids = tokenized.get("input_ids")
   153	        if not target_ids:
   154	            raise ValueError("empty_targets")
   155
   156	        unk_id = getattr(tokenizer, "unk_token_id", None)
   157	        unk_ratio: float | None = None
   158	        if unk_id is not None:
   159	            unk_count = sum(1 for tid in target_ids if int(tid) == int(unk_id))
   160	            unk_ratio = float(unk_count) / float(len(target_ids))
   161	            if unk_ratio > max_unk_ratio:
   162	                raise ValueError(f"too_many_unk_tokens:{unk_ratio:.3f}")
   163
   164	        # torch ctc_loss expects (T,N,C)
   165	        log_probs_tnc = log_probs.transpose(0, 1)
   166	        device = log_probs_tnc.device
   167	        targets = self._torch.tensor(target_ids, dtype=self._torch.long, device=device)
   168	        target_lengths = self._torch.tensor([targets.numel()], dtype=self._torch.long, device=device)
   169
   170	        # Reduction=mean normalizes by target length; good for comparability.
   171	        loss = F.ctc_loss(
   172	            log_probs_tnc,
   173	            targets,
   174	            input_lengths,
   175	            target_lengths,
   176	            blank=self.blank_id,
   177	            reduction="mean",
   178	            zero_infinity=True,
   179	        )
   180	        loss_value = float(loss.detach().cpu().item())
   181
   182	        # Forced alignment token scores (coverage/fit). Best-effort: alignment may fail for short audio vs long text.
   183	        token_score_mean = None
   184	        token_score_min = None
   185	        token_coverage = None
   186	        span_count = None
   187	        span_mismatch = None
   188
   189	        try:
   190	            import torchaudio
   191
   192	            with warnings.catch_warnings():
   193	                warnings.filterwarnings("ignore", message=".*forced_align.*deprecated.*", category=UserWarning)
   194	                aligned_tokens, aligned_scores = torchaudio.functional.forced_align(
   195	                    log_probs,
   196	                    targets.unsqueeze(0),
   197	                    input_lengths=input_lengths,
   198	                    target_lengths=target_lengths,
   199	                    blank=self.blank_id,
   200	                )
   201
   202	            spans = torchaudio.functional.merge_tokens(
   203	                aligned_tokens[0].detach().cpu(), aligned_scores[0].detach().cpu(), blank=self.blank_id
   204	            )
   205	            span_scores = [float(s.score) for s in spans]
   206	            span_count = len(span_scores)
   207	            span_mismatch = (span_count != int(target_lengths.item()))
   208	            if span_scores:
   209	                token_score_mean = sum(span_scores) / float(len(span_scores))
   210	                token_score_min = min(span_scores)
   211	                token_coverage = sum(1 for s in span_scores if s >= token_logp_threshold) / float(len(span_scores))
   212	        except Exception:
   213	            pass
   214
   215	        return CTCScore(
   216	            model_id=self.model_id,
   217	            loss=loss_value,
   218	            text=text,
   219	            unk_ratio=unk_ratio,
   220	            target_len=int(target_lengths.item()),
   221	            token_score_mean=token_score_mean,
   222	            token_score_min=token_score_min,
   223	            token_coverage=token_coverage,
   224	            token_logp_threshold=token_logp_threshold,
   225	            span_count=span_count,
   226	            span_mismatch=span_mismatch,
   227	        )
   228
   229	    def score(
   230	        self,
   231	        audio_path: Path,
   232	        text: str,
   233	        lang: str | None = None,
   234	        *,
   235	        max_unk_ratio: float = 0.2,
   236	        token_logp_threshold: float = -3.0,
   237	    ) -> CTCScore:
   238	        log_probs, input_lengths = self._compute_log_probs(audio_path)
   239	        return self._score_from_log_probs(
   240	            log_probs=log_probs,
   241	            input_lengths=input_lengths,
   242	            text=text,
   243	            lang=lang,
   244	            max_unk_ratio=max_unk_ratio,
   245	            token_logp_threshold=token_logp_threshold,
   246	        )
   247
   248	    def score_candidates(
   249	        self,
   250	        *,
   251	        audio_path: Path,
   252	        candidates: dict[str, str],
   253	        lang: str | None,
   254	        max_unk_ratio: float,
   255	        token_logp_threshold: float,
   256	    ) -> tuple[list[tuple[str, CTCScore]], dict[str, str]]:
   257	        log_probs, input_lengths = self._compute_log_probs(audio_path)
   258
   259	        scored: list[tuple[str, CTCScore]] = []
   260	        errors: dict[str, str] = {}
   261
   262	        for key, text in candidates.items():
   263	            text = (text or "").strip()
   264	            if not text:
   265	                continue
   266	            try:
   267	                score = self._score_from_log_probs(
   268	                    log_probs=log_probs,
   269	                    input_lengths=input_lengths,
   270	                    text=text,
   271	                    lang=lang,
   272	                    max_unk_ratio=max_unk_ratio,
   273	                    token_logp_threshold=token_logp_threshold,
   274	                )
   275	                scored.append((key, score))
   276	            except Exception as e:
   277	                errors[key] = f"{type(e).__name__}:{e}"
   278
   279	        return scored, errors
   280
   281
   282	class CTCFitScorer:
   283	    """
   284	    Cache-loaded CTC scoring models for audio-grounded transcript fit.
   285	    """
   286
   287	    def __init__(
   288	        self,
   289	        model_map: dict[str, str] | None = None,
   290	        default_model: str | None = None,
   291	        *,
   292	        max_unk_ratio: float = 0.2,
   293	        token_logp_threshold: float = -3.0,
   294	    ) -> None:
   295	        self.model_map = model_map or {}
   296	        self.default_model = default_model
   297	        self.max_unk_ratio = float(max_unk_ratio)
   298	        self.token_logp_threshold = float(token_logp_threshold)
   299	        self._cache: dict[str, _CTCModel] = {}
   300
   301	    def _get_model_id(self, lang: str | None) -> str | None:
   302	        if lang and lang in self.model_map:
   303	            return self.model_map[lang]
   304	        return self.default_model
   305
   306	    def score(self, audio_path: Path, text: str, lang: str | None) -> CTCScore | None:
   307	        model_id = self._get_model_id(lang)
   308	        if not model_id:
   309	            return None
   310
   311	        model = self._cache.get(model_id)
   312	        if model is None:
   313	            model = _CTCModel(model_id=model_id)
   314	            self._cache[model_id] = model
   315
   316	        return model.score(
   317	            audio_path=audio_path,
   318	            text=text,
   319	            lang=lang,
   320	            max_unk_ratio=self.max_unk_ratio,


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   320	            max_unk_ratio=self.max_unk_ratio,
   321	            token_logp_threshold=self.token_logp_threshold,
   322	        )
   323
   324	    def score_candidates(
   325	        self, audio_path: Path, candidates: dict[str, str], lang: str | None
   326	    ) -> tuple[list[tuple[str, CTCScore]], dict[str, str]] | None:
   327	        model_id = self._get_model_id(lang)
   328	        if not model_id:
   329	            return None
   330
   331	        model = self._cache.get(model_id)
   332	        if model is None:
   333	            model = _CTCModel(model_id=model_id)
   334	            self._cache[model_id] = model
   335
   336	        return model.score_candidates(
   337	            audio_path=audio_path,
   338	            candidates=candidates,
   339	            lang=lang,
   340	            max_unk_ratio=self.max_unk_ratio,
   341	            token_logp_threshold=self.token_logp_threshold,
   342	        )
   343
   344
   345	def rank_candidates_by_ctc(
   346	    scorer: CTCFitScorer, audio_path: Path, candidates: dict[str, str], lang: str | None
   347	) -> dict[str, Any]:
   348	    scored: list[tuple[str, CTCScore]] = []
   349	    errors: dict[str, str] = {}
   350
   351	    batch = scorer.score_candidates(audio_path=audio_path, candidates=candidates, lang=lang)
   352	    if batch is not None:
   353	        scored, errors = batch
   354
   355	    scored.sort(key=lambda kv: kv[1].loss)
   356	    best = scored[0] if scored else None
   357
   358	    # Find the best *different-text* alternative, so duplicates don't create a fake "zero margin".
   359	    second = None
   360	    if best:
   361	        best_text = best[1].text
   362	        for item in scored[1:]:
   363	            if item[1].text != best_text:
   364	                second = item
   365	                break
   366
   367	    margin = (second[1].loss - best[1].loss) if (best and second) else None
   368
   369	    return {
   370	        "model_id": best[1].model_id if best else None,
   371	        "best_key": best[0] if best else None,
   372	        "best_loss": best[1].loss if best else None,
   373	        "second_best_key": second[0] if second else None,
   374	        "second_best_loss": second[1].loss if second else None,
   375	        "margin": float(margin) if margin is not None else None,
   376	        "scores": {
   377	            k: {
   378	                "loss": s.loss,
   379	                "model_id": s.model_id,
   380	                "unk_ratio": s.unk_ratio,
   381	                "target_len": s.target_len,
   382	                "token_score_mean": s.token_score_mean,
   383	                "token_score_min": s.token_score_min,
   384	                "token_coverage": s.token_coverage,
   385	                "token_logp_threshold": s.token_logp_threshold,
   386	                "span_count": s.span_count,
   387	                "span_mismatch": s.span_mismatch,
   388	            }
   389	            for k, s in scored
   390	        },
   391	        "errors": errors,
   392	    }


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	import json
     4	from pathlib import Path
     5
     6	import typer
     7	from dotenv import load_dotenv
     8
     9	from maya_transcribe.audio import compute_qc, ffprobe, normalize_audio
    10	from maya_transcribe.config import Settings
    11	from maya_transcribe.dataset import find_segments_dir
    12	from maya_transcribe.ids import segment_identity
    13	from maya_transcribe.io import write_jsonl
    14	from maya_transcribe.r2 import R2Client
    15	from maya_transcribe.tar import safe_extract_tar
    16
    17	load_dotenv(dotenv_path=Path(".env"), override=False)
    18
    19	app = typer.Typer(add_completion=False, no_args_is_help=True)
    20	bench_app = typer.Typer(add_completion=False, no_args_is_help=True, help="Benchmarking utilities.")
    21	app.add_typer(bench_app, name="bench")
    22
    23
    24	@app.command()
    25	def download(
    26	    video_id: str = typer.Option(..., help="YouTube video id (used to locate the tar in R2)."),
    27	    out: Path = typer.Option(Path("data"), help="Output directory."),
    28	    r2_key: str | None = typer.Option(None, help="Exact R2 object key (skips search)."),
    29	    bucket: str | None = typer.Option(None, help="R2 bucket override (defaults to env R2_BUCKET)."),
    30	    prefix: str | None = typer.Option(None, help="Optional R2 prefix override."),
    31	    use_db: bool = typer.Option(False, help="Fetch `r2_tar_key` from Supabase Postgres (DATABASE_URL)."),
    32	    dry_run: bool = typer.Option(False, help="Only show matched key(s); do not download."),
    33	) -> None:
    34	    """
    35	    Download the cleaned/segmented tar for a video from R2 and extract it.
    36	    """
    37	    settings = Settings.load()
    38	    r2 = R2Client.from_settings(settings)
    39	    if bucket:
    40	        r2 = R2Client(
    41	            endpoint_url=r2.endpoint_url,
    42	            access_key_id=r2.access_key_id,
    43	            secret_access_key=r2.secret_access_key,
    44	            bucket=bucket,
    45	        )
    46
    47	    out.mkdir(parents=True, exist_ok=True)
    48	    r2_prefix = prefix if prefix is not None else settings.r2_prefix
    49
    50	    matches: list[str]
    51	    if use_db and not r2_key:
    52	        from maya_transcribe.db import fetch_video_meta
    53
    54	        meta = fetch_video_meta(video_id)
    55	        if meta and meta.r2_tar_key:
    56	            r2_key = meta.r2_tar_key
    57
    58	    if r2_key:
    59	        matches = [r2_key]
    60	    else:
    61	        best = r2.find_video_tar_key(video_id=video_id, prefix=r2_prefix)
    62	        if not best:
    63	            typer.echo("No matching objects found in R2.")
    64	            raise typer.Exit(code=2)
    65	        matches = [best]
    66
    67	    if dry_run:
    68	        for key in matches:
    69	            typer.echo(key)
    70	        return
    71
    72	    key = matches[0]
    73	    tar_path = r2.download_to(key=key, out_dir=out / video_id)
    74	    extract_dir = out / video_id
    75	    safe_extract_tar(tar_path, extract_dir=extract_dir)
    76	    segments_dir = find_segments_dir(extract_dir)
    77	    typer.echo(f"Downloaded: {tar_path}")
    78	    typer.echo(f"Extracted to: {extract_dir}")
    79	    if segments_dir:
    80	        typer.echo(f"Segments dir: {segments_dir}")
    81
    82
    83	@app.command()
    84	def preprocess(
    85	    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
    86	    out: Path = typer.Option(Path("outputs/preprocessed"), help="Output directory for normalized audio + QC."),
    87	    sample_rate: int = typer.Option(16000, help="Target sample rate."),
    88	    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
    89	    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
    90	    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
    91	) -> None:
    92	    """
    93	    Normalize audio to 16kHz mono WAV, optionally trim silence, and write QC metrics.
    94	    """
    95	    out.mkdir(parents=True, exist_ok=True)
    96
    97	    audio_files: list[Path] = []
    98	    for ext in ("*.wav", "*.mp3", "*.m4a", "*.flac", "*.ogg", "*.opus"):
    99	        audio_files.extend(sorted(segments_dir.rglob(ext)))
   100	    audio_files = sorted(set(audio_files))
   101
   102	    if not audio_files:
   103	        typer.echo(f"No audio files found under: {segments_dir}")
   104	        raise typer.Exit(code=2)
   105
   106	    rows = []
   107	    norm_dir = out / "audio"
   108	    for src in audio_files:
   109	        ident = segment_identity(src, root=segments_dir)
   110	        segment_id = ident.segment_id
   111	        dst = norm_dir / f"{segment_id}.wav"
   112	        normalize_audio(
   113	            src,
   114	            dst,
   115	            sample_rate=sample_rate,
   116	            mono=True,
   117	            trim_silence=trim_silence,
   118	            silence_threshold_db=silence_threshold_db,
   119	            min_silence_sec=min_silence_sec,
   120	        )


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   200
   201	    prompt = load_prompt_template(prompt_file)
   202	    backend = GeminiASR(config=GeminiConfig(model=model, max_output_tokens=max_output_tokens), prompt_template=prompt)
   203	    res = backend.transcribe(audio_path=audio, language=language)
   204	    typer.echo(res.text)
   205
   206
   207	@app.command()
   208	def run(
   209	    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
   210	    out: Path = typer.Option(Path("outputs/run"), help="Output directory."),
   211	    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
   212	    lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
   213	    require_lid_backend: bool = typer.Option(True, help="Fail if MMS-LID backend can't initialize."),
   214	    transformers_whisper_model: list[str] = typer.Option(
   215	        [], help="Extra whisper-family HF models to run via transformers (repeatable)."
   216	    ),
   217	    gemini_model: str | None = typer.Option(None, help="Optional Gemini model name (adds Gemini as a hypothesis)."),
   218	    gemini_prompt_file: Path | None = typer.Option(Path("Prompt.md"), help="Prompt file for Gemini."),
   219	    gemini_hard_only: bool = typer.Option(True, help="Only call Gemini on 'hard' segments."),
   220	    ctc_model: str | None = typer.Option(None, help="Optional HF CTC model id for audio-grounded fit scoring."),
   221	    ctc_model_map: Path | None = typer.Option(None, help="JSON map {lang_code: hf_model_id} for CTC scoring."),
   222	    require_ctc: bool = typer.Option(True, help="When CTC is enabled, require a CTC score to accept."),
   223	    min_ctc_margin: float = typer.Option(0.05, help="Min (second-best - best) CTC loss margin to auto-accept."),
   224	    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
   225	    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
   226	    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
   227	    limit: int | None = typer.Option(None, help="Process only the first N segments."),
   228	    thresholds_json: Path | None = typer.Option(None, help="Optional JSON file of threshold overrides (verify + routing)."),
   229	) -> None:
   230	    """
   231	    End-to-end pipeline: preprocess -> LID -> ASR ensemble -> ROVER -> verify.
   232	    """
   233	    from maya_transcribe.pipeline import run_pipeline
   234
   235	    ctc_map = None
   236	    if ctc_model_map is not None:
   237	        ctc_map = json.loads(ctc_model_map.read_text(encoding="utf-8"))
   238	        if not isinstance(ctc_map, dict):
   239	            raise typer.BadParameter("ctc_model_map must be a JSON object {lang_code: model_id}")
   240
   241	    thresholds: dict[str, object] = {}
   242	    if thresholds_json is not None:
   243	        loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
   244	        if not isinstance(loaded, dict):
   245	            raise typer.BadParameter("thresholds_json must be a JSON object")
   246	        thresholds = loaded
   247	    if ctc_model or ctc_map:
   248	        thresholds["require_ctc"] = require_ctc
   249	        thresholds["min_ctc_margin"] = float(min_ctc_margin)
   250
   251	    results_path = run_pipeline(
   252	        segments_dir=segments_dir,
   253	        out_dir=out,
   254	        lid_model_id=lid_model_id,
   255	        require_lid_backend=require_lid_backend,
   256	        whisper_model=whisper_model,
   257	        extra_transformers_whisper_models=transformers_whisper_model,
   258	        gemini_model=gemini_model,
   259	        gemini_prompt_path=gemini_prompt_file,
   260	        gemini_hard_only=gemini_hard_only,
   261	        ctc_model=ctc_model,
   262	        ctc_model_map=ctc_map,
   263	        trim_silence=trim_silence,
   264	        silence_threshold_db=silence_threshold_db,
   265	        min_silence_sec=min_silence_sec,
   266	        limit=limit,
   267	        thresholds=thresholds,
   268	    )
   269	    typer.echo(f"Wrote: {results_path}")
   270	    typer.echo(f"Wrote: {out / 'summary.json'}")
   271
   272
   273	@app.command()
   274	def eval(
   275	    results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
   276	    refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
   277	    out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
   278	    hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
   279	    oracle: bool = typer.Option(True, help="Compute oracle-best and selection-gap stats across all candidates."),
   280	    language_source: str = typer.Option("route", help="Use language from: 'route' (LID) or 'refs' (refs.jsonl)."),
   281	) -> None:
   282	    """
   283	    Compute WER/CER for a hypothesis against a reference JSONL keyed by `segment_id`.
   284	    """
   285	    from maya_transcribe.eval import evaluate
   286
   287	    summary = evaluate(
   288	        results_path=results,
   289	        refs_path=refs,
   290	        out_path=out,
   291	        hypothesis=hypothesis,
   292	        oracle=oracle,
   293	        language_source=language_source,
   294	    )
   295	    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))
   296
   297
   298	@bench_app.command("build")
   299	def bench_build(
   300	    out: Path = typer.Option(Path("data/bench/maya_hf_v0"), help="Output benchmark corpus directory."),
   301	    samples_per_lang: int = typer.Option(200, help="Target samples per language."),
   302	    seed: int = typer.Option(0, help="Shuffle seed."),
   303	    shuffle: bool = typer.Option(True, help="Shuffle within each source before selecting."),
   304	    lang: list[str] = typer.Option([], help="Limit to ISO-639-3 language codes (repeatable)."),
   305	    min_duration_sec: float = typer.Option(1.0, help="Min audio duration."),
   306	    max_duration_sec: float = typer.Option(30.0, help="Max audio duration."),
   307	    force: bool = typer.Option(False, help="Delete and rebuild output directory if it exists."),
   308	) -> None:
   309	    """
   310	    Build a local benchmark corpus from Hugging Face datasets.
   311	    """
   312	    from maya_transcribe.bench import build_benchmark_corpus
   313
   314	    manifest = build_benchmark_corpus(
   315	        out_dir=out,
   316	        samples_per_lang=samples_per_lang,
   317	        seed=seed,
   318	        shuffle=shuffle,
   319	        langs=lang or None,
   320	        min_duration_sec=min_duration_sec,


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   320	        min_duration_sec=min_duration_sec,
   321	        max_duration_sec=max_duration_sec,
   322	        force=force,
   323	    )
   324	    typer.echo(json.dumps({"wrote": str(out), "total_rows": manifest.get("total_rows"), "counts": manifest.get("counts")}, indent=2))
   325
   326
   327	@bench_app.command("run")
   328	def bench_run(
   329	    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
   330	    out: Path | None = typer.Option(None, help="Output directory (defaults under outputs/bench/)."),
   331	    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
   332	    lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
   333	    require_lid_backend: bool = typer.Option(True, help="Fail if MMS-LID backend can't initialize."),
   334	    transformers_whisper_model: list[str] = typer.Option(
   335	        [], help="Extra whisper-family HF models to run via transformers (repeatable)."
   336	    ),
   337	    ctc_model: str | None = typer.Option(None, help="Optional HF CTC model id for audio-grounded fit scoring."),
   338	    ctc_model_map: Path | None = typer.Option(None, help="JSON map {lang_code: hf_model_id} for CTC scoring."),
   339	    require_ctc: bool = typer.Option(True, help="When CTC is enabled, require a CTC score to accept."),
   340	    min_ctc_margin: float = typer.Option(0.05, help="Min (second-best - best) CTC loss margin to auto-accept."),
   341	    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
   342	    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
   343	    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
   344	    limit: int | None = typer.Option(None, help="Process only the first N segments."),
   345	    thresholds_json: Path | None = typer.Option(None, help="Optional JSON file of threshold overrides (verify + routing)."),
   346	    hypothesis: str = typer.Option("final", help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
   347	) -> None:
   348	    """
   349	    Run the full pipeline on a local benchmark corpus and write eval.json.
   350	    """
   351	    from datetime import datetime
   352
   353	    from maya_transcribe.eval import evaluate
   354	    from maya_transcribe.pipeline import run_pipeline
   355
   356	    audio_dir = bench_dir / "audio"
   357	    refs_path = bench_dir / "refs.jsonl"
   358	    if not audio_dir.exists():
   359	        raise typer.BadParameter(f"Missing: {audio_dir}")
   360	    if not refs_path.exists():
   361	        raise typer.BadParameter(f"Missing: {refs_path}")
   362
   363	    if out is None:
   364	        ts = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
   365	        out = Path("outputs/bench") / bench_dir.name / f"run_{ts}"
   366
   367	    ctc_map = None
   368	    if ctc_model_map is not None:
   369	        ctc_map = json.loads(ctc_model_map.read_text(encoding="utf-8"))
   370	        if not isinstance(ctc_map, dict):
   371	            raise typer.BadParameter("ctc_model_map must be a JSON object {lang_code: model_id}")
   372
   373	    thresholds: dict[str, object] = {}
   374	    if thresholds_json is not None:
   375	        loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
   376	        if not isinstance(loaded, dict):
   377	            raise typer.BadParameter("thresholds_json must be a JSON object")
   378	        thresholds = loaded
   379	    if ctc_model or ctc_map:
   380	        thresholds["require_ctc"] = require_ctc
   381	        thresholds["min_ctc_margin"] = float(min_ctc_margin)
   382
   383	    results_path = run_pipeline(
   384	        segments_dir=audio_dir,
   385	        out_dir=out,
   386	        lid_model_id=lid_model_id,
   387	        require_lid_backend=require_lid_backend,
   388	        whisper_model=whisper_model,
   389	        extra_transformers_whisper_models=transformers_whisper_model,
   390	        ctc_model=ctc_model,
   391	        ctc_model_map=ctc_map,
   392	        trim_silence=trim_silence,
   393	        silence_threshold_db=silence_threshold_db,
   394	        min_silence_sec=min_silence_sec,
   395	        limit=limit,
   396	        thresholds=thresholds,
   397	    )
   398
   399	    eval_path = out / "eval.json"
   400	    summary = evaluate(
   401	        results_path=results_path,
   402	        refs_path=refs_path,
   403	        out_path=eval_path,
   404	        hypothesis=hypothesis,
   405	        oracle=True,
   406	        language_source="refs",
   407	    )
   408	    typer.echo(f"Wrote: {results_path}")
   409	    typer.echo(f"Wrote: {out / 'summary.json'}")
   410	    typer.echo(f"Wrote: {eval_path}")
   411	    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))
   412
   413
   414	@bench_app.command("push")
   415	def bench_push(
   416	    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
   417	    repo_id: str = typer.Option(..., help="Hugging Face dataset repo id (e.g., org/name)."),
   418	    private: bool = typer.Option(True, help="Create/push as a private dataset repo."),
   419	    include_audio: bool = typer.Option(False, help="Include audio bytes in the pushed dataset (check dataset licenses!)."),
   420	    split: str = typer.Option("test", help="Split name to use on the Hub."),
   421	) -> None:
   422	    """
   423	    Push a local benchmark corpus to the Hugging Face Hub as a (private) dataset.
   424	    """
   425	    from maya_transcribe.bench import push_benchmark_to_hub
   426
   427	    repo = push_benchmark_to_hub(bench_dir=bench_dir, repo_id=repo_id, private=private, include_audio=include_audio, split=split)
   428	    typer.echo(f"Pushed: {repo}")
   429
   430
   431	if __name__ == "__main__":
   432	    app()


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	ISO3_TO_WHISPER_LANG: dict[str, str] = {
     4	    "eng": "en",
     5	    "hin": "hi",
     6	    "mar": "mr",
     7	    "guj": "gu",
     8	    "pan": "pa",
     9	    "tam": "ta",
    10	    "tel": "te",
    11	    "kan": "kn",
    12	    "mal": "ml",
    13	    "ben": "bn",
    14	    "asm": "as",
    15	    "urd": "ur",
    16	    "san": "sa",
    17	}
    18
    19	# Human-friendly names and scripts (for prompts/UI).
    20	LANG_CODE_TO_NAME: dict[str, str] = {
    21	    "eng": "English",
    22	    "en": "English",
    23	    "hin": "Hindi",
    24	    "hi": "Hindi",
    25	    "mar": "Marathi",
    26	    "mr": "Marathi",
    27	    "guj": "Gujarati",
    28	    "gu": "Gujarati",
    29	    "pan": "Punjabi",
    30	    "pa": "Punjabi",
    31	    "tam": "Tamil",
    32	    "ta": "Tamil",
    33	    "tel": "Telugu",
    34	    "te": "Telugu",
    35	    "kan": "Kannada",
    36	    "kn": "Kannada",
    37	    "mal": "Malayalam",
    38	    "ml": "Malayalam",
    39	    "ben": "Bengali",
    40	    "bn": "Bengali",
    41	    "asm": "Assamese",
    42	    "as": "Assamese",
    43	    "ori": "Odia",
    44	    "ory": "Odia",
    45	    "or": "Odia",
    46	}
    47
    48	LANG_NAME_TO_SCRIPT: dict[str, str] = {
    49	    "English": "Latin",
    50	    "Hindi": "Devanagari",
    51	    "Marathi": "Devanagari",
    52	    "Gujarati": "Gujarati",
    53	    "Punjabi": "Gurmukhi",
    54	    "Tamil": "Tamil",
    55	    "Telugu": "Telugu",
    56	    "Kannada": "Kannada",
    57	    "Malayalam": "Malayalam",
    58	    "Bengali": "Bengali",
    59	    "Assamese": "Assamese",
    60	    "Odia": "Odia",
    61	}
    62
    63
    64	def to_whisper_lang(lang_code: str | None) -> str | None:
    65	    if not lang_code:
    66	        return None
    67	    lang_code = lang_code.strip().lower()
    68	    # Whisper/faster-whisper does not expose an Odia language token; let it auto-detect instead of erroring.
    69	    if lang_code in {"ori", "ory", "or"}:
    70	        return None
    71	    if len(lang_code) == 2:
    72	        return lang_code
    73	    return ISO3_TO_WHISPER_LANG.get(lang_code, None)
    74
    75
    76	def to_language_name(lang_code: str | None) -> str | None:
    77	    if not lang_code:
    78	        return None
    79	    lang_code = lang_code.strip().lower()
    80	    return LANG_CODE_TO_NAME.get(lang_code)
    81
    82
    83	def to_prompt_language_hint(lang_code: str | None) -> str:
    84	    """
    85	    Return a human-friendly language hint string for prompting audio-native models.
    86	    """
    87	    if not lang_code:
    88	        return "Unknown (auto-detect primary spoken language)"
    89	    lang_code_norm = lang_code.strip().lower()
    90	    name = LANG_CODE_TO_NAME.get(lang_code_norm)
    91	    if not name:
    92	        return f"{lang_code_norm} (auto-detect primary spoken language)"
    93	    script = LANG_NAME_TO_SCRIPT.get(name)
    94	    if script:
    95	        return f"{name} ({script} script)"
    96	    return name


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    40	        if not segment_id:
    41	            continue
    42	        refs[segment_id] = row
    43	    return refs
    44
    45
    46	def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    47	    for k in keys:
    48	        v = row.get(k)
    49	        if isinstance(v, str) and v.strip():
    50	            return v
    51	    return None
    52
    53
    54	def evaluate(
    55	    results_path: Path,
    56	    refs_path: Path,
    57	    out_path: Path | None = None,
    58	    hypothesis: str = "final",
    59	    oracle: bool = True,
    60	    language_source: str = "route",
    61	) -> dict[str, Any]:
    62	    refs = load_refs(refs_path)
    63	    results = read_jsonl(results_path)
    64
    65	    rows: list[EvalRow] = []
    66	    for r in results:
    67	        segment_id = str(r.get("segment_id") or "").strip()
    68	        if not segment_id or segment_id not in refs:
    69	            continue
    70
    71	        ref_row = refs[segment_id]
    72	        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
    73	        if ref_text is None:
    74	            continue
    75
    76	        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
    77	        if hyp_text is None:
    78	            continue
    79
    80	        route_lang = None
    81	        route = r.get("route") or {}
    82	        if isinstance(route, dict):
    83	            route_lang = route.get("language_hint")
    84
    85	        ref_lang = None
    86	        if language_source == "refs":
    87	            v = ref_row.get("lang") or ref_row.get("language")
    88	            if isinstance(v, str) and v.strip():
    89	                ref_lang = v.strip()
    90	        elif language_source != "route":
    91	            raise ValueError("language_source must be 'route' or 'refs'")
    92
    93	        lang = ref_lang or route_lang
    94
    95	        decision = None
    96	        ver = r.get("verification") or {}
    97	        if isinstance(ver, dict):
    98	            decision = ver.get("decision")
    99
   100	        duration_sec = None
   101	        qc = r.get("qc") or {}
   102	        if isinstance(qc, dict):
   103	            try:
   104	                duration_sec = float(qc.get("duration_sec")) if qc.get("duration_sec") is not None else None
   105	            except (TypeError, ValueError):
   106	                duration_sec = None
   107	        bucket = duration_bucket(duration_sec)
   108
   109	        ref_norm = normalize_text(ref_text, lang=lang)
   110	        hyp_norm = normalize_text(hyp_text, lang=lang)
   111	        w = float(wer(ref_norm, hyp_norm))
   112	        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))
   113
   114	        oracle_best_wer = None
   115	        oracle_best_cer = None
   116	        oracle_best_key = None
   117	        selection_gap_wer = None
   118
   119	        if oracle:
   120	            cand_wer: dict[str, float] = {}


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	[build-system]
     2	requires = ["setuptools>=69", "wheel"]
     3	build-backend = "setuptools.build_meta"
     4
     5	[project]
     6	name = "maya-transcribe"
     7	version = "0.1.0"
     8	description = "Quality-first offline transcription pipeline (Indic + English)"
     9	readme = "README.md"
    10	requires-python = ">=3.10"
    11	dependencies = [
    12	  "boto3>=1.34.0",
    13	  "jiwer>=3.0.4",
    14	  "numpy>=1.24.0",
    15	  "python-dotenv>=1.0.1",
    16	  "rich>=13.7.1",
    17	  "typer>=0.12.3",
    18	]
    19
    20	[project.optional-dependencies]
    21	supabase = ["supabase>=2.6.0"]
    22	db = ["psycopg2-binary>=2.9.9"]
    23	gemini = ["google-genai>=1.54.0"]
    24	bench = [
    25	  "datasets>=2.19.0",
    26	  "huggingface_hub>=0.23.0",
    27	]
    28	asr = [
    29	  "faster-whisper>=1.0.3",
    30	  "soundfile>=0.12.1",
    31	]
    32	torch = [
    33	  "torch>=2.2.0",
    34	  "torchaudio>=2.2.0",
    35	  "transformers>=4.41.0",
    36	]
    37	full = [
    38	  "datasets>=2.19.0",
    39	  "huggingface_hub>=0.23.0",
    40	  "supabase>=2.6.0",
    41	  "psycopg2-binary>=2.9.9",
    42	  "google-genai>=1.54.0",
    43	  "faster-whisper>=1.0.3",
    44	  "soundfile>=0.12.1",
    45	  "torch>=2.2.0",
    46	  "torchaudio>=2.2.0",
    47	  "transformers>=4.41.0",
    48	]
    49
    50	[project.scripts]
    51	maya-transcribe = "maya_transcribe.cli:app"
    52
    53	[tool.setuptools.packages.find]
    54	where = ["."]
    55	include = ["maya_transcribe*"]


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 260
Output:
     1	from __future__ import annotations
     2
     3	import json
     4	import os
     5	import random
     6	import re
     7	import wave
     8	from dataclasses import dataclass
     9	from pathlib import Path
    10	from typing import Any
    11
    12	import numpy as np
    13
    14	from maya_transcribe.ids import segment_identity
    15	from maya_transcribe.io import read_jsonl, write_jsonl
    16
    17
    18	@dataclass(frozen=True)
    19	class HFSource:
    20	    dataset: str
    21	    config: str | None
    22	    split: str
    23	    audio_key: str
    24	    text_key: str
    25	    id_key: str | None = None
    26	    trust_remote_code: bool = False
    27	    name: str | None = None
    28
    29
    30	_SAFE_NAME_RE = re.compile(r"[^a-zA-Z0-9._-]+")
    31
    32
    33	def _safe_name(value: str) -> str:
    34	    value = (value or "").strip()
    35	    value = value.replace("/", "_")
    36	    value = value.replace("\\", "_")
    37	    value = _SAFE_NAME_RE.sub("_", value)
    38	    value = value.strip("._-")
    39	    return value or "x"
    40
    41
    42	def _ensure_hf_token_env() -> None:
    43	    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
    44	    if token and not os.getenv("HUGGINGFACE_HUB_TOKEN"):
    45	        os.environ["HUGGINGFACE_HUB_TOKEN"] = token
    46
    47
    48	def _write_wav_mono16(path: Path, audio: np.ndarray, sampling_rate: int) -> None:
    49	    path.parent.mkdir(parents=True, exist_ok=True)
    50
    51	    wav = np.asarray(audio, dtype=np.float32)
    52	    if wav.ndim == 2:
    53	        wav = wav.mean(axis=1)
    54	    wav = np.clip(wav, -1.0, 1.0)
    55	    pcm16 = (wav * 32767.0).astype(np.int16)
    56
    57	    with wave.open(str(path), "wb") as wf:
    58	        wf.setnchannels(1)
    59	        wf.setsampwidth(2)
    60	        wf.setframerate(int(sampling_rate))
    61	        wf.writeframes(pcm16.tobytes())
    62
    63
    64	def _extract_audio(example: dict[str, Any], audio_key: str) -> tuple[np.ndarray, int] | None:
    65	    audio = example.get(audio_key)
    66	    if not isinstance(audio, dict):
    67	        return None
    68	    arr = audio.get("array")
    69	    sr = audio.get("sampling_rate")
    70	    if arr is None or sr is None:
    71	        return None
    72	    try:
    73	        sr_i = int(sr)
    74	    except (TypeError, ValueError):
    75	        return None
    76	    return np.asarray(arr, dtype=np.float32), sr_i
    77
    78
    79	def _extract_text(example: dict[str, Any], text_key: str) -> str | None:
    80	    text = example.get(text_key)
    81	    if not isinstance(text, str):
    82	        return None
    83	    text = text.strip()
    84	    return text or None
    85
    86
    87	def default_sources() -> dict[str, list[HFSource]]:
    88	    """
    89	    Default benchmark sources for the 12 target languages:
    90	    Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English.
    91	    """
    92	    kathbath = "ai4bharat/Kathbath"
    93	    fleurs = "google/fleurs"
    94
    95	    # Canonical ISO-639-3 codes aligned to MMS-LID output (and language.py mappings).
    96	    return {
    97	        "hin": [HFSource(dataset=kathbath, config="hindi", split="valid", audio_key="audio_filepath", text_key="text")],
    98	        "tam": [HFSource(dataset=kathbath, config="tamil", split="valid", audio_key="audio_filepath", text_key="text")],
    99	        "tel": [HFSource(dataset=kathbath, config="telugu", split="valid", audio_key="audio_filepath", text_key="text")],
   100	        "kan": [HFSource(dataset=kathbath, config="kannada", split="valid", audio_key="audio_filepath", text_key="text")],
   101	        "mal": [
   102	            HFSource(dataset=kathbath, config="malayalam", split="valid", audio_key="audio_filepath", text_key="text")
   103	        ],
   104	        "ben": [HFSource(dataset=kathbath, config="bengali", split="valid", audio_key="audio_filepath", text_key="text")],
   105	        "guj": [
   106	            HFSource(dataset=kathbath, config="gujarati", split="valid", audio_key="audio_filepath", text_key="text")
   107	        ],
   108	        "pan": [
   109	            HFSource(dataset=kathbath, config="punjabi", split="valid", audio_key="audio_filepath", text_key="text")
   110	        ],
   111	        "mar": [
   112	            HFSource(dataset=kathbath, config="marathi", split="valid", audio_key="audio_filepath", text_key="text")
   113	        ],
   114	        "ory": [HFSource(dataset=kathbath, config="odia", split="valid", audio_key="audio_filepath", text_key="text")],
   115	        # Kathbath doesn't include Assamese/English; use FLEURS (requires trust_remote_code=True).
   116	        "asm": [
   117	            HFSource(
   118	                dataset=fleurs,
   119	                config="as_in",
   120	                split="test",
   121	                …443 chars truncated… ),
   133	            HFSource(
   134	                dataset=fleurs,
   135	                config="as_in",
   136	                split="train",
   137	                audio_key="audio",
   138	                text_key="transcription",
   139	                trust_remote_code=True,
   140	            ),
   141	        ],
   142	        "eng": [
   143	            HFSource(
   144	                dataset=fleurs,
   145	                config="en_us",
   146	                split="test",
   147	                audio_key="audio",
   148	                text_key="transcription",
   149	                trust_remote_code=True,
   150	            ),
   151	            HFSource(
   152	                dataset=fleurs,
   153	                config="en_us",
   154	                split="validation",
   155	                audio_key="audio",
   156	                text_key="transcription",
   157	                trust_remote_code=True,
   158	            ),
   159	            HFSource(
   160	                dataset=fleurs,
   161	                config="en_us",
   162	                split="train",
   163	                audio_key="audio",
   164	                text_key="transcription",
   165	                trust_remote_code=True,
   166	            ),
   167	        ],
   168	    }
   169
   170
   171	def build_benchmark_corpus(
   172	    *,
   173	    out_dir: Path,
   174	    samples_per_lang: int = 200,
   175	    seed: int = 0,
   176	    shuffle: bool = True,
   177	    langs: list[str] | None = None,
   178	    min_duration_sec: float = 1.0,
   179	    max_duration_sec: float = 30.0,
   180	    force: bool = False,
   181	) -> dict[str, Any]:
   182	    """
   183	    Materialize a local benchmark corpus:
   184	      - `out_dir/audio/<lang>/*.wav`
   185	      - `out_dir/refs.jsonl` with `segment_id`, `text`, `lang`, `audio_relpath`, `source_*`
   186	      - `out_dir/manifest.json` with build config + counts
   187	    """
   188	    _ensure_hf_token_env()
   189
   190	    if out_dir.exists():
   191	        existing = [p for p in out_dir.iterdir()]
   192	        if existing and not force:
   193	            raise RuntimeError(f"Refusing to overwrite non-empty directory: {out_dir} (pass force=True)")
   194	        if force:
   195	            for p in sorted(out_dir.rglob("*"), reverse=True):
   196	                if p.is_file() or p.is_symlink():
   197	                    p.unlink()
   198	                elif p.is_dir():
   199	                    p.rmdir()
   200
   201	    out_dir.mkdir(parents=True, exist_ok=True)
   202	    audio_root = out_dir / "audio"
   203
   204	    sources_by_lang = default_sources()
   205	    target_langs = langs or sorted(sources_by_lang.keys())
   206	    target_langs = [l.strip().lower() for l in target_langs if l.strip()]
   207
   208	    try:
   209	        from datasets import load_dataset
   210	    except Exception as e:  # pragma: no cover
   211	        raise RuntimeError("Benchmark corpus build requires `datasets`. Install with: pip install -e '.[bench]'") from e
   212
   213	    rows: list[dict[str, Any]] = []
   214	    counts: dict[str, int] = {}
   215	    skipped: dict[str, int] = {}
   216
   217	    for lang in target_langs:
   218	        sources = sources_by_lang.get(lang)
   219	        if not sources:
   220	            continue
   221	        remaining = int(samples_per_lang)
   222	        written = 0
   223	        skipped_lang = 0
   224
   225	        for src in sources:
   226	            if remaining <= 0:
   227	                break
   228
   229	            ds = load_dataset(
   230	                src.dataset,
   231	                src.config,
   232	                split=src.split,
   233	                trust_remote_code=bool(src.trust_remote_code),
   234	            )
   235	            idxs = list(range(len(ds)))
   236	            if shuffle:
   237	                # Stable per-source shuffling (seeded by global seed + source identity).
   238	                s = f"{seed}|{lang}|{src.dataset}|{src.config}|{src.split}"
   239	                srng = random.Random(int.from_bytes(s.encode("utf-8"), "little") % (2**32))
   240	                srng.shuffle(idxs)
   241
   242	            for i in idxs:
   243	                if remaining <= 0:
   244	                    break
   245
   246	                ex = ds[int(i)]
   247	                if not isinstance(ex, dict):
   248	                    continue
   249
   250	                text = _extract_text(ex, src.text_key)
   251	                if not text:
   252	                    skipped_lang += 1
   253	                    continue
   254
   255	                audio_ex = _extract_audio(ex, src.audio_key)
   256	                if audio_ex is None:
   257	                    skipped_lang += 1
   258	                    continue
   259	                audio_arr, sr = audio_ex
   260	                dur = float(audio_arr.shape[0]) / float(sr) if sr else 0.0


Summary: auto


**Fixing filename output issue**

I see that the output from bench.py got truncated around line 116 due to a tool output limit. Now, since the file name has changed, I'll need to provide specific line numbers for the section related to filename uniqueness. I think it’s around line 260, but I need to confirm where `fname = ...` is located. To find that, I’ll rerun a command to capture those lines. Let's proceed with that!


nl -ba maya_transcribe/bench.py | sed -n '260,420p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   260	                dur = float(audio_arr.shape[0]) / float(sr) if sr else 0.0
   261	                if dur < float(min_duration_sec) or dur > float(max_duration_sec):
   262	                    skipped_lang += 1
   263	                    continue
   264
   265	                src_id = None
   266	                if src.id_key and isinstance(ex.get(src.id_key), (str, int)):
   267	                    src_id = str(ex.get(src.id_key))
   268	                elif isinstance(ex.get("id"), (str, int)):
   269	                    src_id = str(ex.get("id"))
   270	                elif isinstance(ex.get("fname"), str):
   271	                    src_id = ex.get("fname")
   272	                elif isinstance(ex.get("path"), str):
   273	                    src_id = ex.get("path")
   274
   275	                src_name = src.name or _safe_name(src.dataset.split("/")[-1])
   276	                src_key = _safe_name("__".join([src_name, src.config or "default", src.split]))
   277	                lang_dir = audio_root / lang
   278	                fname = _safe_name(f"{src_key}__{i}__{src_id or 'x'}")
   279	                wav_path = lang_dir / f"{fname}.wav"
   280
   281	                _write_wav_mono16(wav_path, audio_arr, sr)
   282	                ident = segment_identity(wav_path, root=audio_root)
   283
   284	                audio_relpath = wav_path.relative_to(audio_root).as_posix()
   285	                rows.append(
   286	                    {
   287	                        "segment_id": ident.segment_id,
   288	                        "text": text,
   289	                        "lang": lang,
   290	                        "audio_relpath": audio_relpath,
   291	                        "duration_sec": dur,
   292	                        "source_dataset": src.dataset,
   293	                        "source_config": src.config,
   294	                        "source_split": src.split,
   295	                        "source_id": src_id,
   296	                        "source_index": int(i),
   297	                    }
   298	                )
   299
   300	                remaining -= 1
   301	                written += 1
   302
   303	        counts[lang] = written
   304	        skipped[lang] = skipped_lang
   305
   306	    refs_path = out_dir / "refs.jsonl"
   307	    write_jsonl(refs_path, rows)
   308
   309	    manifest = {
   310	        "out_dir": str(out_dir),
   311	        "audio_root": str(audio_root),
   312	        "samples_per_lang": int(samples_per_lang),
   313	        "seed": int(seed),
   314	        "shuffle": bool(shuffle),
   315	        "langs": target_langs,
   316	        "min_duration_sec": float(min_duration_sec),
   317	        "max_duration_sec": float(max_duration_sec),
   318	        "counts": counts,
   319	        "skipped": skipped,
   320	        "total_rows": len(rows),
   321	        "sources_by_lang": {
   322	            k: [vars(s) for s in v] for k, v in sources_by_lang.items() if (not langs or k in set(target_langs))
   323	        },
   324	    }
   325	    (out_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
   326	    return manifest
   327
   328
   329	def push_benchmark_to_hub(
   330	    *,
   331	    bench_dir: Path,
   332	    repo_id: str,
   333	    private: bool = True,
   334	    include_audio: bool = False,
   335	    split: str = "test",
   336	) -> str:
   337	    """
   338	    Push the benchmark corpus as a Hugging Face dataset repo.
   339
   340	    If `include_audio` is False, uploads only the JSONL manifest rows (no audio redistribution).
   341	    """
   342	    _ensure_hf_token_env()
   343	    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
   344	    if not token:
   345	        raise RuntimeError("Missing HF auth token (set HF_TOKEN or HUGGINGFACE_HUB_TOKEN).")
   346
   347	    try:
   348	        from huggingface_hub import HfApi
   349	    except Exception as e:  # pragma: no cover
   350	        raise RuntimeError("Push-to-hub requires `huggingface_hub`. Install with: pip install -e '.[bench]'") from e
   351
   352	    try:
   353	        from datasets import Audio, Dataset, DatasetDict
   354	    except Exception as e:  # pragma: no cover
   355	        raise RuntimeError("Push-to-hub requires `datasets`. Install with: pip install -e '.[bench]'") from e
   356
   357	    refs_path = bench_dir / "refs.jsonl"
   358	    if not refs_path.exists():
   359	        raise RuntimeError(f"Missing refs.jsonl: {refs_path}")
   360	    rows = read_jsonl(refs_path)
   361
   362	    audio_root = bench_dir / "audio"
   363	    if include_audio and not audio_root.exists():
   364	        raise RuntimeError(f"Missing audio/ directory: {audio_root}")
   365
   366	    records: dict[str, list[Any]] = {
   367	        "segment_id": [],
   368	        "lang": [],
   369	        "text": [],
   370	        "audio_relpath": [],
   371	        "duration_sec": [],
   372	        "source_dataset": [],
   373	        "source_config": [],
   374	        "source_split": [],
   375	        "source_id": [],
   376	        "source_index": [],
   377	    }
   378	    audio_paths: list[str] = []
   379
   380	    for r in rows:
   381	        records["segment_id"].append(r.get("segment_id"))
   382	        records["lang"].append(r.get("lang"))
   383	        records["text"].append(r.get("text"))
   384	        records["audio_relpath"].append(r.get("audio_relpath"))
   385	        records["duration_sec"].append(r.get("duration_sec"))
   386	        records["source_dataset"].append(r.get("source_dataset"))
   387	        records["source_config"].append(r.get("source_config"))
   388	        records["source_split"].append(r.get("source_split"))
   389	        records["source_id"].append(r.get("source_id"))
   390	        records["source_index"].append(r.get("source_index"))
   391
   392	        if include_audio:
   393	            rel = str(r.get("audio_relpath") or "")
   394	            audio_paths.append(str(audio_root / rel))
   395
   396	    if include_audio:
   397	        records["audio"] = audio_paths
   398
   399	    ds = Dataset.from_dict(records)
   400	    if include_audio:
   401	        ds = ds.cast_column("audio", Audio())
   402
   403	    api = HfApi(token=token)
   404	    api.create_repo(repo_id=repo_id, repo_type="dataset", private=bool(private), exist_ok=True)
   405
   406	    dsd = DatasetDict({split: ds})
   407	    dsd.push_to_hub(repo_id=repo_id, token=token, private=bool(private))
   408	    return repo_id


**Finalizing evaluation notes**

Okay, I need to mention the baseline results and provide the file path to the evaluation summary in outputs/bench. The user might want that. While I updated `maya_transcribe/bench.py` with unique IDs, we also built bench corpus v2. It's worth noting that we ran the baseline on v0, but the results correspond to that. For v2, we haven’t run it yet; the user might want to run it with a specific command that includes audio in the hub dataset. Plus, we fixed the odia mapping to avoid errors.