**Patching command options**

I'm noticing that the run command right now doesn't include the `thresholds_json` option, and the `thresholds` is just {} without being typed out. So, I need to apply a patch to address this. This involves adding the `thresholds_json` parameter at line 228, right after the limit parameter. I'll also need to tweak how the thresholds are created. Let's get this patch applied!


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
 def run(
     segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
     out: Path = typer.Option(Path("outputs/run"), help="Output directory."),
     whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
     lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
@@
     trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
     silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
     min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
     limit: int | None = typer.Option(None, help="Process only the first N segments."),
+    thresholds_json: Path | None = typer.Option(None, help="Optional JSON file of threshold overrides (verify + routing)."),
 ) -> None:
@@
     if ctc_model_map is not None:
         ctc_map = json.loads(ctc_model_map.read_text(encoding="utf-8"))
         if not isinstance(ctc_map, dict):
             raise typer.BadParameter("ctc_model_map must be a JSON object {lang_code: model_id}")

-    thresholds = {}
+    thresholds: dict[str, object] = {}
+    if thresholds_json is not None:
+        loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
+        if not isinstance(loaded, dict):
+            raise typer.BadParameter("thresholds_json must be a JSON object")
+        thresholds = loaded
     if ctc_model or ctc_map:
         thresholds["require_ctc"] = require_ctc
         thresholds["min_ctc_margin"] = float(min_ctc_margin)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Patching bench_run function**

I’m considering patching the `bench_run` function to accept `thresholds_json`. This is important for our plan in step 3. I’ll look at the current signature, which already includes many options, and add `thresholds_json` to it. I'll locate `bench_run` in the file; I know it's at the bottom, following `bench_build`. Once found, I’ll work on patching it to merge everything smoothly. Let’s get this done!


rg -n "def bench_run\(" -n maya_transcribe/cli.py && nl -ba maya_transcribe/cli.py | sed -n '300,450p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
320:def bench_run(
   300	) -> None:
   301	    """
   302	    Build a local benchmark corpus from Hugging Face datasets.
   303	    """
   304	    from maya_transcribe.bench import build_benchmark_corpus
   305
   306	    manifest = build_benchmark_corpus(
   307	        out_dir=out,
   308	        samples_per_lang=samples_per_lang,
   309	        seed=seed,
   310	        shuffle=shuffle,
   311	        langs=lang or None,
   312	        min_duration_sec=min_duration_sec,
   313	        max_duration_sec=max_duration_sec,
   314	        force=force,
   315	    )
   316	    typer.echo(json.dumps({"wrote": str(out), "total_rows": manifest.get("total_rows"), "counts": manifest.get("counts")}, indent=2))
   317
   318
   319	@bench_app.command("run")
   320	def bench_run(
   321	    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
   322	    out: Path | None = typer.Option(None, help="Output directory (defaults under outputs/bench/)."),
   323	    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
   324	    lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
   325	    require_lid_backend: bool = typer.Option(True, help="Fail if MMS-LID backend can't initialize."),
   326	    transformers_whisper_model: list[str] = typer.Option(
   327	        [], help="Extra whisper-family HF models to run via transformers (repeatable)."
   328	    ),
   329	    ctc_model: str | None = typer.Option(None, help="Optional HF CTC model id for audio-grounded fit scoring."),
   330	    ctc_model_map: Path | None = typer.Option(None, help="JSON map {lang_code: hf_model_id} for CTC scoring."),
   331	    require_ctc: bool = typer.Option(True, help="When CTC is enabled, require a CTC score to accept."),
   332	    min_ctc_margin: float = typer.Option(0.05, help="Min (second-best - best) CTC loss margin to auto-accept."),
   333	    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
   334	    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
   335	    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
   336	    limit: int | None = typer.Option(None, help="Process only the first N segments."),
   337	    hypothesis: str = typer.Option("final", help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
   338	) -> None:
   339	    """
   340	    Run the full pipeline on a local benchmark corpus and write eval.json.
   341	    """
   342	    from datetime import datetime
   343
   344	    from maya_transcribe.eval import evaluate
   345	    from maya_transcribe.pipeline import run_pipeline
   346
   347	    audio_dir = bench_dir / "audio"
   348	    refs_path = bench_dir / "refs.jsonl"
   349	    if not audio_dir.exists():
   350	        raise typer.BadParameter(f"Missing: {audio_dir}")
   351	    if not refs_path.exists():
   352	        raise typer.BadParameter(f"Missing: {refs_path}")
   353
   354	    if out is None:
   355	        ts = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
   356	        out = Path("outputs/bench") / bench_dir.name / f"run_{ts}"
   357
   358	    ctc_map = None
   359	    if ctc_model_map is not None:
   360	        ctc_map = json.loads(ctc_model_map.read_text(encoding="utf-8"))
   361	        if not isinstance(ctc_map, dict):
   362	            raise typer.BadParameter("ctc_model_map must be a JSON object {lang_code: model_id}")
   363
   364	    thresholds: dict[str, object] = {}
   365	    if ctc_model or ctc_map:
   366	        thresholds["require_ctc"] = require_ctc
   367	        thresholds["min_ctc_margin"] = float(min_ctc_margin)
   368
   369	    results_path = run_pipeline(
   370	        segments_dir=audio_dir,
   371	        out_dir=out,
   372	        lid_model_id=lid_model_id,
   373	        require_lid_backend=require_lid_backend,
   374	        whisper_model=whisper_model,
   375	        extra_transformers_whisper_models=transformers_whisper_model,
   376	        ctc_model=ctc_model,
   377	        ctc_model_map=ctc_map,
   378	        trim_silence=trim_silence,
   379	        silence_threshold_db=silence_threshold_db,
   380	        min_silence_sec=min_silence_sec,
   381	        limit=limit,
   382	        thresholds=thresholds,
   383	    )
   384
   385	    eval_path = out / "eval.json"
   386	    summary = evaluate(results_path=results_path, refs_path=refs_path, out_path=eval_path, hypothesis=hypothesis, oracle=True)
   387	    typer.echo(f"Wrote: {results_path}")
   388	    typer.echo(f"Wrote: {out / 'summary.json'}")
   389	    typer.echo(f"Wrote: {eval_path}")
   390	    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))
   391
   392
   393	@bench_app.command("push")
   394	def bench_push(
   395	    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
   396	    repo_id: str = typer.Option(..., help="Hugging Face dataset repo id (e.g., org/name)."),
   397	    private: bool = typer.Option(True, help="Create/push as a private dataset repo."),
   398	    include_audio: bool = typer.Option(False, help="Include audio bytes in the pushed dataset (check dataset licenses!)."),
   399	    split: str = typer.Option("test", help="Split name to use on the Hub."),
   400	) -> None:
   401	    """
   402	    Push a local benchmark corpus to the Hugging Face Hub as a (private) dataset.
   403	    """
   404	    from maya_transcribe.bench import push_benchmark_to_hub
   405
   406	    repo = push_benchmark_to_hub(bench_dir=bench_dir, repo_id=repo_id, private=private, include_audio=include_audio, split=split)
   407	    typer.echo(f"Pushed: {repo}")
   408
   409
   410	if __name__ == "__main__":
   411	    app()


Summary: auto


**Patching thresholds for bench_run**

I’m exploring how to add a `thresholds_json` option to `bench_run`. I could incorporate this feature by patching the code and placing the parameter after the limit. Before the CTC thresholds injection, I’ll need to load the thresholds from a file, which would be similar to the existing structure. It feels important to get this right, so I’ll make sure to follow through with the changes!


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
 def bench_run(
@@
     trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
     silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
     min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
     limit: int | None = typer.Option(None, help="Process only the first N segments."),
+    thresholds_json: Path | None = typer.Option(None, help="Optional JSON file of threshold overrides (verify + routing)."),
     hypothesis: str = typer.Option("final", help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
 ) -> None:
@@
-    thresholds: dict[str, object] = {}
+    thresholds: dict[str, object] = {}
+    if thresholds_json is not None:
+        loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
+        if not isinstance(loaded, dict):
+            raise typer.BadParameter("thresholds_json must be a JSON object")
+        thresholds = loaded
     if ctc_model or ctc_map:
         thresholds["require_ctc"] = require_ctc
         thresholds["min_ctc_margin"] = float(min_ctc_margin)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating pipeline configurations**

Now that the thresholds config is accessible, I’m thinking it might be beneficial to update `pipeline.run_pipeline` to accept the thresholds file—oh, it already does! That’s great! Step 3 is done. We also discussed adjusting the evaluation to use reference language, which might happen later, possibly as part of the bench tool. For now, we’ll stick to the plan of just the thresholds config for step 3. I'll mark it as completed and move on to step 4, generating the sample benchmark corpus and updating the plan accordingly.


Output: Plan updated


**Generating benchmark corpus**

I'm working on generating a sample benchmark corpus now. I'm considering running a command for the benchmark with a sample size of around 200 per language, using the default directory. Wait, I've checked that the output path defaults to `data/bench/maya_hf_v0`. It seems I need to confirm what we've set earlier, especially since this path was mentioned before. So, I'll check the lines in my code to ensure everything aligns. Let’s confirm!


rg -n "def bench_build\(|Path\(\"data/bench" -n maya_transcribe/cli.py | head -n 40


Output:
Exit code: 0
Wall time: 0 seconds
Output:
291:def bench_build(
292:    out: Path = typer.Option(Path("data/bench/maya_hf_v0"), help="Output benchmark corpus directory."),
321:    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
401:    bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:

 Usage: maya-transcribe [OPTIONS] COMMAND [ARGS]...

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                  │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ download     Download the cleaned/segmented tar for a video from R2 and      │
│              extract it.                                                     │
│ preprocess   Normalize audio to 16kHz mono WAV, optionally trim silence, and │


maya-transcribe --help | sed -n '1,120p'


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:

 Usage: maya-transcribe [OPTIONS] COMMAND [ARGS]...

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --help          Show this message and exit.                                  │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ download     Download the cleaned/segmented tar for a video from R2 and      │
│              extract it.                                                     │
│ preprocess   Normalize audio to 16kHz mono WAV, optionally trim silence, and │
│              write QC metrics.                                               │
│ lid          Spoken language identification (MMS-LID).                       │
│ asr          Transcribe a single file with faster-whisper.                   │
│ gemini       Transcribe a single file with Gemini (audio-native LLM). Output │
│              is parsed/validated strictly.                                   │
│ run          End-to-end pipeline: preprocess -> LID -> ASR ensemble -> ROVER │
│              -> verify.                                                      │
│ eval         Compute WER/CER for a hypothesis against a reference JSONL      │
│              keyed by `segment_id`.                                          │
│ bench        Benchmarking utilities.                                         │
╰──────────────────────────────────────────────────────────────────────────────╯


**Checking command options**

I've noted that the bench command is present, which is a good start. Now, I’m planning to run the command `maya-transcribe bench build --help` to see what options and help documentation are available. This step is crucial for understanding how to proceed with the build process. I just want to make sure I'm fully informed before moving on! Let's see what the help output reveals.


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:

 Usage: maya-transcribe bench build [OPTIONS]

 Build a local benchmark corpus from Hugging Face datasets.

╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --out                                 PATH     Output benchmark corpus       │
│                                                directory.                    │
│                                                [default:                     │
│                                                data/bench/maya_hf_v0]        │
│ --samples-per-lang                    INTEGER  Target samples per language.  │
│                                                [default: 200]                │
│ --seed                                INTEGER  Shuffle seed. [default: 0]    │
│ --shuffle             --no-shuffle             Shuffle within each source    │
│                                                before selecting.             │
│                                                [default: shuffle]            │
│ --lang                                TEXT     Limit to ISO-639-3 language   │
│                                                codes (repeatable).           │
│ --min-duration-sec                    FLOAT    Min audio duration.           │
│                                                [default: 1.0]                │
│ --max-duration-sec                    FLOAT    Max audio duration.           │
│                                                [default: 30.0]               │
│ --force               --no-force               Delete and rebuild output     │
│                                                directory if it exists.       │
│                                                [default: no-force]           │
│ --help                                         Show this message and exit.   │
╰──────────────────────────────────────────────────────────────────────────────╯


**Running benchmark build**

I’m ready to run the benchmark build with the default setting of 200 per language. This should create a new directory at `data/bench/maya_hf_v0`. I wonder if this directory already exists, though, so I'll check that first before proceeding. It’s always a good idea to avoid overwriting anything unintentionally while ensuring everything goes smoothly! Let’s see what happens next.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
ls: cannot access 'data/bench': No such file or directory


maya-transcribe bench build --samples-per-lang 200


Output:
Exit code: 0
Wall time: 946.6 seconds
Total output lines: 48
Output:

Downloading data:   0%|          | 0/22 [00:00<?, ?files/s]
Downloading data:   5%|▍         | 1/22 [00:02<00:45,  2.18s/files]
Downloading data:   9%|▉         | 2/22 [00:04<00:43,  2.16s/files]
Downloading data:  14%|█▎        | 3/22 [00:06<00:38,  2.04s/files]
Downloading data:  18%|█▊        | 4/22 [00:08<00:36,  2.02s/files]
Downloading data:  23%|██▎       | 5/22 [00:10<00:35,  2.11s/files]
Downloading data:  27%|██▋       | 6/22 [00:12<00:32,  2.03s/files]
Downloading data:  32%|███▏      | 7/22 [00:14<00:30,  2.02s/files]
Downloading data:  36%|███▋      | 8/22 [00:16<00:27,  1.96s/files]
Downloading data:  41%|████      | 9/22 [00:18<00:26,  2.08s/files]
Downloading data:  45%|████▌     | 10/22 [00:20<00:24,  2.06s/files]
Downloading data:  50%|█████     | 11/22 [00:22<00:23,  2.16s/files]
Downloading data:  55%|█████▍    | 12/22 [00:25<00:21,  2.13s/files]
Downloading data:  59%|█████▉    | 13/22 [00:27<00:20,  2.32s/files]
Downloading data:  64%|██████▎   | 14/22 [00:30<00:18,  2.32s/files]
Downloading data:  68%|██████▊   | 15/22 [00:32<00:15,  2.22s/files]
Downloading data:  73%|███████▎  | 16/22 [00:34<00:14,  2.36s/files]
Downloading data:  77%|███████▋  | 17/22 [00:36<00:11,  2.24s/files]
Downloading data:  82%|████████▏ | 18/22 [00:38<00:08,  2.22s/files]
Downloading data:  86%|████████▋ | 19/22 [00:41<00:07,  2.40s/files]
Downloading data:  91%|█████████ | 20/22 [00:43<00:04,  2.28s/files]
Downloading data:  95%|█████████▌| 21/22 [00:45<00:02,  2.23s/files]
Downloading data: 100%|██████████| 22/22 [00:47<00:00,  2.17s/files]
Downloading data: 100%|██████████| 22/22 [00:47<00:00,  2.18s/files]

Generating valid split:   0%|          | 0/2236 [00:00<?, ? examples/s]
Generating valid split:   4%|▍         | 100/2236 [00:00<00:02, 949.07 examples/s]
Generating valid split:  22%|██▏       | 500/2236 [00:00<00:00, 2199.69 examples/s]
Generating valid split:  45%|████▍     | 1000/2236 [00:00<00:00, 3021.46 examples/s]
Generating valid split:  68%|██████▊   | 1518/2236 [00:00<00:00, 3221.72 examples/s]
Generating valid split:  86%|████████▌ | 1918/2236 [00:00<00:00, 3178.21 examples/s]
Generating valid split: 100%|██████████| 2236/2236 [00:00<00:00, 3124.39 examples/s]
Generating valid split: 100%|██████████| 2236/2236 [00:00<00:00, 2963.00 examples/s]

Generating train split:   0%|          | 0/47417 [00:00<?, ? examples/s]
Generating train split:   1%|          | 400/47417 [00:00<00:14, 3138.97 examples/s]
Generating train split:   2%|▏         | 900/47417 [00:00<00:12, 3685.17 examples/s]
Generating train split:   3%|▎         | 1300/47417 [00:00<00:13, 3433.49 examples/s]
Generating train split:   4%|▎         | 1700/47417 [00:00<00:13, 3407.16 examples/s]
Generating train split:   5%|▍         | 2256/47417 [00:00<00:14, 3204.33 examples/s]
Generating train split:   6%|▌         | 2856/47417 [00:00<00:12, 3544.22 examples/s]
Generating train split:   7%|▋         | 3256/47417 [00:00<00:12, 3399.58 examples/s]
Generating train split:   8%|▊         | 3656/47417 [00:01<00:12, 3474.60 examples/s]
Generating train split:   9%|▊         | 4056/47417 [00:01<00:12, 3519.05 examples/s]
Generating train split:   9%|▉         | 4412/47417 [00:01<00:12, 3372.83 examples/s]
Generating train split:  11%|█         | 5012/47417 [00:01<00:10, 3911.43 examples/s]
Generating train split:  12%|█▏        | 5612/47417 [00:01<00:10, 3919.69 examples/s]
Generating train split:  13%|█▎        | 6112/47417 [00:01<00:11, 3715.45 examples/s]
Generating train split:  14%|█▍        | 6568/47417 [00:01<00:11, 3678.99 examples/s]
Generating train split:  15%|█▌        | 7168/47417 [00:01<00:10, 3881.74 examples/s]
Generating train split:  17%|█▋        | 7868/47417 [00:02<00:08, 4403.13 examples/s]
Generating train split:  18%|█▊        | 8368/47417 [00:02<00:08, 4417.61 examples/s]
Generating train split:  19%|█▉        | 9024/47417 [00:02<00:08, 4384.61 examples/s]
Generating train split:  20%|██        | 9624/47417 [00:02<00:08, 4506.16 examples/s]
Generating train split:  22%|██▏       | 10224/47417 [00:02<00:08, 4526.55 examples/s]
Generating train split:  23%|██▎       | 10880/47417 [00:02<00:08, 4217.44 examples/s]
Generating train split:  24%|██▍       | 11580/47417 [00:02<00:08, 4450.14 examples/s]
Generating train split:  25%|██▌       | 12080/47417 [00:03<00:08, 4409.96 examples/s]
Generating train split:  27%|██▋       | 12680/47417 [00:03<00:07, 4602.09 examples/s]
Generating train split:  28%|██▊       | 13336/47417 [00:03<00:07, 4520.25 examples/s]
Generating train split: …126736 chars truncated…]
Generating train split:  72%|███████▏  | 50963/70673 [00:11<00:04, 4107.36 examples/s]
Generating train split:  73%|███████▎  | 51504/70673 [00:11<00:04, 3852.46 examples/s]
Generating train split:  74%|███████▎  | 52104/70673 [00:12<00:04, 4155.77 examples/s]
Generating train split:  75%|███████▍  | 52804/70673 [00:12<00:04, 4054.66 examples/s]
Generating train split:  76%|███████▌  | 53404/70673 [00:12<00:04, 4245.41 examples/s]
Generating train split:  76%|███████▌  | 53845/70673 [00:12<00:04, 3941.22 examples/s]
Generating train split:  77%|███████▋  | 54245/70673 [00:12<00:04, 3798.65 examples/s]
Generating train split:  77%|███████▋  | 54745/70673 [00:12<00:04, 3835.84 examples/s]
Generating train split:  78%|███████▊  | 55145/70673 [00:12<00:04, 3658.01 examples/s]
Generating train split:  79%|███████▊  | 55545/70673 [00:12<00:04, 3625.42 examples/s]
Generating train split:  79%|███████▉  | 55986/70673 [00:13<00:04, 3208.98 examples/s]
Generating train split:  80%|███████▉  | 56386/70673 [00:13<00:04, 3178.27 examples/s]
Generating train split:  80%|████████  | 56886/70673 [00:13<00:04, 3237.02 examples/s]
Generating train split:  81%|████████  | 57286/70673 [00:13<00:04, 3136.49 examples/s]
Generating train split:  82%|████████▏ | 57686/70673 [00:13<00:04, 3073.08 examples/s]
Generating train split:  82%|████████▏ | 58127/70673 [00:13<00:04, 2961.81 examples/s]
Generating train split:  83%|████████▎ | 58627/70673 [00:13<00:03, 3308.57 examples/s]
Generating train split:  84%|████████▎ | 59027/70673 [00:14<00:03, 3286.70 examples/s]
Generating train split:  84%|████████▍ | 59427/70673 [00:14<00:03, 3254.83 examples/s]
Generating train split:  85%|████████▍ | 59827/70673 [00:14<00:03, 3206.48 examples/s]
Generating train split:  85%|████████▌ | 60268/70673 [00:14<00:03, 3082.60 examples/s]
Generating train split:  86%|████████▌ | 60668/70673 [00:14<00:03, 3157.74 examples/s]
Generating train split:  87%|████████▋ | 61168/70673 [00:14<00:02, 3286.89 examples/s]
Generating train split:  87%|████████▋ | 61568/70673 [00:14<00:02, 3275.87 examples/s]
Generating train split:  88%|████████▊ | 61968/70673 [00:15<00:02, 3133.78 examples/s]
Generating train split:  88%|████████▊ | 62509/70673 [00:15<00:02, 3109.58 examples/s]
Generating train split:  89%|████████▉ | 62909/70673 [00:15<00:02, 3118.49 examples/s]
Generating train split:  90%|████████▉ | 63309/70673 [00:15<00:02, 3212.26 examples/s]
Generating train split:  90%|█████████ | 63709/70673 [00:15<00:02, 3288.77 examples/s]
Generating train split:  91%|█████████ | 64109/70673 [00:15<00:02, 3250.90 examples/s]
Generating train split:  91%|█████████▏| 64650/70673 [00:15<00:01, 3185.38 examples/s]
Generating train split:  92%|█████████▏| 65250/70673 [00:16<00:01, 3421.43 examples/s]
Generating train split:  93%|█████████▎| 65650/70673 [00:16<00:01, 3500.91 examples/s]
Generating train split:  93%|█████████▎| 66050/70673 [00:16<00:01, 3392.45 examples/s]
Generating train split:  94%|█████████▍| 66491/70673 [00:16<00:01, 3035.14 examples/s]
Generating train split:  95%|█████████▍| 66891/70673 [00:16<00:01, 3102.16 examples/s]
Generating train split:  95%|█████████▌| 67291/70673 [00:16<00:01, 3241.83 examples/s]
Generating train split:  96%|█████████▌| 67791/70673 [00:16<00:00, 3362.06 examples/s]
Generating train split:  96%|█████████▋| 68191/70673 [00:16<00:00, 3281.50 examples/s]
Generating train split:  97%|█████████▋| 68632/70673 [00:17<00:00, 2918.35 examples/s]
Generating train split:  98%|█████████▊| 69032/70673 [00:17<00:00, 2981.99 examples/s]
Generating train split:  98%|█████████▊| 69532/70673 [00:17<00:00, 3134.99 examples/s]
Generating train split:  99%|█████████▉| 70032/70673 [00:17<00:00, 3251.94 examples/s]
Generating train split: 100%|█████████▉| 70432/70673 [00:17<00:00, 3147.28 examples/s]
Generating train split: 100%|██████████| 70673/70673 [00:17<00:00, 3985.74 examples/s]
{
  "wrote": "data/bench/maya_hf_v0",
  "total_rows": 2400,
  "counts": {
    "asm": 200,
    "ben": 200,
    "eng": 200,
    "guj": 200,
    "hin": 200,
    "kan": 200,
    "mal": 200,
    "mar": 200,
    "ory": 200,
    "pan": 200,
    "tam": 200,
    "tel": 200
  }
}


**Noting build progress**

The build command succeeded, creating 2400 rows. I see some progress bars, but they’re pretty truncated in the output, which is a bit frustrating. Still, it built successfully! Now I can plan for step 4, which is completed, and it only took about 946 seconds, or 15.8 minutes. I’m thinking about adjusting the build to avoid generating train splits, but I can handle that later. Now, step 5 is underway: running the pipeline and evaluating the corpus.


Output: Plan updated


Output:
Exit code: 0
Wall time: 0 seconds
Output:
ls: cannot access 'outputs/bench': No such file or directory


maya-transcribe bench run --bench-dir data/bench/maya_hf_v0 --out outputs/bench/maya_hf_v0/baseline --whisper-model large-v3


Output:
Exit code: 0
Wall time: 6312.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 07:31:31.407132: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1:44:50
Wrote: outputs/bench/maya_hf_v0/baseline/results.jsonl
Wrote: outputs/bench/maya_hf_v0/baseline/summary.json
Wrote: outputs/bench/maya_hf_v0/baseline/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 2188,
  "wer_mean": 0.958137176179309,
  "cer_mean": 0.6892070914668146,
  "wer_p50": 1.0,
  "wer_p90": 1.2,
  "wer_p95": 1.6666666666666667,
  "cer_p50": 0.7303370786516854,
  "cer_p90": 1.0,
  "cer_p95": 1.1979166666666667,
  "segment_error_rate": 0.9972577696526508,
  "pct_wer_gt_5pct": 0.9972577696526508,
  "pct_wer_gt_10pct": 0.9894881170018281,
  "wer_by_language_mean": {
    "asm": 1.2855066632628924,
    "bak": 1.0,
    "ben": 0.8297152641597085,
    "ell": 2.3333333333333335,
    "eng": 0.5861004342584378,
    "guj": 0.5656685455465943,
    "hin": 0.48697925451172186,
    "hne": 0.15384615384615385,
    "kan": 0.850996614108502,
    "mal": 1.3949920384779537,
    "mar": 1.0165202659994323,
    "npi": 1.3510926573426572,
    "pan": 0.836297166666294,
    "pus": 1.6709090909090907,
    "rwr": 0.4444444444444444,
    "san": 4.814814814814815,
    "sin": 0.8611111111111112,
    "snd": 0.75,
    "som": 1.0,
    "tam": 0.5683152987674263,
    "tel": 0.9430083435083432,
    "urd": 0.5772435897435897,
    "war": 1.0,
    "zlm": 1.0
  },
  "wer_by_decision_mean": {
    "accept": 0.6600290144894885,
    "human": 1.068130605528151,
    "reject": 1.1809687125866046
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 1.1180530532878201,
    "10-30s": 1.3224812060525328,
    "3-10s": 0.8010562896560971
  },
  "cer_by_language_mean": {
    "asm": 1.0873550520794881,
    "bak": 1.0,
    "ben": 0.4543486859042112,
    "ell": 3.979591836734694,
    "eng": 0.5752812640135805,
    "guj": 0.2573701819623728,
    "hin": 0.3705180414752323,
    "hne": 0.09090909090909091,
    "kan": 0.5116578101968073,
    "mal": 0.9972329408341366,
    "mar": 0.47915245953251684,
    "npi": 1.342610448807854,
    "pan": 0.4378836925945288,
    "pus": 1.0964646464646464,
    "rwr": 0.05,
    "san": 1.488888888888889,
    "sin": 0.79,
    "snd": 0.6704545454545454,
    "som": 1.0,
    "tam": 0.2548852431753345,
    "tel": 0.5696049548133477,
    "urd": 0.4483899594320487,
    "war": 1.0,
    "zlm": 1.0
  },
  "cer_by_decision_mean": {
    "accept": 0.3739248809149772,
    "human": 0.7492229748256926,
    "reject": 1.0849652909765737
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.9660921611102244,
    "10-30s": 1.0225491434559282,
    "3-10s": 0.42673370170984454
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.8459395614677641,
    "selection_gap_wer_mean": 0.11219761471154517,
    "selection_gap_wer_p90": 0.11111111111111116
  }
}


**Analyzing benchmark results**

The benchmark run took 1 hour and 44 minutes, resulting in an evaluation summary. Unfortunately, the Word Error Rate (WER) is extremely high at approximately 0.96, indicating that almost all transcripts are incorrect. The segment error rate is around 0.997, which suggests that nearly every segment is wrong. I'm puzzled about the cause; perhaps the references are in scripts while the output is in romanization, or maybe there are many empty outputs? This is concerning!


**Investigating WER issues**

The Word Error Rate (WER) by language includes several languages not found in our dataset, like 'bak', 'ell', and 'hne'. This indicates that the evaluation is grouping languages based on the predicted language from the pipeline (route_lang) instead of the reference language. Our references include specific language codes like 'hin' and 'tam', which suggests potential mismatches. Alarmingly, while we processed 2400 segments, only 2188 were evaluated, meaning 212 segments might be missing, possibly due to errors in the pipeline output. This could point to bigger issues!


**Checking evaluation output**

The evaluation expects the `final` text to be present; if there's an error row, the final text will be missing, and it will skip that row. That could explain some issues, but not all of them. The extremely high WER suggests that the pipeline output might be in English transliteration or using the wrong script. I should inspect a sample results row to confirm this, so I'll view the first lines of results.jsonl using Python. Let's check!


python3 - <<'PY'
import json
from pathlib import Path
p=Path('outputs/bench/maya_hf_v0/baseline/results.jsonl')
for line in p.open(encoding='utf-8'):
    r=json.loads(line)
    if 'error' in r:
        continue
    print('segment_id', r.get('segment_id'))
    print('route_lang', (r.get('route') or {}).get('language_hint'))
    print('final', (r.get('final') or {}).get('text'))
    print('rover', (r.get('rover') or {}).get('text'))
    print('asr_keys', list((r.get('asr') or {}).keys())[:3])
    # show one asr
    ak=list((r.get('asr') or {}).keys())[0]
    print('asr0 model', (r['asr'][ak] or {}).get('model'))
    print('asr0 text', (r['asr'][ak] or {}).get('text'))
    break
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
segment_id fleurs__1663-4117f7bc63f17a7e
route_lang asm
final कोशाम उपज्ट का सिलेर मूल पर्वता रखन धाई
rover कोशाम उपज्ट का सिलेर मूल पर्वता रखन धाई
asr_keys ['faster-whisper:large-v3|det', 'faster-whisper:large-v3|samp_t0.2']
asr0 model faster-whisper:large-v3
asr0 text कोशाम उपज्ट का सिलेर मूल पर्वता रखन धाई।


**Loading samples for analysis**

I'm planning to load a sample from the results along with the references to see how they differ. It sounds like a good idea to use Python to join this with the refs.jsonl file. I think sampling a few will give me a clearer picture. The process feels straightforward, and I'm curious about the outcome! Let's see how it all comes together.


python3 - <<'PY'
import json
from pathlib import Path

refs_path=Path('data/bench/maya_hf_v0/refs.jsonl')
refs={}
for line in refs_path.open(encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

res_path=Path('outputs/bench/maya_hf_v0/baseline/results.jsonl')

count=0
for line in res_path.open(encoding='utf-8'):
    r=json.loads(line)
    if 'error' in r:
        continue
    seg=r['segment_id']
    ref=refs.get(seg)
    if not ref:
        continue
    # pick some hindi ref
    if ref.get('lang')!='hin':
        continue
    print('SEG', seg)
    print('REF', ref['text'])
    print('HYP', (r.get('final') or {}).get('text'))
    print('ROUTE', (r.get('route') or {}).get('language_hint'))
    print('---')
    count += 1
    if count>=3:
        break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
SEG Kathbath__844424930552454-229-f.m4a-747a7f4324b66bb3
REF इस दौरान पेट्रोलियम क्षेत्र में सहयोग पर काफी अहम वार्ता होनी है
HYP इस दोरान पेट्रोलियम शेत्र में सईयोग पर काफी एहम वारता होनी है
ROUTE hin
---
SEG Kathbath__844424930561673-229-f.m4a-875b420cf77acb54
REF सिर में चोट लगने से पूजा की हालत गंभीर बनी हुई है
HYP सिर्ब में चोट लगने से पूजा की हालत गंभीर बनी हुई है
ROUTE hin
---
SEG Kathbath__844424930568740-939-f.m4a-e87ba01c86a5e61a
REF गांव में पहुंचने पर बलराज का भव्य स्वागत किया गया
HYP गाउ में पहुचने पर
ROUTE None
---


Summary: auto


python3 - <<'PY'
import json
from pathlib import Path

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

count=0
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    if 'error' in r:
        continue
    seg=r['segment_id']
    ref=refs.get(seg)
    if not ref or ref.get('lang')!='mal':
        continue
    print('SEG', seg)
    print('REF', ref['text'])
    print('HYP', (r.get('final') or {}).get('text'))
    print('ROUTE', (r.get('route') or {}).get('language_hint'))
    print('---')
    count += 1
    if count>=3:
        break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
SEG Kathbath__844424930310407-814-f.m4a-239d3a2b1121cfe1
REF ജില്ലാ തിരഞ്ഞെടുപ്പ് ഓഫീസറുടെ നിര്ദേശപ്രകാരമാണ് ആദായനികുതി വകുപ്പും തിരഞ്ഞെടുപ്പ് ഫ്ളൈയിങ് സ്ക്വാഡും ദുരൈ മുരുകന്റെ വസതിയില് പരിശോധന നടത്തുന്നത്
HYP ਦੀਲਾ ਤਰੀਣਿਲ ਪੂ ਅਫੀਸਰੋ ਵੋਡੇ ਨਰੁਦੇਸ਼ ਪਰਾਗਾਰਮ ਆਨੇ
ROUTE mal
---
SEG Kathbath__844424930337869-814-f.m4a-abfc73343aa07824
REF കാരണം പൂക്കള്ക്കും ചെടികള്ക്കും മനുഷ്യര്ക്കുമിടയിലുള്ള സ്നേഹത്തിന്റെ അഭൌമമായ സൌന്ദര്യം ചൊരിയുന്ന പെണ്ണുങ്ങള്ക്കൊന്നും അവളവളെ കൊത്തിവെക്കാതെ നടന്ന് പോകാന് കഴിയില്ല
HYP ਕਾਰਣ ਪੂਕਲ ਕੁਂ ਚੇਡੀਗਲ ਕੁਂ ਮਨੁਸਰ ਕੁ ਕੁਂ ਇਡੈਲ ਸ੍ਨੇਗਤੀਂਦੇ ਅਬੋਉਮਾ ਮਾਯ ਸੌਂਦੇਰ ਚੇਡੀਗਲ ਪੁਣੁਂਗਲ ਕੁਂ ਨੁਂ
ROUTE mal
---
SEG Kathbath__844424930337915-814-f.m4a-3bbbedcaa01f08ce
REF ഇന്നസെന്റിനോടുള്ള ജനത്തിന്റെ കടുത്ത എതിര്പ്പും മണ്ഡലത്തിലെ യു ഡി എഫ് മേല്ക്കോയ്മയും ബെന്നി ബെഹന്നാനും നേട്ടമായി
HYP ഇന്നസ്നിട്ടിനോഡോളുള്ളാ ജനത്തീന്നെ കഡുത്താ ഇദര്പും മണ്ടരത്ത്തുലെ യുഡീയൾഫ് മേല്കോയ്മൈം ബിന്നീ ബഹന്നാന
ROUTE mal
---


**Evaluating script discrepancies**

I’m looking at the outputs for Malayalam; it seems like some are in Punjabi script. The first two hypotheses start with 'ਦੀਲਾ', which is definitely Punjabi. It’s strange because Whisper should be transcribing Malayalam audio correctly. Maybe the model is outputting in the wrong script? I wonder if the normalize_text function is altering the original script. Actually, while the first two look like Gurmukhi, it seems they might just be Malayalam words represented incorrectly. It's all quite confusing!


**Analyzing language detection issues**

Okay, so the route_lang is set as 'mal', and it should translate to 'ml' for Malayalam in the Whisper model. However, I see Punjabi output for some segments, which raises questions. It might be that those segments aren't actually Malayalam; maybe there’s code-switching happening or the data is mislabeled. I wonder if the language identification model is misclassifying, leading to these unexpected results. The first hypothesis I checked clearly appears to be in Gurmukhi, which adds to the confusion!


**Exploring language output issues**

It looks like the model is outputting in Gurmukhi script, which raises questions since we expected Malayalam. I wonder if the faster-whisper model uses ISO-639-1 codes, and maybe it didn’t recognize 'ml' for Malayalam, defaulting to Punjabi. While the route_lang from LID is 'mal', it translates to 'ml', which should be supported. I plan to check the faster-whisper documentation to confirm if 'ml' is listed as a valid language code!


python3 - <<'PY'
from faster_whisper.tokenizer import LANGUAGES
print('ml' in LANGUAGES)
print('as' in LANGUAGES)
print('or' in LANGUAGES)
print('kn' in LANGUAGES)
print('te' in LANGUAGES)
print('bn' in LANGUAGES)
print('pa' in LANGUAGES)
print('gu' in LANGUAGES)
print('mr' in LANGUAGES)
print('ta' in LANGUAGES)
PY


Output:
Exit code: 1
Wall time: 2.5 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'LANGUAGES' from 'faster_whisper.tokenizer' (/home/ubuntu/.local/lib/python3.10/site-packages/faster_whisper/tokenizer.py)


python3 - <<'PY'
import inspect
import faster_whisper.tokenizer as t
print([n for n in dir(t) if 'LANG' in n])
print('module path', t.__file__)
PY


Output:
Exit code: 0
Wall time: 2.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
['_LANGUAGE_CODES']
module path /home/ubuntu/.local/lib/python3.10/site-packages/faster_whisper/tokenizer.py


python3 - <<'PY'
from faster_whisper import tokenizer
codes = getattr(tokenizer,'_LANGUAGE_CODES',{})
print('ml' in codes)
print('as' in codes)
print('or' in codes)
print('kn' in codes)
print('te' in codes)
print('bn' in codes)
print('pa' in codes)
print('gu' in codes)
print('mr' in codes)
print('ta' in codes)
print('en' in codes)
PY


Output:
Exit code: 0
Wall time: 2.5 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
True
True
False
True
True
True
True
True
True
True
True


**Inspecting audio inconsistencies**

I think the audio might be mislabeled; it could be in Punjabi but noted as Malayalam. But, if that's true, maybe the LID would still recognize it as Malayalam due to noise—though that seems unlikely. I should look at the specific audio file. The naming convention suggests it's from a Malayalam dataset, so if that’s the case, the audio should match the transcription. I'm wondering if the transcripts are incorrect instead. It might be worth checking for possible shuffling issues in the dataset.


**Analyzing dataset language filtering**

I'm thinking about the dataset that has multiple languages and whether the config selection filters it correctly—maybe it doesn’t. I’ll check the 'lang' field in the Kathbath dataset example; could it be 'malayalam'? Earlier, I saw that keys include a 'lang' field, so for Hindi it might say 'hindi.' I suspect the config doesn’t filter, yet the dataset includes this field consistently. Additionally, the evaluation summary suggests segments might be misclassified, possibly due to the audio's quality or trimming affecting length. I need to verify that!


**Investigating WER issues**

There seems to be a significant issue with the WER mean being extremely high. I wonder if the evaluation is using reference texts in the original script while our hypothesis was normalized, meaning punctuation was removed. That should still align, but is there a chance the wrong references are being used due to a mismatched segment ID? The Hindi examples suggest that matching references is correct. However, overall WER is close to 1, possibly because segments in languages like Malayalam are completely incorrect. I should compute WER by reference language using our reference file to get a clearer picture!


**Updating evaluation metrics**

I think we need to modify the evaluation process to use the reference language and compute the mean WER per reference language to get a clearer picture. This step was part of our earlier plan but hasn’t been completed yet. Implementing this should also help remove unnecessary languages from the summary. Currently, the overall WER mean is sitting at 0.958, which feels way too high, likely influenced by segments with short or empty hypotheses. Let’s quickly compute the distribution and confirm the mean WER per reference language!


**Computing WER with jiwer**

I’m planning to calculate the WER using the jiwer library in Python, normalizing the reference and hypothesis with the correct language from the references. This should help confirm whether the WER mean is really 0.958 or perhaps even lower. I’ll run this calculation quickly to get our results. It’s important to ensure that the evaluation reflects the actual performance accurately! Let’s proceed with that.


python3 - <<'PY'
import json
from collections import defaultdict
from pathlib import Path
from jiwer import wer
from maya_transcribe.textnorm import normalize_text

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

by_lang=defaultdict(list)
overall=[]
missing=0
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if not ref:
        missing+=1
        continue
    final=(r.get('final') or {}).get('text')
    if not isinstance(final,str):
        continue
    ref_text=ref.get('text','')
    lang=ref.get('lang')
    ref_n=normalize_text(ref_text, lang=lang)
    hyp_n=normalize_text(final, lang=lang)
    w=float(wer(ref_n, hyp_n))
    by_lang[lang].append(w)
    overall.append(w)

print('pairs', len(overall), 'missing_ref', missing)
print('overall mean', sum(overall)/len(overall))
for lang, ws in sorted(by_lang.items()):
    print(lang, 'n', len(ws), 'mean', sum(ws)/len(ws))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
pairs 2188 missing_ref 0
overall mean 0.958137176179309
asm n 164 mean 1.2136130359678567
ben n 196 mean 0.9608822300148822
eng n 171 mean 0.9179840475746643
guj n 200 mean 0.6482344183594183
hin n 200 mean 0.6929420024420025
kan n 200 mean 0.9317456016206016
mal n 200 mean 1.3426665695415694
mar n 200 mean 1.0868887640137634
ory n 57 mean 1.5957731352468192
pan n 200 mean 0.8963747224997225
tam n 200 mean 0.6000177461427461
tel n 200 mean 1.1066519036519034


**Analyzing WER performance**

I noticed that the overall mean is 0.958, matching the evaluation summary, which indicates it’s correct. However, the word error rate (WER) for most languages is alarmingly high. Only Gujarati (0.65) and Tamil (0.60) are performing relatively well; even Hindi is at 0.69! This suggests that the transcripts aren't great overall. I think we need to inspect some Hindi examples to determine if many are truncated. To improve WER, we should consider using specialized Indic models like IndicWhisper, which reportedly has a much lower WER.


**Searching for model IDs**

I need to find specific model IDs, possibly for something like "ai4bharat/indicwhisper" or "ai4bharat/indic-whisper." I think it makes sense to search on Hugging Face quickly. Using the huggingface_hub, I can query for models that match "IndicWhisper." I’ll employ a script to help list and filter the available models. It seems straightforward, and since I have access to the network, I’ll run this search right away.


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN']=os.environ['HF_TOKEN']

from huggingface_hub import HfApi
api=HfApi()
# search for indicwhisper models
models = api.list_models(search='IndicWhisper', limit=20)
print('found', len(models))
for m in models[:20]:
    print(m.modelId)
PY


Output:
Exit code: 1
Wall time: 0.2 seconds
Output:
Traceback (most recent call last):
  File "<stdin>", line 12, in <module>
TypeError: object of type 'generator' has no len()


Summary: auto


**Modifying model iteration**

I’m working with `api.list_models`, which returns a generator that we can iterate over. This gives me a chance to look through all the models available. I'm thinking about how to modify this process for better efficiency or clarity. It’s always good to ensure everything functions smoothly, and I want to make sure I tackle any potential issues that might come up while iterating through the models. Let’s see what adjustments I can make!


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN']=os.environ['HF_TOKEN']

from huggingface_hub import HfApi
api=HfApi()

print('Top matches for IndicWhisper:')
for i,m in enumerate(api.list_models(search='IndicWhisper', limit=25)):
    print(m.modelId)
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
Top matches for IndicWhisper:
sawradip/indicwhisper-m-hi-float16-ct2
parthiv11/indic_whisper_hi_multi_gpu
parthiv11/indic_whisper_nodcil
Vignesh-M/Indic-whisper
HariprasathSB/indic-whisper-vulnerable
HariprasathSB/indic-whisper-vulnerable1
HariprasathSB/indic-whisper-vulnerable2
vrclc/indicwhisper-Mal
kavyamanohar/AI4B-Indicwhisper-ml
kavyamanohar/AI4B-Indicwhisper-or
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-8
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-9
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-10
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-11
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-13
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-14
Koosh0610/indic-whisper-reverse-ml-mft-1-0-1-15
adalat-ai/ct2-AI4B-Indicwhisper-ml


**Searching for models**

I’m checking the search results, and it seems there are many derivative models but not the official AI4Bharat IndicWhisper models. It might be a good idea to look into the repo `ai4bharat/indicwhisper`, but the search didn’t show that exact name — maybe it’s a case sensitivity issue since I used uppercase. I’ll refine my search for models with 'ai4bharat' and 'IndicWhisper' to find what I need!


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
for i,m in enumerate(api.list_models(author='ai4bharat', search='whisper', limit=50)):
    if 'whisper' in m.modelId.lower():
        print(m.modelId)
PY


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:


**Considering language options**

I’m thinking about adding a pipeline option, `--language-hint`, to take control over language detection and force the right one for all segments. This could be great for benchmarking because it lets us ensure that the ASR uses the correct code, potentially reducing word error rate (WER). Currently, it’s set up for unknown-language segments, but if we know the language from our benchmark dataset, we can optimize things by inferring from the path, which could be really handy!


**Exploring language hint options**

I’m considering adding a CLI option to the `run` command, allowing for a `--language` hint for all segments. Previously, the CLI included 'lid' commands but not this feature. For benchmarking, I think implementing `--path-language-hints` could help. If the segments directory has subdirectories named with language codes, we could use those as `route_lang`, thereby enabling forced language without relying on LID. This might significantly improve results, especially since high WER may stem from incorrect script output. There are some inconsistencies with languages like Malayalam and Odia, where the mapping causes fallback issues.


**Analyzing Odia segment results**

I’m trying to figure out why only 57 out of 200 Odia samples were evaluated. It seems there could be pipeline errors or maybe issues with missing segment IDs, leading to many results not having final text. I need to check if the Odia audio path with 'ory' contributed to any errors. But why only 57 segments matched? The script computed a reference for 57, suggesting that 143 segments didn’t result—either due to errors or were never processed. This requires further investigation.


**Reviewing pipeline processing**

I’m looking at the pipeline, which processed a total of 2400 audio files and writes results for each source. If any exceptions occur, the records only contain an error field. The evaluation script has a line that skips any rows where 'final' text is missing or not a string. This means many Odia segments might be categorized as error rows. I really should check the error count by language to get a clearer picture of what's happening. Let’s compute that now!


python3 - <<'PY'
import json
from collections import Counter

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

errors=Counter(); total=Counter();
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if not ref:
        continue
    lang=ref.get('lang')
    total[lang]+=1
    if 'error' in r:
        errors[lang]+=1
    else:
        # sometimes final missing? but should.
        if not isinstance((r.get('final') or {}).get('text'), str):
            errors[lang]+=1

print('total per lang', total)
print('errors per lang', errors)
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
total per lang Counter({'ben': 200, 'guj': 200, 'hin': 200, 'kan': 200, 'mal': 200, 'mar': 200, 'ory': 200, 'pan': 200, 'tam': 200, 'tel': 200, 'eng': 171, 'asm': 164})
errors per lang Counter({'ory': 143, 'ben': 4})


**Inspecting Odia segment errors**

I see that for Odia (ory), 143 out of 200 segments are showing errors, which is a significant issue. It makes me wonder if ffmpeg has trouble decoding some audio arrays. We wrote the wav files from those arrays, so that should work—unless there are NaNs in the arrays, but I don’t think that's the case. Perhaps the problem lies in overly long or invalid file names? I should check an error row for an Odia segment to investigate further.


python3 - <<'PY'
import json
from pathlib import Path

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    if 'error' not in r:
        continue
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if ref and ref.get('lang')=='ory':
        print('seg', seg)
        print('src', r.get('src_path'))
        print('error', r.get('error'))
        break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
seg Kathbath__844424930363309-883-f.m4a-66f8542cb6246fd2
src data/bench/maya_hf_v0/audio/ory/Kathbath__844424930363309-883-f.m4a.wav
error 'or' is not a valid language code (accepted language codes: af, am, ar, as, az, ba, be, bg, bn, bo, br, bs, ca, cs, cy, da, de, el, en, es, et, eu, fa, fi, fo, fr, gl, gu, ha, haw, he, hi, hr, ht, hu, hy, id, is, it, ja, jw, ka, kk, km, kn, ko, la, lb, ln, lo, lt, lv, mg, mi, mk, ml, mn, mr, ms, mt, my, ne, nl, nn, no, oc, pa, pl, ps, pt, ro, ru, sa, sd, si, sk, sl, sn, so, sq, sr, su, sv, sw, ta, te, tg, th, tk, tl, tr, tt, uk, ur, uz, vi, yi, yo, zh, yue)


**Identifying language mapping issue**

I’ve realized the error arises because ‘or’ is not a valid language code according to the accepted list. This is likely causing many issues, including a high word error rate (WER). To fix this, I need to map Odia’s ISO3 code to a valid whisper language code recognized by faster-whisper. It seems I’m confused about whether OpenAI whisper supports Odia with ‘or’ since the accepted codes list doesn’t include it, while others like ‘oc’ are recognized. I need to clarify this discrepancy.


**Clarifying language support for Odia**

I’m considering whether the code for Odia is ‘or’ but might have been removed to avoid confusion. The accepted codes include things like ‘sa’, ‘sd’, ‘si’, and ‘ur’ but not ‘or’, so it seems faster-whisper or whisper doesn’t support Odia after all. Let me check the whisper tokenizer’s language codes, and it confirms there’s no ‘or’. Therefore, I can’t force Odia through the pipeline; I should set the language to None and depend on auto-detection instead. I'll modify the code accordingly.


**Addressing Odia ASR issues**

I’m considering that there could be good ASR models for Odia, possibly through options like IndicWhisper. It seems that the baseline whisper large-v3 may not support Odia, which could be behind the high WER and errors. To fix this, I need to adjust the language mapping to avoid using invalid codes so that auto language detection can work with language set to None. While this might still yield poor transcripts, it will prevent crashes. I’ll also work on updating mappings for other languages like ‘as’, ‘ml’, ‘gu’, and ‘hi’. To solve this, I’ll remove the mapping for "ori"/"ory" leading to "or" in the language definitions. Instead, mapping to None seems a viable option.


**Fixing language mapping for Odia**

I’m thinking about removing unsupported codes from the mapping and handling special cases in the `to_whisper_lang` function. Specifically, if the language code is in {"ori", "ory", "or"}, I’ll return None since whisper doesn't support Odia. For the two-letter code 'or', even if a user passes it, we can still return None. Additionally, I should update `LANG_CODE_TO_NAME` to include Odia. It’s crucial to ensure robust pipeline performance. I’ve noticed that for Sanskrit, we have to clarify how ‘san’ is mapped as well, since it currently returns None. I’ll go ahead and implement these changes in `language.py`.


**Evaluating ASR models for Indian languages**

I’m noticing high WER for languages like Malayalam and Bengali, possibly due to poor model support. Using IndicWhisper could help improve this. The Language Identification (LID) may also be uncertain, particularly with Hindi, often represented as 'hin'. For Assamese, whisper large seems inadequate, resulting in gibberish outputs. The generated output could be in the wrong script, like Hindi Devanagari instead of Assamese script. This suggests a need for a model that supports the correct native script, likely IndicWhisper.

I’ll need to enhance the evaluation function to improve summaries, adapting it to handle language sources correctly. Additionally, I can implement parameters in the evaluation code to allow for better reference language handling. Overall, it sounds like focusing on the right models and improving our efficacy is paramount.


**Updating language mappings for robustness**

I’m working on implementing keys like "lang" and "language" for language handling within the evaluation. My idea is to set the default `lang_used` to `route_lang`, updating it if the `language_source` is from `refs`. If the language source is unknown, I’ll raise an error.

I also want to modify the language mapping, particularly removing "ori" and "ory" since whisper doesn’t support Odia, meaning I’ll make those return None instead. Additionally, I’ll consider adding mappings for Urdu and Sanskrit to help streamline the error management in the pipeline for better transcription accuracy. This will improve our ASR robustness.


Input:
*** Begin Patch
*** Update File: maya_transcribe/language.py
@@
 ISO3_TO_WHISPER_LANG: dict[str, str] = {
     "eng": "en",
     "hin": "hi",
     "mar": "mr",
     "guj": "gu",
     "pan": "pa",
     "tam": "ta",
     "tel": "te",
     "kan": "kn",
     "mal": "ml",
     "ben": "bn",
     "asm": "as",
-    "ori": "or",
-    "ory": "or",
+    "urd": "ur",
+    "san": "sa",
 }
@@
 def to_whisper_lang(lang_code: str | None) -> str | None:
     if not lang_code:
         return None
     lang_code = lang_code.strip().lower()
+    # Whisper/faster-whisper does not expose an Odia language token; let it auto-detect instead of erroring.
+    if lang_code in {"ori", "ory", "or"}:
+        return None
     if len(lang_code) == 2:
         return lang_code
     return ISO3_TO_WHISPER_LANG.get(lang_code, None)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/language.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Examining transcription pipelines**

I’m considering running a separate pipeline to transcribe only Odia segments from a specific audio directory. The challenge here is that the references file contains all languages, and the segment IDs differ depending on the directory structure. If I don’t adjust for this, the references won’t match up. I might generate the necessary references for that specific subset, but it might be easier to transcribe the entire dataset again. Alternatively, I could focus on just the Odia subset while keeping the same root path.


**Exploring pipeline adjustments**

I’m thinking about how to run the pipeline using a specific directory for Odia files while keeping the segment IDs the same. The current pipeline doesn’t support direct language filtering, but I could implement a `--glob` or `--langs` filter. Alternatively, I could create a temporary directory that contains a copy or symlink of the 'ory' folder. If I establish a new structure where the segment paths remain unchanged, the segment IDs should compute similarly since they rely on the relpath and base ID. This could be a useful workaround!


**Planning directory and evaluation**

I’m glad the segment IDs will remain the same! I can create a new directory just for the Odia segments, which would allow running the pipeline effectively. This means the segments will match the references since the relative paths are the same. While I should plan to rerun the entire benchmark with better mapping and more models, it could be beneficial to do a quick run with just 200 Odia segments to check for errors. I also need to tweak the evaluation script to use the language source references and think about improving the model performance.


**Analyzing WER results**

Looking at the Word Error Rate (WER) for different languages, they are quite high, especially with scores like 0.648 for Gujarati and 0.693 for Hindi. It’s concerning that some outputs are as poor as 0.9-1.3. Maybe the issue stems from reference transcripts that include punctuation and compound words. Also, if the ASR output is in the wrong script, like Gurmukhi for Malayalam, that would affect the WER significantly. I need to verify the detected language from the ASR results, as it could provide insight into this misalignment.


**Checking ASR results**

In the ASRResult from faster-whisper, I’m extracting the detected language with `getattr(info, "language", None)`. This is based on what Whisper detects when no specific language is forced. To understand what's happening with the Malayalam segment, I should check the detected language in the ASR summary. I'll query that specific result to see what Whisper has indicated for the language. This could help clarify any issues with the output.


python3 - <<'PY'
import json
from pathlib import Path
seg='Kathbath__844424930310407-814-f.m4a-239d3a2b1121cfe1'
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    if r.get('segment_id')!=seg:
        continue
    print('route', r.get('route'))
    print('asr_summary', r.get('asr_summary'))
    for k,v in (r.get('asr') or {}).items():
        print('asr key', k)
        print('model', v.get('model'))
        print('language', v.get('language'))
        print('text', v.get('text'))
    break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
route {'language_hint': 'mal'}
asr_summary {'faster-whisper:large-v3|det': {'language': 'ml', 'language_probability': 1, 'avg_logprob_mean': -0.3412247396478749, 'no_speech_prob_mean': 0.50390625, 'compression_ratio_mean': 1.3626373626373627}, 'faster-whisper:large-v3|samp_t0.2': {'language': 'ml', 'language_probability': 1, 'avg_logprob_mean': -0.1756986215379503, 'no_speech_prob_mean': 0.50390625, 'compression_ratio_mean': 7.444444444444445}}
asr key faster-whisper:large-v3|det
model faster-whisper:large-v3
language ml
text ਦੀਲਾ ਤਰੀਣਿਲ ਪੂ ਅਫੀਸਰੋ ਵੋਡੇ ਨਰੁਦੇਸ਼ ਪਰਾਗਾਰਮ ਆਨੇ
asr key faster-whisper:large-v3|samp_t0.2
model faster-whisper:large-v3
language ml
text പിസ്റ്റെ ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക്ക


**Evaluating audio output issues**

I’m considering that the audio might be in Malayalam, yet the model seems incompetent, which could lead to random character outputs. The initial sampling showed a high compression ratio and repetitive nonsense, suggesting hallucination. The det pass had a more reasonable ratio but gave text in Gurmukhi, so it’s likely not correct. This might indicate the whisper large-v3 model isn’t suitable for Malayalam, possibly due to audio quality. We could explore using the Indic Whisper model fine-tuned for Malayalam to improve accuracy.


**Searching for a universal model**

I'm thinking that if we could find a single model that supports all languages, it would simplify things. I wonder if such a model exists. I should search Hugging Face for "IndicWhisper" to see what comes up. For instance, `Vignesh-M/Indic-whisper` could be multi-lingual. It makes sense to check the model card for details, so I plan to use HfApi.model_info to gather more information. Let's see what we can discover!


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
mid='Vignesh-M/Indic-whisper'
info=api.model_info(mid)
print('id', info.modelId)
print('tags', info.tags[:20])
print('pipeline_tag', info.pipeline_tag)
print('library', info.library_name)
print('sha', info.sha)
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
id Vignesh-M/Indic-whisper
tags ['transformers', 'pytorch', 'whisper', 'automatic-speech-recognition', 'endpoints_compatible', 'region:us']
pipeline_tag automatic-speech-recognition
library transformers
sha 18c3bb317e54f0f1952ea14545dab2996bde43d0


**Exploring Whisper model options**

I’m looking into using a Whisper model that can be fine-tuned for Indian languages. I want to check the config and determine if it's whisper-small or medium. I should inspect the model card to see what fits best. Per-language models like kavyamanohar/AI4B-Indicwhisper-ml are an option, but I think a multilingual model, like Vistaar's `indic_whisper_hf_id`, might be better. I’ll run some tests with various models alongside faster-whisper and evaluate for improvements.


**Evaluating speech model options**

I'm considering using TransformersWhisperASR for each segment. It works with decode_pcm16 and AutoProcessor, along with AutoModelForSpeechSeq2Seq. This might be slower than faster-whisper but could handle 2400 segments, although it might take around 4 hours. Alternatively, I could baseline it with a CTranslate2 IndicWhisper model for more speed, like `adalat-ai/ct2-AI4B-Indicwhisper-ml`, but that's specific to one language. I should search for a CTranslate2 multilingual model instead. Let’s see what `ct2 IndicWhisper` returns!


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
print('search ct2 Indicwhisper:')
for m in api.list_models(search='ct2 Indicwhisper', limit=20):
    print(m.modelId)
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
search ct2 Indicwhisper:
sawradip/indicwhisper-m-hi-float16-ct2
adalat-ai/ct2-AI4B-Indicwhisper-ml


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
for m in api.list_models(search='AI4B-Indicwhisper', limit=50):
    print(m.modelId)
PY


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:
kavyamanohar/AI4B-Indicwhisper-ml
kavyamanohar/AI4B-Indicwhisper-or
adalat-ai/ct2-AI4B-Indicwhisper-ml


**Considering model adjustments**

I'm thinking about the Malayalam and Odia languages, but it seems like the official models aren't working on HF. Maybe I should try using another model family like `facebook/mms-1b` for CTC decoding, though we haven't implemented decoding yet. The user wants minimal WER, which might be tough. I can at least fix the bug with Odia mapping, run ablations, and let the user choose the right model, but I'm not sure if we can get the WER low enough overall.


**Evaluating model performance**

The evaluation script included almost all languages, but error rows skewed results for Odia; only 57 segments counted when all 200 could. Fixing that may raise the mean WER if it's high. While those segments might have been better off left out, we definitely need significant improvements. Maybe using `facebook/mms-1b-all` for CTC scoring can help us find the best candidates. While selection isn’t the main issue, improving the ASR hypotheses is crucial, so adding more models like IndicWhisper could enhance our results.


**Planning ASR model improvements**

Next, I’m considering adding an extra ASR model, like `Vignesh-M/Indic-whisper`, to potentially improve WER. I think we should test it on a small subset, maybe 50 Hindi segments. The challenge is that we might not have references just for Hindi segments, but our refs file includes everything. If we run the pipeline on the `audio_root/hin` subfolder, we could create a new `segments_dir` while keeping the same segment IDs. However, it might be easier to run the pipeline on the same root with specific files, but since there’s no filter, I could create a new bench corpus to build a sample size.


**Building language-specific audio corpus**

I realized that the build process creates an audio directory structured by language. For Hindi, the output would have a 'hin' folder within the audio_root. So, I can create a smaller bench corpus with around 200 samples and run the pipeline with different model configurations for evaluation. The user requested tests for all languages, but we can iterate on that. It might be beneficial to run benchmarks across the entire dataset with the extra transformer model, potentially using `distil-whisper/large-v3`, although that may take longer than expected.


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
queries=['indicwhisper','indic-whisper','IndicWhisper']
seen=set()
for q in queries:
    for m in api.list_models(search=q, limit=50):
        mid=m.modelId
        if mid in seen:
            continue
        seen.add(mid)
        if 'whisper' in mid.lower() and ('indic' in mid.lower() or 'ai4b' in mid.lower()):
            print(mid)
PY


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:
sawradip/indicwhisper-m-hi-float16-ct2
parthiv11/indic_whisper_hi_multi_gpu
parthiv11/indic_whisper_nodcil
Vignesh-M/Indic-whisper
HariprasathSB/indic-whisper-vulnerable
HariprasathSB/indic-whisper-vulnerable1
HariprasathSB/indic-whisper-vulnerable2
vrclc/indicwhisper-Mal
kavyamanohar/AI4B-Indicwhisper-ml
kavyamanohar/AI4B-Indicwhisper-or
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-8
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-9
kritianandan/indic-whisper-kn-reverse-mft-1-0-1-10
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-11
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-13
kritianandan/indic-whisper-reverse-ml-mft-1-0-1-14
Koosh0610/indic-whisper-reverse-ml-mft-1-0-1-15
adalat-ai/ct2-AI4B-Indicwhisper-ml
parambharat/whisper-tiny-south-indic
parambharat/whisper-base-south-indic
makaveli10/whisper-tiny-Hindi-shrutilipi_indic_mcv
makaveli10/whisper-base-Hindi-shrutilipi_indic_mcv
makaveli10/whisper-tiny-Hindi-shrutilipi_augmented_indic_mcv
makaveli10/whisper-tiny-Hindi-shrutilipi_aug_indic_mcv
makaveli10/whisper-base-Hindi-shrutilipi_augmented_indic_mcv
makaveli10/whisper-base-Hindi-shrutilipi_aug_indic_mcv
makaveli10/whisper-tiny-Hindi-shrutilipi_indic_mcvfull
makaveli10/whisper-tiny-Hindi-shrutilipi_indic_mcv_indic_norm
makaveli10/whisper-medium-Hindi-shrutilipi_indic_mcv
makaveli10/whisper-small-Hindi-shrutilipi_indic_mcv
makaveli10/whisper-small-Hindi-shrutilipi_indic_mcv_run2
makaveli10/whisper-medium-Hindi-shrutilipi_indic_mcv_run3
makaveli10/whisper-medium-Hindi-shrutilipi_indic_mcv_run4
makaveli10/whisper-medium-Hindi-shrutilipi_indic_mcv_run5
makaveli10/whisper-tiny-Hindi-indic-mcv-hi-only-tokenized
makaveli10/whisper-tiny-Hindi-indic-mcv-shrutilipi-hi-only-tokenized
makaveli10/whisper-small-Hindi-indic-mcv-shrutilipi-hi-only-tokenized
makaveli10/whisper-small-Hindi-indic-mcv-shrutilipi-hi-only-tokenized-vocab-size-10k
callmesan/whisper-tiny-south-indic-audio-abuse-feature
makaveli10/whisper-small-Hindi-indic-mcv-shrutilipi-hi-only-tokenized-vocab-size-15k
makaveli10/whisper-small-Hindi-indic-mcv-shrutilipi-hi-only-tokenized-vocab-size-20k
narainp/whisper-indic
callmesan/whisper-indic-audio-abuse-feature
CXDuncan/whisper-large-v3-indic
soketlabs/whisper_indic_hi_ai4bharat
Danyal-AI/Whisper-FT_on_indicVoices
Superleap/faster_indic_whisper_nodcil
jonahdvt/whisper-fleurs-small-indic
jonahdvt/whisper-fleurs-medium-indic
jonahdvt/whisper-fleurs-large-indic
thoshith/whisper-large-v3-turbo-indic
Archit-01/whisper-large-indic-run22
Archit-01/whisper-large-indic-run26
tiantiaf/voxlect-indic-lid-whisper-large-v3
amarnath1906/whisper-large-indic-run28
amarnath1906/whisper-large-indic-run27
amarnath1906/whisper-large-indic-full-v1
amarnath1906/whisper-large-indic-full-v2


Summary: auto


**Choosing models for Indic languages**

I’m considering various models like `CXDuncan/whisper-large-v3-indic`, `jonahdvt/whisper-fleurs-large-indic`, and `thoshith/whisper-large-v3-turbo-indic`. They seem fine-tuned for Indian languages and could yield improvement. I think `jonahdvt/whisper-fleurs-large-indic` might be particularly good since it’s tailored for FLEURS, potentially enhancing our dataset, but I'm unsure. It might be simplest to opt for `CXDuncan/whisper-large-v3-indic`, as its name suggests it’s large-v3 fine-tuned for Indic. I should look into the model information!


python3 - <<'PY'
from huggingface_hub import HfApi
api=HfApi()
for mid in ['CXDuncan/whisper-large-v3-indic','jonahdvt/whisper-fleurs-large-indic','thoshith/whisper-large-v3-turbo-indic','narainp/whisper-indic']:
    try:
        info=api.model_info(mid)
        print('\n',mid)
        print('pipeline', info.pipeline_tag, 'library', info.library_name)
        print('tags', [t for t in info.tags if t in ('whisper','automatic-speech-recognition','ct2','ctranslate2')][:10])
    except Exception as e:
        print(mid, 'ERR', e)
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:

 CXDuncan/whisper-large-v3-indic
pipeline None library None
tags []

 jonahdvt/whisper-fleurs-large-indic
pipeline automatic-speech-recognition library transformers
tags ['whisper', 'automatic-speech-recognition']

 thoshith/whisper-large-v3-turbo-indic
pipeline automatic-speech-recognition library None
tags ['whisper', 'automatic-speech-recognition']

 narainp/whisper-indic
pipeline None library None
tags []


**Evaluating model options**

I'm looking at model info for CXDuncan and wondering if it has a pipeline or library. It might not be a model at all, perhaps just a space or an incomplete dataset. I think it’s a good idea to check its file list on Hugging Face.

I found `jonahdvt/whisper-fleurs-large-indic` that uses transformers and has potential for my needs. It could improve our pipeline, which currently uses faster-whisper large-v3. I will evaluate how to implement it with our existing system!


**Exploring model configurations**

I’m considering setting the whisper_model to large-v3, but it doesn't perform well. A potentially better option would be to use a fine-tuned model via `TransformersWhisperASR` with two hypotheses: one from the transformer model and maybe a base whisper model. This way, Rover can normalize text from both models, combining the best outputs.

Enabling CTC scoring could help pick the best candidate, especially if I use the mms-1b-all model. Alternatively, I could use fine-tuned models compatible with faster-whisper. I'll need to verify what's available!


**Checking model details**

I'm looking at the model `thoshith/whisper-large-v3-turbo-indic`, and it seems like it might be a CTranslate2 model. If that’s the case, I could use it directly as the whisper_model in my pipeline, replacing openai's large-v3. This could lead to significant speed improvements without the additional overhead of transformers.

I need to verify whether `thoshith/whisper-large-v3-turbo-indic` is indeed a CTranslate2 model, so I'll check the repo files next using `huggingface_hub.list_repo_files`.


python3 - <<'PY'
from huggingface_hub import list_repo_files
mid='thoshith/whisper-large-v3-turbo-indic'
files=list_repo_files(mid)
print('nfiles', len(files))
print('\n'.join(files[:50]))
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
nfiles 13
.gitattributes
README.md
added_tokens.json
config.json
generation_config.json
merges.txt
model.safetensors
normalizer.json
preprocessor_config.json
special_tokens_map.json
tokenizer.json
tokenizer_config.json
vocab.json


python3 - <<'PY'
from huggingface_hub import list_repo_files
mid='CXDuncan/whisper-large-v3-indic'
files=list_repo_files(mid)
print('nfiles', len(files))
print('\n'.join(files))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
nfiles 1
.gitattributes


I'm exploring a situation where there might be a problem with an incomplete gitattributes file, or perhaps it requires LFS, which isn't accessible. I think it might be best to skip that for now. Then, I'm wondering about a CTranslate2 model, specifically `Superleap/faster_indic_whisper_nodcil`, so I should take a closer look at it. I want to ensure I'm not missing anything important.


python3 - <<'PY'
from huggingface_hub import list_repo_files
mid='Superleap/faster_indic_whisper_nodcil'
files=list_repo_files(mid)
print('nfiles', len(files))
print('\n'.join(files[:50]))
PY


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:
nfiles 7
.gitattributes
README.md
config.json
model.bin
preprocessor_config.json
tokenizer.json
vocabulary.json


python3 - <<'PY'
from huggingface_hub import hf_hub_download
mid='Superleap/faster_indic_whisper_nodcil'
path=hf_hub_download(mid,'README.md')
print(open(path,encoding='utf-8').read()[:1500])
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:
---
license: mit
---


**Inspecting the README**

I noticed that the README is empty except for the license, which is not very helpful. I’m thinking this model might be a conversion from `parthiv11/indic_whisper_nodcil`, possibly a faster-whisper version. It could also support multiple languages, but I need to check that. The next step is to look at config.json since it might contain language specifications or other useful details, so I’ll plan to download that file.


python3 - <<'PY'
import json
from huggingface_hub import hf_hub_download
mid='Superleap/faster_indic_whisper_nodcil'
path=hf_hub_download(mid,'config.json')
print(json.load(open(path)))
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
{'alignment_heads': [[12, 0], [12, 1], [12, 2], [12, 3], [12, 4], [12, 5], [12, 6], [12, 7], [12, 8], [12, 9], [12, 10], [12, 11], [12, 12], [12, 13], [12, 14], [12, 15], [13, 0], [13, 1], [13, 2], [13, 3], [13, 4], [13, 5], [13, 6], [13, 7], [13, 8], [13, 9], [13, 10], [13, 11], [13, 12], [13, 13], [13, 14], [13, 15], [14, 0], [14, 1], [14, 2], [14, 3], [14, 4], [14, 5], [14, 6], [14, 7], [14, 8], [14, 9], [14, 10], [14, 11], [14, 12], [14, 13], [14, 14], [14, 15], [15, 0], [15, 1], [15, 2], [15, 3], [15, 4], [15, 5], [15, 6], [15, 7], [15, 8], [15, 9], [15, 10], [15, 11], [15, 12], [15, 13], [15, 14], [15, 15], [16, 0], [16, 1], [16, 2], [16, 3], [16, 4], [16, 5], [16, 6], [16, 7], [16, 8], [16, 9], [16, 10], [16, 11], [16, 12], [16, 13], [16, 14], [16, 15], [17, 0], [17, 1], [17, 2], [17, 3], [17, 4], [17, 5], [17, 6], [17, 7], [17, 8], [17, 9], [17, 10], [17, 11], [17, 12], [17, 13], [17, 14], [17, 15], [18, 0], [18, 1], [18, 2], [18, 3], [18, 4], [18, 5], [18, 6], [18, 7], [18, 8], [18, 9], [18, 10], [18, 11], [18, 12], [18, 13], [18, 14], [18, 15], [19, 0], [19, 1], [19, 2], [19, 3], [19, 4], [19, 5], [19, 6], [19, 7], [19, 8], [19, 9], [19, 10], [19, 11], [19, 12], [19, 13], [19, 14], [19, 15], [20, 0], [20, 1], [20, 2], [20, 3], [20, 4], [20, 5], [20, 6], [20, 7], [20, 8], [20, 9], [20, 10], [20, 11], [20, 12], [20, 13], [20, 14], [20, 15], [21, 0], [21, 1], [21, 2], [21, 3], [21, 4], [21, 5], [21, 6], [21, 7], [21, 8], [21, 9], [21, 10], [21, 11], [21, 12], [21, 13], [21, 14], [21, 15], [22, 0], [22, 1], [22, 2], [22, 3], [22, 4], [22, 5], [22, 6], [22, 7], [22, 8], [22, 9], [22, 10], [22, 11], [22, 12], [22, 13], [22, 14], [22, 15], [23, 0], [23, 1], [23, 2], [23, 3], [23, 4], [23, 5], [23, 6], [23, 7], [23, 8], [23, 9], [23, 10], [23, 11], [23, 12], [23, 13], [23, 14], [23, 15]], 'lang_ids': [50259, 50260, 50261, 50262, 50263, 50264, 50265, 50266, 50267, 50268, 50269, 50270, 50271, 50272, 50273, 50274, 50275, 50276, 50277, 50278, 50279, 50280, 50281, 50282, 50283, 50284, 50285, 50286, 50287, 50288, 50289, 50290, 50291, 50292, 50293, 50294, 50295, 50296, 50297, 50298, 50299, 50300, 50301, 50302, 50303, 50304, 50305, 50306, 50307, 50308, 50309, 50310, 50311, 50312, 50313, 50314, 50315, 50316, 50317, 50318, 50319, 50320, 50321, 50322, 50323, 50324, 50325, 50326, 50327, 50328, 50329, 50330, 50331, 50332, 50333, 50334, 50335, 50336, 50337, 50338, 50339, 50340, 50341, 50342, 50343, 50344, 50345, 50346, 50347, 50348, 50349, 50350, 50351, 50352, 50353, 50354, 50355, 50356, 50357], 'suppress_ids': [1, 2, 7, 8, 9, 10, 14, 25, 26, 27, 28, 29, 31, 58, 59, 60, 61, 62, 63, 90, 91, 92, 93, 359, 503, 522, 542, 873, 893, 902, 918, 922, 931, 1350, 1853, 1982, 2460, 2627, 3246, 3253, 3268, 3536, 3846, 3961, 4183, 4667, 6585, 6647, 7273, 9061, 9383, 10428, 10929, 11938, 12033, 12331, 12562, 13793, 14157, 14635, 15265, 15618, 16553, 16604, 18362, 18956, 20075, 21675, 22520, 26130, 26161, 26435, 28279, 29464, 31650, 32302, 32470, 36865, 42863, 47425, 49870, 50254, 50258, 50360, 50361, 50362], 'suppress_ids_begin': [220, 50257]}


**Considering model testing strategies**

I realize we already have 200 segments per language, and running again took a bit over an hour. I’m thinking that it might be worth it but also could take similar time again. Maybe limiting the segments could help. I could use bench_run with `--limit` to handle the first N segments in alphabetical order. For a quick test, 300 segments could work, even if the user wants it per language. Alternatively, I might build a new corpus with fewer samples, like 50 per language. It's complicated!


**Exploring corpus creation options**

I’m thinking about creating a new bench corpus by selecting a subset of existing audio files—specifically the first 50 of each language—without needing to re-download anything. That might speed things up! I could implement a script for efficiency, but it might not be necessary. Instead, I could run the pipeline on the existing corpus and set a limit, like `--limit 600`, just to process the first 600 files. This setup will let me test selected languages. But I'm also considering expanding the limits to include more languages for a comprehensive evaluation.


**Exploring language model improvements**

The user has asked to run all languages, and since we already did a baseline, we can propose some improvements. The main issue seems to be the quality of the base model. I’m considering trying `Superleap/faster_indic_whisper_nodcil` and possibly other fine-tuned models. Running a second run with this model on the full corpus could take around 1 hour and 44 minutes again. While we have the time, the token budget is large, but I'm thinking we can still make it work! Also, we should improve our evaluation setup and implement bench analysis for suggestions.


**Planning evaluation and updates**

I think it's important to implement a fix for the evaluation so that it uses reference languages. Updating the `bench_run` to call with `language_source` references will help provide accurate summaries by language. Our first benchmark already showed WER by reference language, which we can incorporate. Now, I need to ensure that the pipeline handles Odia mapping correctly. Running a smaller pipeline specifically for Odia could help identify any errors. While step 5 of our plan is already running, we may still need to incorporate these fixes and rerun to ensure robustness, especially for that Odia bug. We'll decide on that, and don’t forget about step 6 for creating the private HF benchmark dataset afterward!


**Analyzing run results**

The long run has produced some results, and now I can analyze why the WER is high and propose some tweaks. I’ve already discovered a bug with Odia, which is a good start. I’m starting to wonder if the LID thresholds are too strict, causing some segments to show `route_lang` as None. This could lead to auto detection that isn't accurate. I’ll need to check the proportion of `route_lang` None per language and compute those numbers to get better insights!


python3 - <<'PY'
import json
from collections import Counter, defaultdict

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

none_counts=Counter(); total=Counter(); mismatch=Counter();
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    if 'error' in r:
        continue
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if not ref:
        continue
    lang=ref.get('lang')
    total[lang]+=1
    route=(r.get('route') or {}).get('language_hint')
    if not route:
        none_counts[lang]+=1
    else:
        # normalize odia aliases
        ref_lang=lang
        route_lang=str(route)
        if ref_lang in {'ory','ori'}:
            ref_lang='ory'
        if route_lang in {'ori'}:
            route_lang='ory'
        if route_lang!=ref_lang:
            mismatch[lang]+=1

print('route_none fraction')
for lang in sorted(total):
    n=total[lang]
    print(lang, none_counts[lang], '/', n, f"{none_counts[lang]/n:.2%}", 'mismatch', mismatch[lang], f"{mismatch[lang]/n:.2%}")
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
route_none fraction
asm 21 / 164 12.80% mismatch 1 0.61%
ben 56 / 196 28.57% mismatch 10 5.10%
eng 143 / 171 83.63% mismatch 0 0.00%
guj 42 / 200 21.00% mismatch 1 0.50%
hin 105 / 200 52.50% mismatch 20 10.00%
kan 55 / 200 27.50% mismatch 8 4.00%
mal 49 / 200 24.50% mismatch 13 6.50%
mar 57 / 200 28.50% mismatch 1 0.50%
ory 43 / 57 75.44% mismatch 14 24.56%
pan 51 / 200 25.50% mismatch 3 1.50%
tam 11 / 200 5.50% mismatch 4 2.00%
tel 47 / 200 23.50% mismatch 9 4.50%


Summary: auto


python3 - <<'PY'
import json
from pathlib import Path

refs={}
for line in open('data/bench/maya_hf_v0/refs.jsonl',encoding='utf-8'):
    r=json.loads(line)
    refs[r['segment_id']]=r

count=0
for line in open('outputs/bench/maya_hf_v0/baseline/results.jsonl',encoding='utf-8'):
    r=json.loads(line)
    if 'error' in r:
        continue
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if not ref or ref.get('lang')!='eng':
        continue
    route=(r.get('route') or {}).get('language_hint')
    if route is not None:
        continue
    print('REF', ref['text'][:120])
    print('HYP', (r.get('final') or {}).get('text')[:120])
    print('LID', (r.get('lid') or {}).get('predictions')[:3] if isinstance(r.get('lid'),dict) else None)
    print('---')
    count+=1
    if count>=3:
        break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
REF all nouns alongside the word sie for you always begin with a capital letter even in the middle of a sentence
HYP all nouns alongside the world say for you
LID [{'code': 'eng', 'prob': 0.8402271866798401}, {'code': 'haw', 'prob': 0.10337050259113312}, {'code': 'sco', 'prob': 0.011973042041063309}]
---
REF today the only insects that cannot fold back their wings are dragon flies and mayflies
HYP
LID []
---
REF oliver sacks in his paper the president's speech indicated how people who are unable to understand speech because of bra
HYP
LID []
---


**Evaluating language detection options**

For benchmarking and improving Word Error Rate (WER), I’m considering implementing optional features like "use_ref_lang_hints" or "path_language_hint" to enforce the correct language based on folder names. This could significantly improve WER since whisper struggles with auto-detection. Alternatively, I could lower thresholds for Language ID (LID) to always choose the top language, but that could lead to misrouting. Since the user seeks minimal WER for known languages, perhaps I should ensure top LID is passed, even with uncertainties.


**Adjusting thresholds for language detection**

I think we should tweak thresholds to better handle short audio segments. For segments longer than a certain duration, we could use a high `lid_min_prob`, but for shorter ones, a lower threshold might work better. Even if uncertain, routing could be preferable to not routing at all. I’m considering implementing dynamic LID gating based on segment duration. In our benchmark, I may need to pass a `thresholds_json` with lower values to improve our results.


**Testing smaller benchmarks**

I think it might be a good idea to test on a smaller dataset first, like running a benchmark with a limit of 600 (asm+ben+eng) to check for improvements. I can use a `thresholds_json` with lower thresholds for a quicker test. Even though we’ve done a full run, it could be worthwhile. The user wants a comprehensive analysis, so it’s important to fix any errors in Odia first. After that, I can run the benchmarks again with the new configurations and report the results.


**Implementing language hints**

I can implement a feature to lower LID thresholds through `thresholds_json`, and I also need to fix the Odia mapping bug, which is a step toward robustness. To enhance this further, I’m considering adding a "language hints from folder name" feature, as their dataset may have language-labeled directories, although it wasn't specified. For benchmarking, since we know the languages, we could derive the correct language code from the file path before LID. Using ISO3 codes from directory names could streamline routing and likely reduce WER.


**Preparing to run pipeline again**

In production, segments may not follow a specific structure, but if they do after categorizing by language, it could be beneficial. Given the user's request for a robust and adjustable system, adding a `--path-lang-hint` flag might help. I’ve patched the Odia error, so I need to rerun the benchmark to ensure no errors and get updated WER. I'll create a new directory for just Odia segments, linking them properly, then execute the pipeline with the adjustments. This will confirm everything works and evaluate the new results.


mkdir -p data/bench/maya_hf_v0/audio_only_ory && ln -sf ../audio/ory data/bench/maya_hf_v0/audio_only_ory/ory && ls -la data/bench/maya_hf_v0/audio_only_ory


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 8
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 13 09:25 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 13 09:25 ..
lrwxrwxrwx 1 ubuntu ubuntu   12 Jan 13 09:25 ory -> ../audio/ory


**Executing transcription pipeline**

I've created a symlink, and now I need to run the pipeline on the specified segments directory using the large-v3 Whisper model. I’ll output the results to a specified directory. I’ll run the command for transcription with the parameters necessary, including the limit and possibly setting the language identification backend. Setting a timeout, maybe around an hour, seems reasonable, but I might adjust it to less if needed. I’m ready to proceed with the process!


maya-transcribe run --segments-dir data/bench/maya_hf_v0/audio_only_ory --out outputs/bench/maya_hf_v0/odia_only_largev3 --whisper-model large-v3 --limit 200


Output:
Exit code: 1
Wall time: 0.3 seconds
Output:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/maya-transcribe/maya_transcribe/cli.py:251 in run               │
│                                                                              │
│   248 │   │   thresholds["require_ctc"] = require_ctc                        │
│   249 │   │   thresholds["min_ctc_margin"] = float(min_ctc_margin)           │
│   250 │                                                                      │
│ ❱ 251 │   results_path = run_pipeline(                                       │
│   252 │   │   segments_dir=segments_dir,                                     │
│   253 │   │   out_dir=out,                                                   │
│   254 │   │   lid_model_id=lid_model_id,                                     │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                    ctc_map = None                                        │ │
│ │                  ctc_model = None                                        │ │
│ │              ctc_model_map = None                                        │ │
│ │           gemini_hard_only = True                                        │ │
│ │               gemini_model = None                                        │ │
│ │         gemini_prompt_file = PosixPath('Prompt.md')                      │ │
│ │               lid_model_id = 'facebook/mms-lid-256'                      │ │
│ │                      limit = 200                                         │ │
│ │             min_ctc_margin = 0.05                                        │ │
│ │            min_silence_sec = 0.15                                        │ │
│ │                        out = PosixPath('outputs/bench/maya_hf_v0/odia_o… │ │
│ │                require_ctc = True                                        │ │
│ │        require_lid_backend = True                                        │ │
│ │               segments_dir = PosixPath('data/bench/maya_hf_v0/audio_onl… │ │
│ │       silence_threshold_db = -40.0                                       │ │
│ │                 thresholds = {}                                          │ │
│ │            thresholds_json = None                                        │ │
│ │ transformers_whisper_model = []                                          │ │
│ │               trim_silence = True                                        │ │
│ │              whisper_model = 'large-v3'                                  │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py:77 in run_pipeline  │
│                                                                              │
│    74 │                                                                      │
│    75 │   audio_files = _iter_audio_files(segments_dir)                      │
│    76 │   if not audio_files:                                                │
│ ❱  77 │   │   raise RuntimeError(f"No audio files found under: {segments_dir │
│    78 │   if limit is not None:                                              │
│    79 │   │   audio_files = audio_files[: max(0, int(limit))]                │
│    80                                                                        │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                       audio_files = []                                   │ │
│ │                         ctc_model = None                                 │ │
│ │                     ctc_model_map = None                                 │ │
│ │ extra_transformers_whisper_models = []                                   │ │
│ │                  gemini_hard_only = True                                 │ │
│ │          gemini_max_output_tokens = 512                                  │ │
│ │                      gemini_model = None                                 │ │
│ │                gemini_prompt_path = PosixPath('Prompt.md')               │ │
│ │                      lid_model_id = 'facebook/mms-lid-256'               │ │
│ │                             limit = 200                                  │ │
│ │                   min_silence_sec = 0.15                                 │ │
│ │                           out_dir = PosixPath('outputs/bench/maya_hf_v0… │ │
│ │               require_lid_backend = True                                 │ │
│ │                      segments_dir = PosixPath('data/bench/maya_hf_v0/au… │ │
│ │              silence_threshold_db = -40.0                                │ │
│ │                        thresholds = {}                                   │ │
│ │                      trim_silence = True                                 │ │
│ │                     whisper_model = 'large-v3'                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: No audio files found under: data/bench/maya_hf_v0/audio_only_ory


**Resolving audio file issue**

It looks like the audio files weren't found because the `_iter_audio_files` function uses rglob, which doesn't follow symlinks by default. So, I can’t use the symlink after all. Instead, I’ll copy the audio folder to a new location called `audio_only_ory/ory`. That should be manageable since it’s only around 200 files. I’ll remove the symlink and perform the copy using `cp -a` to keep attributes intact.


rm -f data/bench/maya_hf_v0/audio_only_ory/ory && cp -a data/bench/maya_hf_v0/audio/ory data/bench/maya_hf_v0/audio_only_ory/ory && find data/bench/maya_hf_v0/audio_only_ory -maxdepth 2 -type f | head