**Analyzing encoding issues**

I noticed something interesting about loading the results.jsonl line in Python. The text showed correctly in Devanagari, but earlier it displayed as mojibake when using the `tail` command. This suggests the file has the correct Unicode, yet the `tail` output seems to have encoding issues, possibly from the CLI/harness. Python printed the text correctly though, which means it decodes as UTF-8, while the harness might be handling the `tail` output differently, perhaps as Latin-1.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


set -euo pipefail
# show tail again but through python to avoid encoding issues
python3 - <<'PY'
from pathlib import Path
import json
p=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
lines=p.read_text(encoding='utf-8').splitlines()
print('total lines', len(lines))
for line in lines[-2:]:
    r=json.loads(line)
    txt=((r.get('final') or {}).get('text') or '')
    print(r.get('segment_id'), txt[:80])
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
total lines 860
Kathbath__hindi__valid__198__844424930578479-252-f.m4a-286bad3aca7eb4b2 गुजरात में बच्ची से रेप
Kathbath__hindi__valid__1991__844424931210669-186-m.m4a-29a2fa488a950bc2 भारत में अगले महिने लॉन्च हो सकता है रियल मी लाइप


Summary: auto


set -euo pipefail
rg -n "language_hint_source" -S maya_transcribe | head


Output:
Exit code: 0
Wall time: 0 seconds
Output:
maya_transcribe/pipeline.py:316:                    "route": {"language_hint": route_lang, "language_hint_source": route_lang_source},


**Reviewing pipeline.py changes**

It looks like pipeline.py has some updates! Our earlier `nl -ba` output showed truncated details. Specifically, at line 281, we saw just `{"language_hint": route_lang}`. But now, line 316 indicates an addition of `language_hint_source`. This update suggests changes have been made since we last checked, possibly by another agent. The modification time seems to vary. I think I'll take a closer look at the relevant section around the route_lang logic to confirm!


set -euo pipefail
nl -ba maya_transcribe/pipeline.py | sed -n '130,210p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   130	            prompt = load_prompt_template(gemini_prompt_path or Path("Prompt.md"))
   131	            gemini_backend = GeminiASR(
   132	                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
   133	                prompt_template=prompt,
   134	            )
   135	        except Exception as e:
   136	            raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e
   137
   138	    results_path = out_dir / "results.jsonl"
   139	    summary_path = out_dir / "summary.json"
   140
   141	    counts = {"accept": 0, "human": 0, "reject": 0, "error": 0}
   142	    timings_total: list[float] = []
   143
   144	    with results_path.open("w", encoding="utf-8") as f:
   145	        for src in track(audio_files, description="Transcribing"):
   146	            ident = segment_identity(src, root=segments_dir)
   147	            segment_id = ident.segment_id
   148	            t0 = time.perf_counter()
   149
   150	            try:
   151	                # 1) Preprocess
   152	                t_pre0 = time.perf_counter()
   153	                src_info = ffprobe(src)
   154	                wav_path = out_dir / "audio" / f"{segment_id}.wav"
   155	                normalize_audio(
   156	                    src,
   157	                    wav_path,
   158	                    trim_silence=trim_silence,
   159	                    silence_threshold_db=silence_threshold_db,
   160	                    min_silence_sec=min_silence_sec,
   161	                )
   162	                info = ffprobe(wav_path)
   163	                qc = compute_qc(wav_path)
   164	                t_pre1 = time.perf_counter()
   165
   166	                # 2) LID
   167	                t_lid0 = time.perf_counter()
   168	                lid: LIDResult | None = None
   169	                if lid_backend is not None:
   170	                    lid = lid_backend.predict(wav_path, top_k=5)
   171	                t_lid1 = time.perf_counter()
   172
   173	                # Route language hint (only if confident).
   174	                route_lang = None
   175	                route_lang_source = None
   176	                if lid and lid.predictions:
   177	                    top1 = lid.predictions[0]
   178	                    top2p = lid.predictions[1].prob if len(lid.predictions) > 1 else 0.0
   179	                    if float(top1.prob) >= float(thresholds.get("lid_min_prob", 0.90)) and (
   180	                        float(top1.prob) - float(top2p)
   181	                    ) >= float(thresholds.get("lid_min_margin", 0.20)):
   182	                        route_lang = top1.code
   183	                        route_lang_source = "lid"
   184
   185	                if route_lang is None and bool(thresholds.get("route_lang_from_path", False)):
   186	                    inferred = _infer_lang_from_relpath(ident.relpath)
   187	                    if inferred:
   188	                        route_lang = inferred
   189	                        route_lang_source = "path"
   190
   191	                # 3) ASR ensemble
   192	                t_asr0 = time.perf_counter()
   193	                asr_results: dict[str, ASRResult] = {}
   194
   195	                # Whisper: deterministic + a small sampling diversity pass.
   196	                det = whisper_backend.transcribe(wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0)
   197	                primary_asr_key = f"{det.model}|det"
   198	                asr_results[primary_asr_key] = det
   199	                samp = whisper_backend.transcribe(
   200	                    wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2
   201	                )
   202	                asr_results[f"{samp.model}|samp_t0.2"] = samp
   203
   204	                for tf_backend in transformers_backends:
   205	                    res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
   206	                    asr_results[res.model] = res
   207
   208	                if transformers_router is not None:
   209	                    routed = transformers_router.transcribe(
   210	                        wav_path,


**Evaluating model options**

So, I've seen that the pipeline has been updated to handle a new route language setting. The user doesn’t trust non-official models and prefers standard ones with better coverage. They’re asking about the Vignesh-M model. While it's fine-tuned, it's not official, and its evaluation will help determine its utility. It could be expensive and potentially degrade. I might suggest standard models like `openai/whisper-large-v3` or `distil-whisper`, as they may yield better results. For Indian languages, there are official models like Meta's `facebook/mms-1b-all` for ASR.


**Examining ASR model options**

So, I realized that MMS includes the `facebook/mms-1b-all` model, which is CTC and can be used for ASR, but the pipeline employs CTC scoring rather than generation. The user is looking for offline solutions, and while Whisper may not be optimal for low-resource languages, we could suggest `facebook/mms-1b-all` for transcription. We’ll need to check the WER on the new v2 baseline, as the previous one had an ID mismatch. I’ll compute the partial WER from the current results to evaluate the Vignesh-M model's effectiveness.


**Evaluating ASR hypothesis extraction**

I need to note that the `evaluate` function's `extract_hypothesis_text` may not include `transformers:Vignesh-M/Indic-whisper` because the ASR keys in the results JSON don't match this format. Instead, they use keys like `faster-whisper:large-v3|det`. In the pipeline, ASR results dictionaries have keys like `primary_asr_key`. When calling eval with the hypothesis, I need to ensure I'm using the full key format. Since the user wants to discuss, I might compute evaluation on partial results to analyze per-language performance quickly.


**Computing WER for hypotheses**

Let's proceed to compute the evaluation for the hypotheses: 'final,' 'rover,' 'asr:faster-whisper:large-v3|det,' and 'asr:transformers:Vignesh-M/Indic-whisper.' However, I should note that the partial results may not include all languages—only about three may be processed. I can calculate the WER by averaging for each hypothesis based on these processed segments, which might reveal the effectiveness of the Vignesh model, even with limited samples. There are also warnings regarding transcription behavior with multilingual Whisper, but I don’t think it’s a major issue.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from maya_transcribe.eval import evaluate

results=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')

hyps=[
    'final',
    'rover',
    'asr:faster-whisper:large-v3|det',
    'asr:transformers:Vignesh-M/Indic-whisper',
]

for h in hyps:
    s=evaluate(results, refs, out_path=None, hypothesis=h, oracle=False, language_source='refs')
    print('---', h)
    print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'], 'cer_mean', s['cer_mean'])
    # show a couple languages
    by=s.get('wer_by_language_mean') or {}
    for k in sorted(by)[:6]:
        print(' ',k, by[k])
PY


Output:
Exit code: 0
Wall time: 1 seconds
Output:
--- final
pairs 870 wer_mean 0.8465785158603064 cer_mean 0.6892020806609707
  asm 1.0578857154789767
  ben 0.8953795354305555
  eng 0.9215304733130019
  guj 0.6010849705849705
  hin 0.6071444321444321
--- rover
pairs 870 wer_mean 0.8465785158603064 cer_mean 0.6892020806609707
  asm 1.0578857154789767
  ben 0.8953795354305555
  eng 0.9215304733130019
  guj 0.6010849705849705
  hin 0.6071444321444321
--- asr:faster-whisper:large-v3|det
pairs 870 wer_mean 0.8392867525714433 cer_mean 0.6816572724546356
  asm 1.0552457349884528
  ben 0.8610824832763606
  eng 0.9220567891024756
  guj 0.6062444222444221
  hin 0.6040259290259289
--- asr:transformers:Vignesh-M/Indic-whisper
pairs 870 wer_mean 0.9975660348291288 cer_mean 1.0730285455905253
  asm 1.003814484126984
  ben 1.016864456405273
  eng 0.9651380049199638
  guj 1.0039325951825955
  hin 1.0


**Evaluating model performance**

I'm looking at the results from partial evaluations for various models. The Vignesh model performs significantly worse, indicating it may have issues with mismatched scripts or languages. This supports the user's concerns, but it could still be useful for languages not previously processed. Most languages have a WER around 1.0. Based on these results, I'll communicate that Vignesh-M isn’t helping and that our ROVER pipeline should still maintain quality.


**Analyzing model integration issues**

I’m considering how ROVER might degrade if the Vignesh model introduces different tokens. However, since ROVER picks the majority output between two faster-whisper models and one from Vignesh, the final output should remain consistent with faster-whisper’s results. This means no actual harm, but it could increase decision-making tasks for humans if disagreements rise. Adding a less reliable model doesn’t seem worth it. Regarding superior LLMs, we should evaluate models like Gemini series and their performance across languages, acknowledging their network reliance and potential for hallucination.


**Examining models and capabilities**

I’m not entirely certain about the "Gemma(3n)" model; it seems to be a text LLM from Google and might not support audio. I wonder if "Voxtral" is related to an audio version of Mistral, possibly as an open audio model. As of 2026, there could be an audio model from Mistral, but I can't verify that. I think it's important to be honest about my limitations regarding knowledge of these models and just discuss the topics generally.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from collections import Counter

results_path=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs_path=Path('data/bench/maya_hf_v2/refs.jsonl')

refs={}
for line in refs_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    refs[r['segment_id']]=r

lang_counts=Counter()
dec_counts=Counter()
reason_counts=Counter()

for line in results_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    seg=r.get('segment_id')
    ref=refs.get(seg)
    if ref:
        lang_counts[ref.get('lang')] += 1
    ver=r.get('verification') or {}
    if isinstance(ver, dict):
        dec=ver.get('decision')
        if dec:
            dec_counts[dec]+=1
        for reason in ver.get('reasons') or []:
            reason_counts[reason]+=1

print('processed_by_lang', dict(lang_counts))
print('decision_counts', dict(dec_counts))
print('top_reasons', reason_counts.most_common(10))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
processed_by_lang {'asm': 200, 'ben': 200, 'eng': 200, 'guj': 200, 'hin': 79}
decision_counts {'human': 655, 'reject': 205, 'accept': 15}
top_reasons [('asr_disagreement', 707), ('too_short', 198), ('lid_uncertain', 178), ('lid_missing', 147), ('high_compression_ratio', 60), ('no_speech', 26), ('low_logprob', 1)]


**Checking progress status**

The user asked me to check the current status since it has been running for a while. They've completed about 3 out of 11 or 3 out of 12 languages. I’ll need to report the exact progress, including the count of lines in results.jsonl and the processed languages. I see the lines have gone from 860 to 875, and the run started around 16:23. I might need to check how many lines there are now, possibly around 900.


set -euo pipefail
wc -l < outputs/bench/maya_hf_v2/indic_whisper/results.jsonl


Output:
Exit code: 0
Wall time: 0 seconds
Output:
887


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
err=0
for line in p.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if 'error' in r:
        err += 1
print('total', sum(1 for _ in p.read_text(encoding='utf-8').splitlines()))
print('errors', err)
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
total 889
errors 4


**Evaluating file reading discrepancies**

Hmm, it looks like there's a discrepancy between the total count of 889 and what `wc -l` reports as 887. This might be due to how the file is being read as text and splitting lines, potentially including trailing newlines. It's possible that there's a timing issue when reading the file while it's being written.

The user also asked about achieving optimal performance on WER minimization. I can suggest using reliable models, like OpenAI's Whisper series, including `openai/whisper-large-v2`, for diversity.


set -euo pipefail
curl -s "https://huggingface.co/api/models/openai/whisper-large-v2" | python3 -c 'import sys,json; j=json.load(sys.stdin); print("gated=",j.get("gated"),"pipeline_tag=",j.get("pipeline_tag"))'


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
gated= False pipeline_tag= automatic-speech-recognition


Summary: auto


**Considering trust in models**

The user is asking an important question about trust in models and LLMs. I think I should evaluate the reliability of current models and the factors that contribute to trust. There are various elements at play, like transparency, biases, and how they handle misinformation. I want to explore the user's concerns and provide a thoughtful, nuanced answer that addresses any doubts they may have. It’s crucial to approach this carefully!


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from maya_transcribe.eval import evaluate
results=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')

s=evaluate(results, refs, hypothesis='asr:faster-whisper:large-v3|det', oracle=False, language_source='refs')
print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'])
by=s.get('wer_by_language_mean') or {}
# print in processed languages order
for k in ['asm','ben','eng','guj','hin','kan','mal','mar','ory','pan','tam','tel']:
    if k in by:
        print(k, by[k])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
pairs 894 wer_mean 0.8334037095029051
asm 1.0552457349884528
ben 0.8610824832763606
eng 0.9220567891024756
guj 0.6062444222444221
hin 0.6079730643506152


set -euo pipefail
python3 - <<'PY'
from pathlib import Path
from maya_transcribe.eval import evaluate
results=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')

s=evaluate(results, refs, hypothesis='asr:transformers:Vignesh-M/Indic-whisper', oracle=False, language_source='refs')
print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'])
by=s.get('wer_by_language_mean') or {}
for k in ['asm','ben','eng','guj','hin']:
    if k in by:
        print(k, by[k])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
pairs 895 wer_mean 0.9976340226830637
asm 1.003814484126984
ben 1.016864456405273
eng 0.9651380049199638
guj 1.0039325951825955
hin 1.0


**Evaluating WER metrics**

I need to think about the WER for various languages like Assamese at 1.05, Bengali at 0.861, and English at 0.922. That English WER seems suspiciously high. Maybe it’s due to the FLEURS transcripts having punctuation and capitalization, while our normalization removes those. I wonder if the transcripts aren't matching up with the audio, or if something was trimmed. Since we're evaluating just 200 English samples, the WER indicates our transcripts aren't aligning well with the references.


**Analyzing evaluation normalization**

I’m thinking that our evaluation normalization might be causing issues. For Indic languages, we remove punctuation but still retain the text. The WER calculation should focus on word-level errors, but if the transcript has a different script or whitespace than our hypothesis, that can inflate the WER. It seems there may also be mismatches between the dataset transcripts and audio due to how we've built our bench. Specifically, our method for extracting audio might not align with the Kathbath dataset's structure.


**Clarifying audio extraction**

I’m considering that if the audio_key value is a string, then _extract_audio would return None, meaning we’d skip all examples. Yet, our bench build produced 200 samples per language, which suggests that audio_key must return a dict. I suspect that the Kathbath dataset utilizes an `Audio` feature, where the column name might actually be `audio_filepath` and formatted as a dict. This could explain the discrepancy we’re encountering.


**Evaluating transcript accuracy**

I'm noticing that a WER greater than 0.6 indicates frequent mismatches, possibly due to language issues or errors in the processing pipeline. For Hindi, the route language appears correct. Still, the transcript may be inaccurate, especially if aggressive silence removal cut actual speech, which can be problematic in low amplitude languages. The high WER for English suggests randomness; maybe the audio is too short or not properly scaled. I should check the dataset arrays to understand the format more clearly.


**Analyzing audio feature normalization**

For the HF Audio feature, I'm noting that the "array" is float32 and normalized between -1 and 1, which sounds good. However, I wonder if the Kathbath audio arrays are actually int16 values and not normalized, especially if the dataset stores audio as integers. I'll check the amplitude stats for a sample to ensure they aren't near zero or clipped. The high WER might be linked to differences in evaluating raw versus normalized transcripts since the pipeline normalizes outputs and modifies certain text formats. Punctuation is removed, but that seems acceptable.


**Examining transcript normalization issues**

It seems that the per-language references might still include punctuation, and I'm evaluating how `normalize_text` works on both references and hypotheses, which looks fine. The high WER could be due to dataset transcripts having punctuation while my normalization is too aggressive. For English, we remove punctuation, and I see that the `jiwer.wer` function tokenizes by spaces, which should help. With a WER of 0.92 showing many mismatches, the issue might be alignment between transcripts and audio segments. I need to confirm the bench corpus split name used for Kathbath; it should be split="valid."


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from maya_transcribe.eval import evaluate
results=Path('outputs/bench/maya_hf_v0/baseline/results.jsonl')
refs=Path('data/bench/maya_hf_v0/refs.jsonl')
# compute quickly for asr det (should exist)
try:
    s=evaluate(results, refs, hypothesis='asr:faster-whisper:large-v3|det', oracle=False, language_source='refs')
    print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'])
    print('wer_by_language_mean keys sample', list((s.get('wer_by_language_mean') or {}).keys())[:10])
except Exception as e:
    print('error', e)
PY


Output:
Exit code: 0
Wall time: 0.6 seconds
Output:
pairs 2188 wer_mean 0.847196813139187
wer_by_language_mean keys sample ['asm', 'ben', 'eng', 'guj', 'hin', 'kan', 'mal', 'mar', 'ory', 'pan']


**Evaluating transcription accuracy**

I’m finding that the baseline version 0 of the ASR has a WER mean of 0.8472, and the current version 2 is at 0.833. That's a pretty high WER, which is surprising since IDs matched. It could be related to the evaluation process using normalized text instead of specific tokenization. For languages that typically don’t use spaces, like some Indic scripts, I wonder if space usage impacts WER, even though Hindi and Bengali do use spaces. That WER mean of 0.84 suggests many words are incorrect, and I'm questioning if whisper large-v3 is genuinely struggling with these languages.


**Analyzing WER discrepancies**

Even considering lower resource transcripts, I’m seeing that a WER of 0.3-0.5 is possible, but 0.84 is massive. This could be because datasets like Kathbath may not have verbatim transcripts; they might differ in orthography from whisper outputs, which includes aspects like canonicalization and punctuation. My normalization efforts are limited to removing punctuation and unifying digits, but many Indic orthographic variants aren't addressed. Reaching a WER of 0.84 suggests there must be significant discrepancies at play.


set -euo pipefail
python3 - <<'PY'
from datasets import load_dataset

ds=load_dataset('ai4bharat/Kathbath','hindi',split='valid')
print('len', len(ds))
ex=ds[0]
print(ex.keys())
print('text', ex.get('text')[:80])
print('audio keys', ex.get('audio_filepath').keys() if isinstance(ex.get('audio_filepath'), dict) else type(ex.get('audio_filepath')))
print('audio path', ex.get('audio_filepath').get('path') if isinstance(ex.get('audio_filepath'), dict) else None)
PY


Output:
Exit code: 0
Wall time: 4.6 seconds
Output:
len 3151
dict_keys(['fname', 'text', 'audio_filepath', 'lang', 'duration', 'gender', 'speaker_id'])
text हमने ओबीसी तबके के बच्चों के लिए उच्च शिक्षा में आरक्षण दिया
audio keys dict_keys(['path', 'array', 'sampling_rate'])
audio path 844424930524026-252-f.flac


**Checking transcription accuracy**

The dataset looks normal, so I want to check the Word Error Rate (WER) on a small sample using our pipeline with one audio and its reference to see if Whisper’s performance truly is poor. I'll take the first sample for transcription using faster-whisper large-v3 and compare it with the reference. I’m planning to run the transcription function on that audio file, possibly using the bench file path from refs.jsonl. Let's see how this sample performs!


**Examining audio durations and WER**

I’m looking into audio durations; some might be really short, like 1-3 seconds. This can lead to a WER exceeding 1.0 due to insertion counts, even though the average is 0.84. It could be that our normalization process removes punctuation but doesn’t ensure spaces between words for certain languages. In languages with spaces, like Gujarati and Bengali, this shouldn't happen. I should compute the median WER since, in the baseline evaluation, over half of the segments showed a WER of 1.0, which is quite concerning.


**Examining audio transcription issues**

I’m considering that many audio segments might be low amplitude, causing quiet speech to be trimmed and resulting in missing content. The high word error rate makes me wonder if our WAV writing process has a format mismatch or time scaling issue. I remember that we set the framerate correctly, but the dataset audio arrays might differ in sampling rates. Our pipeline normalizes audio, which should be fine, though I need to double-check the sampling rates for the features.


**Investigating poor transcript quality**

I’m trying to figure out why transcripts for FLEURS are so poor. While the sampling rate seems fine, I wonder if the reference texts might be in a different language than the audio, possibly due to code-switching. Listening directly isn't possible, but I can inspect the transcripts against the audio. Computing a word error rate (WER) showed poor results. It makes me think that while our language mapping seems correct, the model's performance could still be an issue, especially since Whisper usually handles English well.


**Checking language specifications**

I’m considering if we mis-specified the language hint for English. Looking into the processed languages, it seems we might not have set the route language correctly. I should set it to English, and if “to_whisper_lang” maps correctly to 'en', then Whisper should produce high-quality transcriptions. The high WER suggests a mismatch between the reference and hypothesis. I think inspecting one English sample against its reference should help clarify if there's truly a language issue. I'll start by computing the WER for the first English segment.


set -euo pipefail
python3 - <<'PY'
import json, random
from pathlib import Path
refs_path=Path('data/bench/maya_hf_v2/refs.jsonl')
res_path=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')

refs=[]
for line in refs_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if r.get('lang')=='eng':
        refs.append(r)

# map results
res_map={}
for line in res_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    sid=r.get('segment_id')
    if sid:
        res_map[sid]=r

random.seed(0)
for r in random.sample(refs, 5):
    sid=r['segment_id']
    hyp=res_map.get(sid)
    if not hyp:
        continue
    h=((hyp.get('final') or {}).get('text') or '')
    print('---', sid)
    print('ref:', r['text'])
    print('hyp:', h)
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
--- fleurs__en_us__test__2__1950-347909632f55412d
ref: to the north and within easy reach is the romantic and fascinating town of sintra and which was made famous to foreigners after a glowing account of its splendours recorded by lord byron
hyp:
--- fleurs__en_us__test__206__1977-aa76ed322ed40270
ref: prides are made up of one to three related adult males along with as many as thirty females and cubs
hyp:
--- fleurs__en_us__test__636__1785-9df9c7622009f5eb
ref: scaffolding is not a method of learning but rather an aid that provides support to individuals whom are undergoing a new learning experience such as using a new computer program or beginning a new project
hyp: scaffolding is not a method of learning
--- fleurs__en_us__test__549__1898-7b2bacd5bfe5bba3
ref: "he [wales] basically lied to us from the start. first by acting as if this was for legal reasons. second by pretending he was listening to us right up to his art deletion.
hyp: he will
--- fleurs__en_us__test__298__1804-9af6ee015a98712c
ref: the haitian institute for justice and democracy has referenced independent studies that suggest the nepalese un peacekeeping battalion unknowingly brought the disease to haiti
hyp:


Summary: auto


**Considering audio filtering**

I’m thinking about why we’re using `normalize_audio` to convert audio and trim silence. There might be an issue with the `silenceremove` filter being misconfigured, which could result in removing too much audio, especially for FLEURS that may have long pauses. We’ve set the silence threshold at -40 dB, so if there are large sections of audio below -40 dB, it should ideally retain the speech above that threshold. Let’s make sure that’s working correctly!


set -euo pipefail
nl -ba maya_transcribe/audio.py | sed -n '1,220p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	import json
     4	import subprocess
     5	from dataclasses import dataclass
     6	from pathlib import Path
     7
     8	import numpy as np
     9
    10
    11	@dataclass(frozen=True)
    12	class AudioInfo:
    13	    path: str
    14	    duration_sec: float | None
    15	    sample_rate: int | None
    16	    channels: int | None
    17
    18
    19	@dataclass(frozen=True)
    20	class AudioQCMetrics:
    21	    duration_sec: float
    22	    rms_dbfs: float
    23	    peak_dbfs: float
    24	    clipping_fraction: float
    25
    26
    27	def ffprobe(path: Path) -> AudioInfo:
    28	    cmd = [
    29	        "ffprobe",
    30	        "-v",
    31	        "error",
    32	        "-show_entries",
    33	        "format=duration",
    34	        "-show_entries",
    35	        "stream=sample_rate,channels",
    36	        "-of",
    37	        "json",
    38	        str(path),
    39	    ]
    40	    proc = subprocess.run(cmd, check=True, capture_output=True, text=True)
    41	    data = json.loads(proc.stdout or "{}")
    42
    43	    duration_sec = None
    44	    try:
    45	        duration_sec = float(data.get("format", {}).get("duration"))
    46	    except (TypeError, ValueError):
    47	        duration_sec = None
    48
    49	    sample_rate = None
    50	    channels = None
    51	    streams = data.get("streams") or []
    52	    if streams:
    53	        s0 = streams[0] or {}
    54	        try:
    55	            sample_rate = int(s0.get("sample_rate")) if s0.get("sample_rate") else None
    56	        except (TypeError, ValueError):
    57	            sample_rate = None
    58	        try:
    59	            channels = int(s0.get("channels")) if s0.get("channels") else None
    60	        except (TypeError, ValueError):
    61	            channels = None
    62
    63	    return AudioInfo(path=str(path), duration_sec=duration_sec, sample_rate=sample_rate, channels=channels)
    64
    65
    66	def decode_pcm16(path: Path, sample_rate: int = 16000, mono: bool = True) -> tuple[np.ndarray, int]:
    67	    """
    68	    Decode audio with ffmpeg into signed 16-bit PCM (optionally mono) at `sample_rate`.
    69	    Returns (float32 waveform in [-1, 1], sample_rate).
    70	    """
    71	    cmd = ["ffmpeg", "-v", "error", "-i", str(path), "-f", "s16le", "-ar", str(sample_rate)]
    72	    if mono:
    73	        cmd += ["-ac", "1"]
    74	    cmd += ["-"]
    75
    76	    proc = subprocess.run(cmd, check=True, capture_output=True)
    77	    pcm = np.frombuffer(proc.stdout, dtype=np.int16)
    78	    if pcm.size == 0:
    79	        return np.zeros((0,), dtype=np.float32), sample_rate
    80	    wav = pcm.astype(np.float32) / 32768.0
    81	    return wav, sample_rate
    82
    83
    84	def compute_qc(path: Path, sample_rate: int = 16000) -> AudioQCMetrics:
    85	    wav, sr = decode_pcm16(path, sample_rate=sample_rate, mono=True)
    86	    duration_sec = float(wav.shape[0]) / float(sr) if sr else 0.0
    87	    if wav.size == 0:
    88	        return AudioQCMetrics(duration_sec=0.0, rms_dbfs=float("-inf"), peak_dbfs=float("-inf"), clipping_fraction=0.0)
    89
    90	    peak = float(np.max(np.abs(wav)))
    91	    rms = float(np.sqrt(np.mean(np.square(wav))))
    92
    93	    peak_dbfs = 20.0 * float(np.log10(max(peak, 1e-12)))
    94	    rms_dbfs = 20.0 * float(np.log10(max(rms, 1e-12)))
    95
    96	    clipping_threshold = 0.999
    97	    clipping_fraction = float(np.mean(np.abs(wav) >= clipping_threshold))
    98
    99	    return AudioQCMetrics(
   100	        duration_sec=duration_sec,
   101	        rms_dbfs=rms_dbfs,
   102	        peak_dbfs=peak_dbfs,
   103	        clipping_fraction=clipping_fraction,
   104	    )
   105
   106
   107	def normalize_audio(
   108	    in_path: Path,
   109	    out_path: Path,
   110	    sample_rate: int = 16000,
   111	    mono: bool = True,
   112	    trim_silence: bool = True,
   113	    silence_threshold_db: float = -40.0,
   114	    min_silence_sec: float = 0.15,
   115	) -> Path:
   116	    """
   117	    Convert audio to 16kHz mono WAV and optionally trim leading/trailing silence.
   118	    """
   119	    out_path.parent.mkdir(parents=True, exist_ok=True)
   120
   121	    cmd = ["ffmpeg", "-v", "error", "-y", "-i", str(in_path)]
   122
   123	    if trim_silence:
   124	        # Trim leading/trailing silence; tuned for already-clean segments.
   125	        # https://ffmpeg.org/ffmpeg-filters.html#silenceremove
   126	        thr = f"{silence_threshold_db}dB"
   127	        sil = (
   128	            f"silenceremove=start_periods=1:start_duration={min_silence_sec}:start_threshold={thr}:"
   129	            f"stop_periods=1:stop_duration={min_silence_sec}:stop_threshold={thr}"
   130	        )
   131	        cmd += ["-af", sil]
   132
   133	    cmd += ["-ar", str(sample_rate)]
   134	    if mono:
   135	        cmd += ["-ac", "1"]
   136	    cmd += ["-c:a", "pcm_s16le", str(out_path)]
   137
   138	    subprocess.run(cmd, check=True)
   139	    return out_path
   140


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
seg='fleurs__en_us__test__2__1950-347909632f55412d'
for line in Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl').read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if r.get('segment_id')==seg:
        print('found')
        print('qc', r.get('qc'))
        print('lid top', (r.get('lid') or {}).get('predictions', [])[:2])
        asr=r.get('asr') or {}
        for k in ['faster-whisper:large-v3|det','faster-whisper:large-v3|samp_t0.2']:
            if k in asr:
                print(k,'text_len', len(asr[k].get('text') or ''), 'no_speech', (r.get('asr_summary') or {}).get(k,{}).get('no_speech_prob_mean'))
        print('verification', r.get('verification'))
        break
else:
    print('not found')
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
found
qc {'duration_sec': 0.0, 'rms_dbfs': -inf, 'peak_dbfs': -inf, 'clipping_fraction': 0.0}
lid top []
faster-whisper:large-v3|det text_len 0 no_speech None
faster-whisper:large-v3|samp_t0.2 text_len 0 no_speech None
verification {'decision': 'reject', 'reasons': ['too_short', 'lid_missing'], 'scores': {'duration_sec': 0.0, 'clipping_fraction': 0.0, 'rover_disagreement_rate': 0.0, 'lid': {'available': False, 'top1': None, 'top1_prob': None, 'margin': None, 'confident': False}, 'no_speech_prob_max': None, 'heuristics_asr_key': 'faster-whisper:large-v3|det', 'avg_logprob_mean': None, 'compression_ratio_mean': None, 'ctc': None}}


**Investigating audio processing issue**

I'm examining a record that indicates quality control shows a duration of 0.0, which means the audio is empty after preprocessing. This suggests that the normalize_audio function might have produced an empty file, potentially because the silenceremove function trimmed all the audio. That's significant! It could lead to high WER for certain segments if the audio had leading and trailing silence but not enough amplitude. I'm considering whether the silence_threshold_db is set too high, but I’m puzzled as to why the processed output would be empty.


**Checking audio record**

I'm thinking it might be useful to check the src_ffprobe for more information. I want to look for that specific record to understand better what might be going wrong. This could help me figure out if there’s anything unusual in the audio file or its properties. It’s important to gather more details before jumping to conclusions or fixes! Let’s see what src_ffprobe reveals.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
seg='fleurs__en_us__test__2__1950-347909632f55412d'
for line in Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl').read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if r.get('segment_id')==seg:
        print('src_ffprobe', r.get('src_ffprobe'))
        print('ffprobe', r.get('ffprobe'))
        print('src_path', r.get('src_path'))
        print('wav_path', r.get('wav_path'))
        break
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
src_ffprobe {'path': 'data/bench/maya_hf_v2/audio/eng/fleurs__en_us__test__2__1950.wav', 'duration_sec': 11.46, 'sample_rate': 16000, 'channels': 1}
ffprobe {'path': 'outputs/bench/maya_hf_v2/indic_whisper/audio/fleurs__en_us__test__2__1950-347909632f55412d.wav', 'duration_sec': None, 'sample_rate': 16000, 'channels': 1}
src_path data/bench/maya_hf_v2/audio/eng/fleurs__en_us__test__2__1950.wav
wav_path outputs/bench/maya_hf_v2/indic_whisper/audio/fleurs__en_us__test__2__1950-347909632f55412d.wav


**Investigating audio file issue**

I'm looking at an audio file where the src_ffprobe shows a duration of 11.46 seconds, but ffprobe on the output indicates a duration of None. This could mean the output file is corrupted or has an invalid header. The compute_qc function returned a duration of 0.0 due to decode_pcm16 returning empty data. I’m wondering why normalize_audio might produce a corrupted file. ffmpeg runs fine without errors, but ffprobe not parsing duration suggests there could be a problem with the file format, possibly raw PCM without a header.


**Examining WAV file output**

I'm looking at the command where I use `-c:a pcm_s16le` with a `.wav` extension, which should create a proper WAV file with a header. I'm wondering if ffmpeg might default to raw PCM output if the extension isn't recognized. However, since I specified `.wav`, it should work. If the output file is very short or even empty, ffprobe could show a duration of None, indicating a possible issue. It's also possible that silenceremove trimmed everything, leaving a file with a header but zero data. I should check the file size of the output WAV to clarify.


set -euo pipefail
ls -l "outputs/bench/maya_hf_v2/indic_whisper/audio/fleurs__en_us__test__2__1950-347909632f55412d.wav"
python3 - <<'PY'
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/indic_whisper/audio/fleurs__en_us__test__2__1950-347909632f55412d.wav')
print('size', p.stat().st_size)
print('first16', p.read_bytes()[:16])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
-rw-rw-r-- 1 ubuntu ubuntu 78 Jan 13 17:11 outputs/bench/maya_hf_v2/indic_whisper/audio/fleurs__en_us__test__2__1950-347909632f55412d.wav
size 78
first16 b'RIFFF\x00\x00\x00WAVEfmt '


**Investigating audio issues**

I’ve got a file size of only 78 bytes, meaning it’s just headers without any audio data. It seems silenceremove trimmed everything for an 11.46-second speech audio, which is strange. Maybe the threshold of -40 dB is too high given the speech amplitude? The amplitude should exceed -40, unless the audio was scaled down a lot during the bench build. I need to compute the quality check on the source audio file to get more insight.


set -euo pipefail
python3 - <<'PY'
from maya_transcribe.audio import compute_qc, ffprobe
from pathlib import Path
src=Path('data/bench/maya_hf_v2/audio/eng/fleurs__en_us__test__2__1950.wav')
print('ffprobe', ffprobe(src))
print('qc', compute_qc(src))
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
ffprobe AudioInfo(path='data/bench/maya_hf_v2/audio/eng/fleurs__en_us__test__2__1950.wav', duration_sec=11.46, sample_rate=16000, channels=1)
qc AudioQCMetrics(duration_sec=11.46, rms_dbfs=-62.13068501120141, peak_dbfs=-44.37569489396374, clipping_fraction=0.0)


**Analyzing audio amplitude**

It looks like the source audio has an extremely low amplitude, with rms_dbfs at -62 and peak at -44, which is basically silence. If the audio is that quiet, using a silenceremove threshold of -40 would remove everything. I’m curious why the audio is so quiet. It could be a scaling issue from the FLEURS dataset; if the audio array is already in int16 range, converting incorrectly could lead to low amplitude. I’ll need to investigate this further.


**Investigating FLEURS audio**

I’m noting that the HF Audio feature should decode to float32 and be normalized to the range of [-1, 1]. So, why is the FLEURS audio so low in amplitude? I suspect it might be due to how we’re handling the audio; our bench build uses `audio_key="audio"`, and the extraction process expects a dictionary format.

This gives us an array that's converted to float32, which is typical. If the audio values are already scaled in the range of [-32768, 32767], then clipping could saturate the audio instead of keeping it low. But it seems the FLEURS audio arrays may already be normalized, possibly leading to extremely low amplitude, around 0.006.


**Evaluating audio normalization**

I'm calculating audio levels and figuring that -44/20 gives me around 0.0063, which could be plausible if the audio was normalized in a different way. Many audio signals typically range from 0.1 to 0.5 in amplitude—so 0.006 does seem quiet but not overly so. However, an RMS of -62 means an average amplitude of about 0.0008, which is very quiet. This might suggest some mixing issues in averaging the stereo channels incorrectly based on the shape of the audio array.


**Investigating audio amplitude**

I’m wondering about the low amplitude issue with the audio arrays—maybe it’s because I’m treating them as floats in the range of [-1,1] before converting them to int16, which is the right approach. So, why is the amplitude still low? It could be that the FLEURS audio is near silence or maybe we've selected examples with little to no audio. I should inspect the dataset directly, load an example from FLEURS, and compute quality control on its audio array to see the actual levels.


set -euo pipefail
python3 - <<'PY'
from datasets import load_dataset
import numpy as np

ds=load_dataset('google/fleurs','en_us',split='test',trust_remote_code=True)
ex=ds[2]
a=ex['audio']
arr=np.asarray(a['array'], dtype=np.float32)
if arr.ndim==2:
    arr=arr.mean(axis=1)
peak=float(np.max(np.abs(arr)))
rms=float(np.sqrt(np.mean(np.square(arr))))
import math
print('sr', a['sampling_rate'], 'len', arr.shape[0])
print('peak', peak, 'peak_db', 20*math.log10(max(peak,1e-12)))
print('rms', rms, 'rms_db', 20*math.log10(max(rms,1e-12)))
print('text', ex['transcription'][:80])
PY


Output:
Exit code: 0
Wall time: 3.5 seconds
Output:
sr 16000 len 183360
peak 0.006062984466552734 peak_db -44.34627088863755
rms 0.0007914397865533829 rms_db -62.03164242114555
text to the north and within easy reach is the romantic and fascinating town of sintr


Summary: auto


**Evaluating WER factors**

The user didn’t specifically ask about WER, but it’s relevant to minimizing WER and improving pipeline robustness, especially for datasets. A high WER might result from trimming that leads to empty transcripts. In our processed counts for English, we have 200 segments, many of which might be trimmed to empty. I should check how many segments have a duration of zero or below a certain threshold and compute that.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from collections import Counter

path=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs={}
for line in Path('data/bench/maya_hf_v2/refs.jsonl').read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    refs[r['segment_id']]=r

zero=0
zero_by_lang=Counter()
short=0
short_by_lang=Counter()

for line in path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    seg=r.get('segment_id')
    ref=refs.get(seg)
    lang=(ref or {}).get('lang')
    qc=r.get('qc') or {}
    dur=qc.get('duration_sec')
    try:
        dur=float(dur)
    except Exception:
        dur=None
    if dur is None:
        continue
    if dur==0.0:
        zero += 1
        if lang:
            zero_by_lang[lang]+=1
    if dur<0.8:
        short += 1
        if lang:
            short_by_lang[lang]+=1

print('total lines', len(path.read_text(encoding="utf-8").splitlines()))
print('dur==0.0', zero, 'by_lang', dict(zero_by_lang))
print('dur<0.8', short, 'by_lang', dict(short_by_lang))
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
total lines 964
dur==0.0 147 by_lang {'eng': 146, 'guj': 1}
dur<0.8 213 by_lang {'asm': 13, 'ben': 19, 'eng': 155, 'guj': 7, 'hin': 19}


**Identifying trimming issues**

I've found we have 147 segments with a duration of 0.0, mostly English, which means silence trimming eliminated FLEURS' speech. This causes high WER and reasons for rejection like "too short." There are also 213 segments with durations less than 0.8, which suggests that our trimming is affecting evaluation in the pipeline. This could be a critical bug that needs to be fixed. Even if the user didn't ask, we should consider adjusting silence trimming for better WER minimization and a more robust pipeline.


**Considering silence trimming**

I’m thinking about whether to disable the trim-silence option in the CLI bench_run since it's default set to True. To protect FLEURS audio, we should adjust this for the evaluation. Although the user is running a background job right now, I can recommend rerunning with trim_silence set to false or adjusting the threshold to -60 dB later on. We should also consider a dynamic approach for YouTube audio, since volumes can vary. It might be good to link this trimming issue to our broader discussion on model exploration and realistic WER.


**Exploring AI model options**

I’m considering proposing AI4Bharat IndicWhisper as a second candidate since the user mentioned it's battle-tested. However, I couldn't find the official `ai4bharat/IndicWhisper` model on Hugging Face. There are variants like `ai4bharat/indicwav2vec_v1_*`. Perhaps their models aren't on Hugging Face but in another repo. Their other agent, Vignesh-M/Indic-whisper, might be community-supported. If the user wants the official version, AI4Bharat's Vistaar repo could have it. Meanwhile, using OpenAI’s Whisper large-v3 seems like a safe fallback.


**Considering LLM options**

The user is curious about LLMs like Gemma(3n), Voxtral (the audio version of Mistral), or superior models like Gemini 3 Pro, as they cover low-resource languages better. I think it’s worth discussing that audio-native LLMs have improved for ASR tasks, and Gemini 3 Pro could perform well. However, they are cloud-based solutions, which raises concerns about cost and privacy, especially for users looking for offline options.


**Exploring evaluation methods**

I think we should consider a more sophisticated evaluation approach beyond just the mean Word Error Rate (WER). Using metrics like selective WER, rejection rate, and AUROC could refine how we assess error detection. For practical improvements, I propose we fix preprocessing by disabling silent trimming for certain audio cases and implementing a fallback strategy. Additionally, we can use whisper-large-v3 as a baseline for ASR models. These adjustments could help in minimizing WER effectively.


**Considering model improvements**

To enhance performance, I can either add a second model from trusted sources, such as `openai/whisper-large-v2`, or maybe investigate `openai/whisper-large-v3-turbo` if it exists. There’s also the option of incorporating a high-quality CTC model like `facebook/mms-1b-all`. Alternatively, adding a cloud model like Gemini for challenging segments could be useful, though usage would need restrictions.

For evaluation, I plan to run an ablation matrix while checking the progress of my background process, which is currently at 40% completion.


**Assessing segment processing**

I’ve completed the processing for languages like asm, ben, eng, and guj, and now I’m working on hin. However, I encountered a trimming issue that resulted in 146 English segments becoming zero-length, indicating a flaw in the run. This might be wasting time, and I consider suggesting a stop and restart with adjusted thresholds to the user.

They mentioned it's running in the background, so I should inform them about the situation. I’ll also compute how many segments have been processed per language and update those counts once I get the latest numbers.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from collections import Counter

res=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')
ref_map={}
for line in refs.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    ref_map[r['segment_id']]=r

by_lang=Counter()
errors=0
for line in res.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if 'error' in r:
        errors += 1
        continue
    sid=r.get('segment_id')
    lang=(ref_map.get(sid) or {}).get('lang')
    if lang:
        by_lang[lang]+=1

print('total_lines', sum(1 for _ in res.open('r',encoding='utf-8')))
print('errors', errors)
print('processed_by_lang', dict(by_lang))
print('processed_total', sum(by_lang.values()))
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
total_lines 977
errors 4
processed_by_lang {'asm': 200, 'ben': 196, 'eng': 200, 'guj': 200, 'hin': 177}
processed_total 973


**Evaluating trimming effects**

I think we should take a closer look at the impact of trimming silence because the current approach, using trim_silence, is resulting in zero-length outputs. This is likely affecting the Word Error Rate (WER) for speech recognition. To set a realistic baseline, running with trim_silence disabled or adjusting the threshold could lead to significant improvements in WER and potentially other metrics. I’d propose notifying the user about the possibility of rerouting the current run for better benchmarking.


**Considering audio normalization improvements**

I’m thinking about implementing a fallback in the `normalize_audio` function or pipeline. If the `trim_silence` option is used and the output duration is too short or has an invalid quality measure, we could rerun the normalization without trimming or use a lower threshold. This robust approach could be beneficial. The user didn’t explicitly ask for coding, but it might help them, so proposing this to another agent could be a good idea. However, it seems the user's request is more about discussion.


**Discussing process changes**

I’m thinking about the background run that’s happening right now. It seems the user isn't asking for code changes but might appreciate some adjustments to make the pipeline more robust, especially regarding the trimming issue. I wonder if changing the pipeline would interrupt the ongoing process. Since the run started with the current code, any changes wouldn’t affect it unless it reloads. It looks like the user is just discussing and brainstorming rather than requesting implementation.


**Considering code modifications**

I’m realizing that we shouldn't modify the code right now; instead, our focus should be on providing analysis, suggestions, and the current status. If there's a bug related to silence trimming, it might be worth asking the user if they’d like us to patch it after our discussion. I think proactively patching could help with WER minimization and the overall robustness of the pipeline. The user also previously requested feedback on their other coding agent, so I’ll ensure to provide updates on the current run status and findings about the Vignesh model.


**Evaluating model impact**

I’ve concluded that adding this model isn't worthwhile. The bigger issue seems to be silence trimming, which is causing a lot of FLEURS English clips to be wiped out due to low amplitude. It's concerning that 146 out of 200 English clips have turned into 0-second audio. This means that our current benchmark run is likely to produce an artificially poor WER, which isn't ideal. I think we need to address this issue to improve our results.


**Evaluating alternative models**

I’m considering Meta MMS models for CTC scoring as potential alternatives. For LLMs like Gemini 3 Pro, they could improve WER on challenging languages, but they come with costs and are online. I should treat them as optional hypotheses for difficult segments. I also need to discuss whether Gemma 3n and Voxtral are audio models, and if so, treat them similarly.

My practical plan is to evaluate a solid baseline, add one trusted model at a time, and track metrics such as WER to gauge performance.


**Clarifying model scenarios**

I’m realizing that the user might prefer not to interrupt the current run, but I can suggest they consider aborting if they're comfortable. This run is being impacted by the trim-silence process, which is distorting English results and leading to misleading numbers. Since the pipeline is offline, we should focus on open-source offline models. If they’re okay with an online fallback for challenging segments like Gemini, we can explore that.

Additionally, they inquired if our superior LLM bridged the gap in WER from the last run. I should clarify that the earlier WER of 0.958 on the v0 baseline was inflated due to dataset ID collisions and the silence trim threshold affecting quiet audio.


**Analyzing model performance**

It seems we didn’t run properly due to a misconfiguration in the preprocessing for the dataset. With the fix in place, I expect the WER to improve. The user is also asking if the superior model bridged the gap in performance. I’ll have to clarify that the issues we faced affected our results, but once we correct the preprocessing, we should see better outcomes. I'll prepare to explain this so the user understands what to expect going forward.


**Classifying trusted models**

I want to craft a concise yet information-packed answer while aligning with the user's preference for official models. I previously suggested some community fine-tunes, but now I'm leaning towards using reliable ones.

There are trusted offline options like OpenAI Whisper large-v3, which can be efficient, or Meta's SeamlessM4T, an official model that excels in transcription and translation across many languages, including low-resource ones. So, I think proposing SeamlessM4T as a solid alternative to random fine-tunes might be beneficial!


**Considering model options**

I’m looking at `facebook/seamless-m4t-v2-large` as a potential suggestion, but I’m not entirely sure about its accessibility. Integrating SeamlessM4T would mean modifying our ASR backend, which complicates things since we currently support faster-whisper and transformers whisper.

Another option is `openai/whisper-large-v3` or even `microsoft/speech`. When minimizing offline WER, model selection is crucial, but the pipeline may be limited by base model capabilities. It’s okay to test new models, treating them as optional candidates, yet this could raise compute needs or affect gating. I think we should evaluate quickly and make a decision!


**Evaluating audio models**

I’m thinking about audio-first ASR models. Text LLMs like Gemma need a separate speech encoder to transcribe audio directly. If we’re considering audio-native models like Gemini Pro or GPT-4o, they can perform well but may risk paraphrasing or hallucinations. Our approach pushes for verbatim transcription and uses <unk> to manage uncertainty. I’ll also check on Voxtral as a potential model. The user wants to know its status and some practical suggestions for improvement, which I can provide.


**Examining bench run commands**

I’m looking at the bench run command for audio transcription, and it seems to have default parameters that could affect the audio quality. The current setup doesn't include options like `--trim-silence`, which defaults to trimming and could potentially wipe useful audio segments. I think it would be wise to default `trim_silence=False` for certain datasets. However, it might be appropriate to allow trimming for YouTube segments. I want to check how thresholds are set, especially regarding language detection.


**Clarifying thresholds and settings**

Alright, I've realized that the default setting is `False` for `route_lang_from_path` unless specified in the bench run's thresholds. I'm investigating where that happens, and it looks like it’s supposed to load from a file in `cli.bench_run`, but doesn't actually set `route_lang_from_path`. However, if we can see that the path indicates a source, it suggests that the thresholds should be set to `True`. I wonder if `pipeline.run_pipeline` might automatically adjust this if the segment directory has specific naming conventions.


set -euo pipefail
rg -n "route_lang_from_path" -S maya_transcribe | head -n 20


Output:
Exit code: 0
Wall time: 0 seconds
Output:
maya_transcribe/cli.py:392:    thresholds: dict[str, object] = {"route_lang_from_path": True}
maya_transcribe/pipeline.py:185:                if route_lang is None and bool(thresholds.get("route_lang_from_path", False)):


Summary: auto


**Inspecting code thresholds**

I’m noticing that cli.py sets the `route_lang_from_path` threshold to `True` at line 392, and it seems that this might be specifically for `bench_run`. I need to take a closer look at that line and the surrounding code to fully understand its implications. It's interesting to think about how these settings might affect the function's behavior and overall performance. Let's keep digging into that!


set -euo pipefail
nl -ba maya_transcribe/cli.py | sed -n '360,420p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   360	) -> None:
   361	    """
   362	    Run the full pipeline on a local benchmark corpus and write eval.json.
   363	    """
   364	    from datetime import datetime
   365
   366	    from maya_transcribe.eval import evaluate
   367	    from maya_transcribe.pipeline import run_pipeline
   368
   369	    audio_dir = bench_dir / "audio"
   370	    refs_path = bench_dir / "refs.jsonl"
   371	    if not audio_dir.exists():
   372	        raise typer.BadParameter(f"Missing: {audio_dir}")
   373	    if not refs_path.exists():
   374	        raise typer.BadParameter(f"Missing: {refs_path}")
   375
   376	    if out is None:
   377	        ts = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
   378	        out = Path("outputs/bench") / bench_dir.name / f"run_{ts}"
   379
   380	    ctc_map = None
   381	    if ctc_model_map is not None:
   382	        ctc_map = json.loads(ctc_model_map.read_text(encoding="utf-8"))
   383	        if not isinstance(ctc_map, dict):
   384	            raise typer.BadParameter("ctc_model_map must be a JSON object {lang_code: model_id}")
   385
   386	    tf_map = None
   387	    if transformers_whisper_model_map is not None:
   388	        tf_map = json.loads(transformers_whisper_model_map.read_text(encoding="utf-8"))
   389	        if not isinstance(tf_map, dict):
   390	            raise typer.BadParameter("transformers_whisper_model_map must be a JSON object {lang_code: model_id}")
   391
   392	    thresholds: dict[str, object] = {"route_lang_from_path": True}
   393	    if thresholds_json is not None:
   394	        loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
   395	        if not isinstance(loaded, dict):
   396	            raise typer.BadParameter("thresholds_json must be a JSON object")
   397	        thresholds.update(loaded)
   398	    if ctc_model or ctc_map:
   399	        thresholds["require_ctc"] = require_ctc
   400	        thresholds["min_ctc_margin"] = float(min_ctc_margin)
   401
   402	    results_path = run_pipeline(
   403	        segments_dir=audio_dir,
   404	        out_dir=out,
   405	        lid_model_id=lid_model_id,
   406	        require_lid_backend=require_lid_backend,
   407	        whisper_model=whisper_model,
   408	        extra_transformers_whisper_models=transformers_whisper_model,
   409	        transformers_whisper_model_map=tf_map,
   410	        ctc_model=ctc_model,
   411	        ctc_model_map=ctc_map,
   412	        trim_silence=trim_silence,
   413	        silence_threshold_db=silence_threshold_db,
   414	        min_silence_sec=min_silence_sec,
   415	        limit=limit,
   416	        thresholds=thresholds,
   417	    )
   418
   419	    eval_path = out / "eval.json"
   420	    summary = evaluate(


**Evaluating model options**

I see that bench_run uses thresholds and has a parameter for `transformers_whisper_model_map` that may have been updated. The user mentioned they don’t trust non-official models, which is crucial to consider. I can recommend exploring only OpenAI's Whisper large-v3 and large-v2. For a second model, maybe `facebook/seamless-m4t-v2-large` or `facebook/mms-1b-all` for variety. We should focus on minimizing WER for YouTube segments while keeping in mind the user’s mention of superior models like Gemini 3 Pro.


**Exploring model capabilities**

I think for truly low-resource languages, cloud audio-native models might actually outperform open-source options. However, it's important to integrate them as optional fallbacks and verify through CTC scoring and gating. I should only use these when LID is confident, especially if Whisper shows low log probability or high compression.

I'm curious about "Gemma 3n," which I suspect is intended to be Gemini 3 — I need to clarify that text-only models can't perform ASR if they don't support audio input. Additionally, there's Voxtral, which could be Mistral's audio model, but we should check its licensing and availability first!


**Evaluating WER computation**

I’m considering running a pipeline while always evaluating on ASR detection, but I think we already do that. To get an approximation, I could compute WER for English segments, excluding those with zero duration; those transcripts might be empty due to trimming. If we didn’t trim, the WER would likely improve. I want to compute WER for English segments with duration greater than 0.8, but I'll check to see the potential for accuracy.


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from collections import defaultdict
from jiwer import wer
from maya_transcribe.textnorm import normalize_text

results_path=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs_path=Path('data/bench/maya_hf_v2/refs.jsonl')

refs={}
for line in refs_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    refs[r['segment_id']]=r

wers=[]
wers_nonzero=[]

for line in results_path.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    sid=r.get('segment_id')
    ref=refs.get(sid)
    if not ref or ref.get('lang')!='eng':
        continue
    # hypothesis: asr det
    asr=r.get('asr') or {}
    det=asr.get('faster-whisper:large-v3|det') or {}
    hyp=det.get('text') or ''
    ref_text=ref.get('text') or ''
    qc=r.get('qc') or {}
    dur=qc.get('duration_sec')
    try:
        dur=float(dur)
    except Exception:
        dur=None
    ref_n=normalize_text(ref_text, lang='eng')
    hyp_n=normalize_text(hyp, lang='eng')
    w=float(wer(ref_n, hyp_n))
    wers.append(w)
    if dur and dur>0:
        wers_nonzero.append(w)

print('eng segments', len(wers))
print('wer_mean_all', sum(wers)/len(wers) if wers else None)
print('wer_mean_dur>0', sum(wers_nonzero)/len(wers_nonzero) if wers_nonzero else None)
print('count_dur0', len(wers)-len(wers_nonzero))
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
eng segments 200
wer_mean_all 0.9220567891024756
wer_mean_dur>0 0.7113214411202803
count_dur0 146


**Evaluating transcription quality**

Excluding cases with a duration of 0, the mean word error rate (WER) in English is still relatively high at 0.711. This might suggest that many hypotheses are truncated or incomplete due to trimming, leading to inconsistencies. A rerun is necessary, especially since many English segments may have a duration under 0.8. It looks like the audio quality could be improved by normalizing the amplitude, which could significantly enhance transcription accuracy. Right now, we only convert and trim without addressing loudness normalization, but we could consider using filters like `loudnorm` in ffmpeg.


**Improving audio processing**

To minimize the word error rate (WER) in quiet audio, applying loudness normalization to low amplitude FLEURS audio could be a practical improvement. While the user didn’t ask for this, it’s worth mentioning. For robust automatic speech recognition (ASR), bringing the RMS to around -20 dBFS before processing helps to avoid clipping. We can preprocess the audio by converting it to 16k mono WAV, applying normalization, and optionally trimming silence. If looking for a superior model, choosing one with official support, like Gemini Pro or GPT-4o, would be beneficial, especially for online use.


**Considering online fallback options**

If the user can accept online options as a fallback, that could be worth exploring. It’s important to mention that when evaluating the word error rate (WER), a model's capability to produce transcripts in the correct native script is crucial. This might impact the overall accuracy and quality of the transcripts, so it’s something to keep in mind during the selection process. It’s all about making sure we get the best results possible!


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from maya_transcribe.eval import evaluate

results=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')

s=evaluate(results, refs, hypothesis='asr:faster-whisper:large-v3|det', oracle=False, language_source='refs')
print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'])
# show by decision too
print('wer_by_decision_mean', s.get('wer_by_decision_mean'))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
pairs 1006 wer_mean 0.8124858947002888
wer_by_decision_mean {'accept': 0.580308424264946, 'human': 0.7586025695799016, 'reject': 1.0136744855029247}


**Analyzing WER results**

I’ve noticed that the WER mean decreased to 0.812 as more Hindi segments were processed, which might relate to fewer English segments. English segments are still at 200, but since there are now 200 Hindi ones with a WER of about 0.6, the overall mean drops. The run isn’t finished, so I need to carefully craft a response addressing the status, early results, safe model selection recommendations, and a mention of the silence trim issue as a major bug. I should also include a caution about the unreliability of partial WER due to preprocessing.


**Drafting response status**

I’m thinking of crafting a response in bullet format to keep it clear. First, for the **Status:** I want to mention that the file `outputs/bench/maya_hf_v2/indic_whisper/results.jsonl` currently contains around 1000 lines and includes the processed languages. However, there won’t be a `summary.json` until everything is completed. This structure should make it easy for the user to understand where we are at without getting bogged down in details!


**Checking model processing**

I need to verify if the TransformersWhisperASR uses `AutoProcessor.from_pretrained(model_id)` and `AutoModelForSpeechSeq2Seq`. There’s a chance that for some models, the pipeline_tag might be None, but that could still be functional. For the Vignesh model though, the pipeline_tag is set as automatic-speech-recognition, so that part seems fine. I’ll keep this in mind as I move forward with the task.


set -euo pipefail
wc -l < outputs/bench/maya_hf_v2/indic_whisper/results.jsonl
python3 - <<'PY'
import json
from pathlib import Path
from collections import Counter

res=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')
ref_map={}
for line in refs.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    ref_map[r['segment_id']]=r

by_lang=Counter()
errors=0
for line in res.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if 'error' in r:
        errors += 1
        continue
    sid=r.get('segment_id')
    lang=(ref_map.get(sid) or {}).get('lang')
    if lang:
        by_lang[lang]+=1

print('errors', errors)
print('processed_by_lang', dict(by_lang))
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
1016
errors 4
processed_by_lang {'asm': 200, 'ben': 196, 'eng': 200, 'guj': 200, 'hin': 200, 'kan': 16}


set -euo pipefail
python3 - <<'PY'
from pathlib import Path
from maya_transcribe.eval import evaluate
results=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')

for h in ['final','asr:faster-whisper:large-v3|det','asr:transformers:Vignesh-M/Indic-whisper']:
    s=evaluate(results, refs, hypothesis=h, oracle=False, language_source='refs')
    print('---', h)
    print('pairs', s['pairs_evaluated'], 'wer_mean', s['wer_mean'])
PY


Output:
Exit code: 0
Wall time: 0.9 seconds
Output:
--- final
pairs 1014 wer_mean 0.8217366940494726
--- asr:faster-whisper:large-v3|det
pairs 1014 wer_mean 0.8123271576010043
--- asr:transformers:Vignesh-M/Indic-whisper
pairs 1014 wer_mean 0.9979875477103206


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
from collections import Counter

res=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
refs=Path('data/bench/maya_hf_v2/refs.jsonl')
ref_map={}
for line in refs.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    ref_map[r['segment_id']]=r

counts=Counter()
reasons=Counter()
for line in res.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    ver=r.get('verification') or {}
    if isinstance(ver, dict):
        dec=ver.get('decision')
        if dec:
            counts[dec]+=1
        for reason in ver.get('reasons') or []:
            reasons[reason]+=1

print('decision_counts', dict(counts))
print('top_reasons', reasons.most_common(8))

# show zeros by lang
zero_by_lang=Counter()
for line in res.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    sid=r.get('segment_id')
    ref=ref_map.get(sid) or {}
    lang=ref.get('lang')
    qc=r.get('qc') or {}
    dur=qc.get('duration_sec')
    try:
        dur=float(dur)
    except Exception:
        continue
    if dur==0.0 and lang:
        zero_by_lang[lang]+=1
print('dur==0 by lang', dict(zero_by_lang))
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
decision_counts {'human': 777, 'reject': 223, 'accept': 15}
top_reasons [('asr_disagreement', 847), ('lid_uncertain', 255), ('too_short', 215), ('lid_missing', 147), ('high_compression_ratio', 66), ('no_speech', 36), ('low_logprob', 1)]
dur==0 by lang {'eng': 146, 'guj': 1}


Summary: auto


**Considering benchmarking commands**

For benchmarking reruns, I should use `--trim-silence false` or `--silence-threshold-db -70` to avoid deleting low-volume datasets. It’s important to note that this specific run does not include Gemini at all, as it’s not called in the command. Exploring random models might be okay for hypothesis testing but shouldn’t be relied on for production—best to stick with trusted ones. Also, when using LLM audio models, it’s crucial to set the temperature to 0 and follow strict prompts.


**Status**
- Your background run is progressing: `outputs/bench/maya_hf_v2/indic_whisper/results.jsonl` is at ~1016 rows (errors=4); languages done so far are `asm=200`, `eng=200`, `guj=200`, `hin=200`, `ben=196`, and it has started `kan=16`. (`summary.json`/`eval.json` only appear when it finishes.)

**What the data already says**
- The added model `Vignesh-M/Indic-whisper` is not helping on the processed set: `WER(asr:faster-whisper:large-v3|det) ≈ 0.812` vs `WER(asr:transformers:Vignesh-M/Indic-whisper) ≈ 0.998`, and it’s also blowing up routing (`asr_disagreement` is the #1 reason; only 15 segments auto-accepted so far).
- Much bigger benchmark issue: your preprocessing `--trim-silence` is deleting *quiet* FLEURS clips (146 English samples ended up `qc.duration_sec=0.0`), so any WER you see for `eng` (and some `asm`) is currently “pessimistic by construction”.

**Practical safe approach (trusted + WER-minimizing)**
- Offline + reliable: stick to `faster-whisper:large-v3` (optionally add only other “official” families like Meta `seamless-m4t`/`mms` if you want diversity), then use audio-grounded checks (CTC-fit + LID + disagreement) to only auto-accept high-confidence segments.
- “Superior LLM” ASR (Gemini 3 Pro / similar): potentially great for low-resource languages, but it’s online and can hallucinate—treat it as a *fallback hypothesis on hard segments only*, and accept it only when CTC/LID/consensus gates pass.
- For benchmarking reruns: use `maya-transcribe bench run ... --trim-silence false` (or `--silence-threshold-db -70`) so we don’t wipe low-volume datasets like FLEURS.


# Context from my IDE setup:

## Open tabs:
- results.jsonl: outputs/bench/maya_hf_v2/indic_whisper/results.jsonl
- summary.json: outputs/bench/maya_hf_v2/indic_whisper/summary.json
- eval.json: outputs/bench/maya_hf_v2/indic_whisper/eval.json
- results.jsonl: outputs/bench/maya_hf_v2/per_lang_vaani/results.jsonl
- summary.json: outputs/test_run5/summary.json

## My request for Codex:
can you explain the current benchmarks and all experiments done by the other model. what does the WER 0.812 vs WER 0.998 mean. does this mean we are at 81% of what ? can you explain. Also, the other model is trying to perform something from long time, not sure where it;s trying to reach. check the output dir for recent experiments. what about industry standard models on ASR, do they usually get these 0.8 as scores ? or is it because we are doing ensembling we are at such low score, what does it mean. also why are these tests taking so long.


**Explaining benchmarks and WER**

I need to clarify the current benchmarks and experiments related to word error rate (WER). For instance, a WER of 0.812 means 81.2% of words are incorrect, which is extremely poor. I should explain that WER is calculated as (S + D + I)/N, and can exceed 1.0 with numerous insertions. A WER of 0.998 is even worse, indicating almost one error per word. Ideally, a good ASR should have a WER of less than 0.1. Also, the evaluation seems mismatched due to issues like trimmed silence and duplicate segment_ids affecting results.


**Analyzing audio issues and WER**

It seems V2 is correct now, but trimmed silence is still causing zero-length audio issues for English due to low amplitude. Regarding ensembling, it’s not the cause of low scores; in fact, it could help reduce errors. Our verification gate may yield empty transcripts or rejections, leading to high WER since evaluations include all segments. In the run with `Vignesh-M/Indic-whisper`, there are two separate hypotheses evaluated—0.812 for faster-whisper and 0.998 for Indic-whisper. I’ll need to check the output directories for recent experiments and user tabs to gather more information.


**Reviewing experiments and benchmarks**

It looks like there's a per_language_Vaani experiment possibly using new models. I need to check the output directories to identify existing experiments, their settings, and results. The user asked me to explain the current benchmarks and other model experiments. I should gather a list of experiments and summarize metrics like WER. I’ll use a `shell_command` tool to access the data. Additionally, regarding industry standard models on ASR, a WER of 0.8 typically indicates issues like wrong scripting, mapping errors, or evaluation mismatches, which isn't standard.


**Explaining delays and WER**

The tests are taking a while because we’re working with a dataset of 2,400 samples, each lasting 1 to 30 seconds. Each sample goes through multiple processes, including normalization, language identification, and transcription, which can be heavy, especially with the transformers model. The slower speed comes from using PyTorch and beam search. With 2,400 samples, it adds up to hours of processing. I see the user has open tabs for summary and eval files, so I should inspect those results as well as clarify the meaning of WER scores like 0.812 and 0.998.


**Gathering evaluation metrics**

I want to keep the response concise while still providing enough explanation since the user asked multiple questions. I’ll focus on key details, perhaps using bullet points. I need to check the WER values, which may have been computed on a subset, but now they should have a full evaluation available. I'll read the `eval.json` files for relevant metrics like pairs evaluated and mean WER. First, I'll use shell commands to inspect the `outputs/bench/maya_hf_v2/indic_whisper/eval.json` and summary, as well as any eval files for per_lang_vaani. Let's get started with that!


set -euo pipefail
echo "== outputs/bench/maya_hf_v2 =="
ls -lt outputs/bench/maya_hf_v2 | head -n 20

echo
for d in outputs/bench/maya_hf_v2/*; do
  [ -d "$d" ] || continue
  echo "--- $(basename "$d")"
  for f in summary.json eval.json; do
    if [ -f "$d/$f" ]; then
      echo "$f: $(stat -c '%y %s bytes' "$d/$f")"
    fi
  done
  if [ -f "$d/summary.json" ]; then
    python3 - <<PY
import json
from pathlib import Path
p=Path('$d')/'summary.json'
j=json.loads(p.read_text(encoding='utf-8'))
print('segments_total', j.get('segments_total'))
print('counts', j.get('counts'))
print('avg_total_sec', j.get('avg_total_sec'))
PY
  fi
  if [ -f "$d/eval.json" ]; then
    python3 - <<PY
import json
from pathlib import Path
p=Path('$d')/'eval.json'
j=json.loads(p.read_text(encoding='utf-8'))
print('pairs_evaluated', j.get('pairs_evaluated'))
print('wer_mean', j.get('wer_mean'))
print('cer_mean', j.get('cer_mean'))
print('oracle_best_wer_mean', (j.get('oracle') or {}).get('oracle_best_wer_mean'))
print('selection_gap_wer_mean', (j.get('oracle') or {}).get('selection_gap_wer_mean'))
PY
  fi
  echo
 done


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:
== outputs/bench/maya_hf_v2 ==
total 52
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:44 per_lang_vaani_limit250_finalpref
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:43 per_lang_vaani_limit20_finalpref
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:39 per_lang_vaani_limit20_routerpref
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:32 per_lang_vaani_limit20_scriptfilter2
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:29 per_lang_vaani_limit20_scriptfilter
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:22 per_lang_vaani_limit20_newdecode
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 23:03 per_lang_vaani_limit250
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 22:10 per_lang_vaani_v2
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 22:00 per_lang_vaani
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 19:15 indic_whisper
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 16:23 smoke_vaani3
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 16:21 smoke_vaani2
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 16:19 smoke_vaani

--- indic_whisper
summary.json: 2026-01-13 19:15:07.116767371 +0000 236 bytes
eval.json: 2026-01-13 19:15:09.530809958 +0000 1952 bytes
segments_total 2400
counts {'accept': 23, 'human': 1784, 'reject': 403, 'error': 190}
avg_total_sec 4.253917385618375
pairs_evaluated 2210
wer_mean 0.8606466002054953
cer_mean 0.6257349337304581
oracle_best_wer_mean 0.7856508433404377
selection_gap_wer_mean 0.07499575686505668

--- per_lang_vaani
summary.json: 2026-01-13 22:00:17.221911884 +0000 238 bytes
eval.json: 2026-01-13 22:00:17.843922501 +0000 1753 bytes
segments_total 2400
counts {'accept': 172, 'human': 309, 'reject': 211, 'error': 1708}
avg_total_sec 4.784006895194627
pairs_evaluated 692
wer_mean 0.9472045966164909
cer_mean 0.8145643353155686
oracle_best_wer_mean 0.8602210671031469
selection_gap_wer_mean 0.08698352951334401

--- per_lang_vaani_limit20_finalpref
summary.json: 2026-01-13 23:43:28.464274705 +0000 228 bytes
eval.json: 2026-01-13 23:43:28.557276313 +0000 1111 bytes
segments_total 20
counts {'accept': 0, 'human': 19, 'reject': 1, 'error': 0}
avg_total_sec 8.127414167392999
pairs_evaluated 20
wer_mean 0.7959979979171156
cer_mean 0.604361854712759
oracle_best_wer_mean 0.7959979979171156
selection_gap_wer_mean 0.0

--- per_lang_vaani_limit20_newdecode
summary.json: 2026-01-13 23:22:22.660356369 +0000 226 bytes
eval.json: 2026-01-13 23:22:22.740357764 +0000 1110 bytes
segments_total 20
counts {'accept': 0, 'human': 19, 'reject': 1, 'error': 0}
avg_total_sec 8.37206342893187
pairs_evaluated 20
wer_mean 1.1262712418300656
cer_mean 0.9962529251329467
oracle_best_wer_mean 0.7959979979171156
selection_gap_wer_mean 0.3302732439129498

--- per_lang_vaani_limit20_routerpref
summary.json: 2026-01-13 23:39:22.702031943 +0000 225 bytes
eval.json: 2026-01-13 23:39:22.791033476 +0000 1142 bytes
segments_total 20
counts {'accept': 0, 'human': 19, 'reject': 1, 'error': 0}
avg_total_sec 8.090264038648456
pairs_evaluated 20
wer_mean 0.8340232080011492
cer_mean 0.6475969721102806
oracle_best_wer_mean 0.7959979979171156
selection_gap_wer_mean 0.03802521008403362

--- per_lang_vaani_limit20_scriptfilter
summary.json: 2026-01-13 23:29:12.079500066 +0000 226 bytes
eval.json: 2026-01-13 23:29:12.161501497 +0000 1192 bytes
segments_total 20
counts {'accept': 15, 'human': 4, 'reject': 1, 'error': 0}
avg_total_sec 8.504367303987966
pairs_evaluated 20
wer_mean 0.7959979979171156
cer_mean 0.7564283992793037
oracle_best_wer_mean 0.7959979979171156
selection_gap_wer_mean 0.0

--- per_lang_vaani_limit20_scriptfilter2
summary.json: 2026-01-13 23:32:44.793213018 +0000 226 bytes
eval.json: 2026-01-13 23:32:44.883214589 +0000 1125 bytes
segments_total 20
counts {'accept': 0, 'human': 19, 'reject': 1, 'error': 0}
avg_total_sec 8.088198340917007
pairs_evaluated 20
wer_mean 0.9744574175824177
cer_mean 0.7177615882678585
oracle_best_wer_mean 0.7959979979171156
selection_gap_wer_mean 0.17845941966530204

--- per_lang_vaani_limit250
summary.json: 2026-01-13 23:03:53.009062606 +0000 231 bytes
eval.json: 2026-01-13 23:03:53.380069016 +0000 1193 bytes
segments_total 250
counts {'accept': 0, 'human': 232, 'reject': 18, 'error': 0}
avg_total_sec 8.601316591991111
pairs_evaluated 250
wer_mean 0.9812964748151245
cer_mean 0.8422105201989761
oracle_best_wer_mean 0.7537238928734906
selection_gap_wer_mean 0.2275725819416343

--- per_lang_vaani_limit250_finalpref

--- per_lang_vaani_v2

--- smoke_vaani
summary.json: 2026-01-13 16:19:37.715025955 +0000 185 bytes
eval.json: 2026-01-13 16:19:37.770026897 +0000 629 bytes
segments_total 5
counts {'accept': 0, 'human': 0, 'reject': 0, 'error': 5}
avg_total_sec None
pairs_evaluated 0
wer_mean None
cer_mean None
oracle_best_wer_mean None
selection_gap_wer_mean None

--- smoke_vaani2
summary.json: 2026-01-13 16:21:39.726120046 +0000 185 bytes
eval.json: 2026-01-13 16:21:39.780120974 +0000 629 bytes
segments_total 5
counts {'accept': 0, 'human': 0, 'reject': 0, 'error': 5}
avg_total_sec None
pairs_evaluated 0
wer_mean None
cer_mean None
oracle_best_wer_mean None
selection_gap_wer_mean None

--- smoke_vaani3
summary.json: 2026-01-13 16:23:15.068760701 +0000 225 bytes
eval.json: 2026-01-13 16:23:15.131761786 +0000 1061 bytes
segments_total 5
counts {'accept': 0, 'human': 5, 'reject': 0, 'error': 0}
avg_total_sec 10.57227277578786
pairs_evaluated 5
wer_mean 0.8984126984126984
cer_mean 0.8444139194139193
oracle_best_wer_mean 0.7285714285714285
selection_gap_wer_mean 0.16984126984126985


**Calculating WER metrics**

I need to clarify the WER numbers of 0.812 and 0.998, which come from our manual evaluation on different models. Now that we have complete results, we can compute again for ASR candidates. The user asks about these specific values: 0.812 corresponds to the output from the faster-whisper model, while 0.998 is from Indic-whisper. I'll compute WER for both models using the evaluate function on full results, along with per-language WER for clarity.


**Examining per_language Vaani errors**

I’m looking into the per_language Vaani experiment using models from ARTPARK-IISc. I found a lot of errors—1708 total—leading to only 692 evaluated, resulting in a WER of 0.947, which is quite poor. This suggests that the route or model is failing. I need to inspect the error lines for possible issues, like missing model paths or transformer maps. The high error count could be due to missing pipeline tags or configuration differences. Opening the per_lang_vaani directory should help clarify the situation.


**Investigating smoke_vaani experiments**

I’m checking on the smoke_vaani experiments that have some errors, likely from initial attempts. The user mentioned that another model has been running for a while, and I'm curious about its progress. Maybe it's a long benchmark for per_lang_vaani or something else. We can check the process or file modification times to see if it's still running. I'll use the command to check for `maya-transcribe` processes, and inspect the relevant directories for summaries or evaluations to understand what's happening.


**Addressing ASR model performance**

The user asked about industry-standard ASR models and whether scores typically reach around 0.8. I need to clarify that a WER of 0.8 indicates extremely high error rates—standard models usually hit single-digit WERs with clean speech. For low-resource languages, a WER of 20-40% could happen, but not 80%. In our case with English FLEURS, many samples ended up with 0 seconds of audio due to extremely low amplitude, which led to empty transcripts and an artificially inflated WER of 1.0.


**Explaining WER in Kathbath languages**

For the Kathbath languages, normalization issues may arise as our `normalize_text` function removes punctuation but doesn't account for script-specific challenges. In many cases, transcripts may not match audio due to data quality. The high WER suggests that the base ASR model isn't performing well for some languages. To improve accuracy, we could enhance volume normalization or adjust trimming thresholds. It's also important to highlight that WER can yield strange results for languages without spaces; in these cases, CER might be more meaningful. The high WER, particularly in the per_lang_vaani context, indicates a significant number of errors—likely due to model issues.


set -euo pipefail
nl -ba maya_transcribe/pipeline.py | sed -n '208,260p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   208	                    det = whisper_backend.transcribe(
   209	                        wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
   210	                    )
   211	                    primary_asr_key = f"{det.model}|det"
   212	                    asr_results[primary_asr_key] = det
   213	                except Exception as e:
   214	                    asr_errors[f"{whisper_backend.name}|det"] = f"{type(e).__name__}:{e}"
   215
   216	                try:
   217	                    samp = whisper_backend.transcribe(wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2)
   218	                    asr_results[f"{samp.model}|samp_t0.2"] = samp
   219	                except Exception as e:
   220	                    asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"
   221
   222	                for tf_backend in transformers_backends:
   223	                    try:
   224	                        res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
   225	                        asr_results[res.model] = res
   226	                    except Exception as e:
   227	                        asr_errors[tf_backend.name] = f"{type(e).__name__}:{e}"
   228
   229	                if transformers_router is not None:
   230	                    try:
   231	                        routed = transformers_router.transcribe(
   232	                            wav_path,
   233	                            language=route_lang,
   234	                            num_beams=5,
   235	                            max_new_tokens=256,
   236	                        )
   237	                        if routed is not None:
   238	                            asr_results[routed.model] = routed
   239	                            router_key = routed.model
   240	                    except Exception as e:
   241	                        asr_errors[f"transformers_router:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"
   242
   243	                if primary_asr_key is None and asr_results:
   244	                    primary_asr_key = sorted(asr_results.keys())[0]
   245	                if not asr_results:
   246	                    raise RuntimeError(f"no_asr_hypotheses:{asr_errors or 'unknown'}")
   247
   248	                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
   249	                if gemini_backend is not None:
   250	                    run_gemini = True
   251	                    if gemini_hard_only:
   252	                        base_for_heur = det or (next(iter(asr_results.values())) if asr_results else None)
   253	                        det_sum = _asr_summary(base_for_heur) if base_for_heur is not None else {}
   254	                        avg_lp = det_sum.get("avg_logprob_mean")
   255	                        comp = det_sum.get("compression_ratio_mean")
   256	                        no_speech = det_sum.get("no_speech_prob_mean")
   257	                        run_gemini = bool(
   258	                            route_lang is None
   259	                            or (no_speech is not None and float(no_speech) > 0.5)
   260	                            or (avg_lp is not None and float(avg_lp) < float(thresholds.get("min_avg_logprob_mean", -1.0)))


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/per_lang_vaani/results.jsonl')
errs=[]
for line in p.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if 'error' in r:
        errs.append(r)
        if len(errs)>=5:
            break
print('total_lines', sum(1 for _ in p.open('r',encoding='utf-8')))
print('sample_errors:')
for e in errs:
    print(e)
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
total_lines 2400
sample_errors:
{'segment_id': 'Kathbath__bengali__valid__2172__844424931098526-99-m.m4a-d6c0e7b3e93239dd', 'src_path': 'data/bench/maya_hf_v2/audio/ben/Kathbath__bengali__valid__2172__844424931098526-99-m.m4a.wav', 'error': 'no_asr_hypotheses:{\'faster-whisper:large-v3|det\': \'RuntimeError:CUDA failed with error out of memory\', \'faster-whisper:large-v3|samp_t0.2\': \'RuntimeError:CUDA failed with error out of memory\', \'transformers_router:ory\': "ValueError:Unsupported language: or. Language should be one of: [\'en\', \'zh\', \'de\', \'es\', \'ru\', \'ko\', \'fr\', \'ja\', \'pt\', \'tr\', \'pl\', \'ca\', \'nl\', \'ar\', \'sv\', \'it\', \'id\', \'hi\', \'fi\', \'vi\', \'he\', \'uk\', \'el\', \'ms\', \'cs\', \'ro\', \'da\', \'hu\', \'ta\', \'no\', \'th\', \'ur\', \'hr\', \'bg\', \'lt\', \'la\', \'mi\', \'ml\', \'cy\', \'sk\', \'te\', \'fa\', \'lv\', \'bn\', \'sr\', \'az\', \'sl\', \'kn\', \'et\', \'mk\', \'br\', \'eu\', \'is\', \'hy\', \'ne\', \'mn\', \'bs\', \'kk\', \'sq\', \'sw\', \'gl\', \'mr\', \'pa\', \'si\', \'km\', \'sn\', \'yo\', \'so\', \'af\', \'oc\', \'ka\', \'be\', \'tg\', \'sd\', \'gu\', \'am\', \'yi\', \'lo\', \'uz\', \'fo\', \'ht\', \'ps\', \'tk\', \'nn\', \'mt\', \'sa\', \'lb\', \'my\', \'bo\', \'tl\', \'mg\', \'as\', \'tt\', \'haw\', \'ln\', \'ha\', \'ba\', \'jw\', \'su\', \'yue\', \'my\', \'ca\', \'nl\', \'ht\', \'lb\', \'ps\', \'pa\', \'ro\', \'ro\', \'si\', \'es\', \'zh\']."}'}
{'segment_id': 'Kathbath__bengali__valid__239__844424930914707-834-f.m4a-bd5b76dca1565991', 'src_path': 'data/bench/maya_hf_v2/audio/ben/Kathbath__bengali__valid__239__844424930914707-834-f.m4a.wav', 'error': "no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:tel': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 9.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.07 GiB memory in use. Of the allocated memory 14.97 GiB is allocated by PyTorch, and 542.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}"}
{'segment_id': 'Kathbath__bengali__valid__294__844424930981325-428-f.m4a-d33a3e3f9b869c86', 'src_path': 'data/bench/maya_hf_v2/audio/ben/Kathbath__bengali__valid__294__844424930981325-428-f.m4a.wav', 'error': "no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory', 'transformers_router:pan': 'OutOfMemoryError:CUDA out of memory. Tried to allocate 14.00 MiB. GPU 0 has a total capacity of 79.25 GiB of which 5.19 MiB is free. Process 738802 has 60.16 GiB memory in use. Including non-PyTorch memory, this process has 19.08 GiB memory in use. Of the allocated memory 14.97 GiB is allocated by PyTorch, and 544.55 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)'}"}
{'segment_id': 'Kathbath__bengali__valid__39__844424931159642-219-f.m4a-8924dd8215bc633f', 'src_path': 'data/bench/maya_hf_v2/audio/ben/Kathbath__bengali__valid__39__844424931159642-219-f.m4a.wav', 'error': "no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory'}"}
{'segment_id': 'fleurs__en_us__test__114__1845-c51194652086b709', 'src_path': 'data/bench/maya_hf_v2/audio/eng/fleurs__en_us__test__114__1845.wav', 'error': "no_asr_hypotheses:{'faster-whisper:large-v3|det': 'RuntimeError:CUDA failed with error out of memory', 'faster-whisper:large-v3|samp_t0.2': 'RuntimeError:CUDA failed with error out of memory'}"}


Summary: auto


**Investigating errors**

I'm noticing that there's an issue with the indic_whisper run, which shows an error count of 190. I'm wondering if this is due to an out-of-memory (OOM) error or something else entirely. It seems wise to inspect the sample error lines to gather more information. I'll need to look into that closely and parse through the details to figure out what went wrong. Let's get to the bottom of this!


set -euo pipefail
python3 - <<'PY'
import json
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/indic_whisper/results.jsonl')
errs=[]
for line in p.read_text(encoding='utf-8').splitlines():
    r=json.loads(line)
    if 'error' in r:
        errs.append(r)
        if len(errs)>=5:
            break
print('errors_sample_count', len(errs))
for e in errs:
    print(e)
PY