python3 - <<'PY'
from pathlib import Path
path = Path('results.md')
text = path.read_text(encoding='utf-8')
needle = "| overall | 16.7 | 69.1 | 52.5 | 63.6 | 22.3 | 26.5 |"
section = """
### Analysis: Why WER Is High (multilingual 100-sample)
- The WER calculation is standard word-level jiwer on normalized text (NFKC, punctuation stripped, digits normalized; only English lowercased). This is not overly aggressive and matches the reasonable scores seen in IndicConformer/OmniASR/M4T.
- The dominant failure mode is wrong script or wrong language output, which yields near-zero token overlap and WER > 100.
  - Whisper large-v3: Odia outputs are 100% non-Odia script (often Devanagari; sample labeled `language=ne`), Malayalam is ~80% Gurmukhi. Assamese stays in Bengali script but is effectively Bengali orthography; CER is ~51% while WER is ~96%, so word overlap is minimal even though characters look similar.
  - Voxtral: Odia is 100% Devanagari; Assamese is ~54% Devanagari. This aligns with Voxtral's limited language support and fallback to Hindi-style output.
  - Gemma-3n: Punjabi is ~91% Devanagari, Gujarati ~64% Devanagari, Odia ~87% non-Odia script. Outputs are often transliterations or Hindi-leaning paraphrases, not strict native-script transcripts.
- LLM-style models also add or paraphrase content. Gemma's Tamil hypotheses are ~1.46x the reference length on average (10% > 2x), which inflates insertions and pushes WER up even when the script matches.
- Models that preserve the expected script (IndicConformer, OmniASR, M4T) show consistent WERs (roughly 12-31), which suggests the evaluator is pairing refs/hyps correctly.

Suggested follow-ups: report CER alongside WER for Bengali-script languages (Assamese/Bengali), and consider transliteration-normalized WER for script-mismatch cases. For Whisper, avoid auto-detect where possible (or explicitly set supported language tokens); for Voxtral/Gemma, enforce supported language/script or post-process via script conversion when benchmarking.
""".strip("\n")

if section in text:
    print('Section already present; no changes made.')
    raise SystemExit(0)

if needle not in text:
    raise SystemExit('Needle not found in results.md')

text = text.replace(needle, needle + "\n\n" + section)
path.write_text(text, encoding='utf-8')
print('Updated results.md')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
Updated results.md


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Checking snippet insertion**


Output:
Exit code: 0
Wall time: 0 seconds
Output:


Summary: auto


**Identifying section location**

I'm checking line numbers for the "Analysis: Why WER" section to display context around the inserted area accurately.


grep -n "Analysis: Why WER" -n results.md


Output:
Exit code: 0
Wall time: 0 seconds
Output:
119:### Analysis: Why WER Is High (multilingual 100-sample)


Output:
Exit code: 0
Wall time: 0 seconds
Output:
### Per-language WER (mean, %)
Source: `eval.json` from each run. Note: IndicConformer HF failed on English (`n/a` below), so its overall WER is computed on 1100 segments.

| Lang | IndicConformer-600M (HF) | Whisper large-v3 | Voxtral-Small | Gemma-3n-E2B | OmniASR LLM 7B v2 | Seamless M4T v2 large |
| --- | --- | --- | --- | --- | --- | --- |
| asm | 19.9 | 95.9 | 107.9 | 78.0 | 21.8 | 29.9 |
| ben | 11.2 | 64.2 | 37.4 | 34.8 | 13.9 | 17.5 |
| eng | n/a | 4.5 | 3.8 | 9.2 | 6.0 | 7.7 |
| guj | 11.7 | 49.3 | 49.8 | 87.3 | 16.3 | 19.4 |
| hin | 9.2 | 30.9 | 14.9 | 18.8 | 12.6 | 25.4 |
| kan | 14.3 | 71.7 | 49.0 | 45.8 | 20.4 | 30.5 |
| mal | 29.9 | 122.5 | 62.6 | 65.1 | 41.2 | 42.6 |
| mar | 14.6 | 79.7 | 39.9 | 38.8 | 25.1 | 24.1 |
| ory | 16.3 | 115.5 | 130.3 | 120.0 | 29.8 | 31.5 |
| pan | 12.6 | 64.7 | 34.2 | 135.0 | 16.0 | 17.4 |
| tam | 20.1 | 50.9 | 52.7 | 78.8 | 32.9 | 31.5 |
| tel | 24.4 | 79.0 | 47.5 | 51.2 | 31.6 | 40.1 |
| overall | 16.7 | 69.1 | 52.5 | 63.6 | 22.3 | 26.5 |

### Analysis: Why WER Is High (multilingual 100-sample)
- The WER calculation is standard word-level jiwer on normalized text (NFKC, punctuation stripped, digits normalized; only English lowercased). This is not overly aggressive and matches the reasonable scores seen in IndicConformer/OmniASR/M4T.
- The dominant failure mode is wrong script or wrong language output, which yields near-zero token overlap and WER > 100.
  - Whisper large-v3: Odia outputs are 100% non-Odia script (often Devanagari; sample labeled `language=ne`), Malayalam is ~80% Gurmukhi. Assamese stays in Bengali script but is effectively Bengali orthography; CER is ~51% while WER is ~96%, so word overlap is minimal even though characters look similar.
  - Voxtral: Odia is 100% Devanagari; Assamese is ~54% Devanagari. This aligns with Voxtral's limited language support and fallback to Hindi-style output.
  - Gemma-3n: Punjabi is ~91% Devanagari, Gujarati ~64% Devanagari, Odia ~87% non-Odia script. Outputs are often transliterations or Hindi-leaning paraphrases, not strict native-script transcripts.
- LLM-style models also add or paraphrase content. Gemma's Tamil hypotheses are ~1.46x the reference length on average (10% > 2x), which inflates insertions and pushes WER up even when the script matches.
- Models that preserve the expected script (IndicConformer, OmniASR, M4T) show consistent WERs (roughly 12-31), which suggests the evaluator is pairing refs/hyps correctly.

Suggested follow-ups: report CER alongside WER for Bengali-script languages (Assamese/Bengali), and consider transliteration-normalized WER for script-mismatch cases. For Whisper, avoid auto-detect where possible (or explicitly set supported language tokens); for Voxtral/Gemma, enforce supported language/script or post-process via script conversion when benchmarking.


**Confirming nested bullet acceptability**

Nested bullets appear in the inserted section, but since style rules apply only to the final answer, using them in internal files like results.md is acceptable and matches existing format.


**Documenting multilingual WER analysis**

Added a new analysis section to `results.md` explaining multilingual WER spikes as script/language mismatches and LLM verbosity rather than evaluator errors.

Details:
- Notes Whisper's auto-detect fallbacks (Odia→Devanagari, Malayalam→Gurmukhi) and Assamese vs Bengali orthography gap (CER vs WER), matching sample comparisons.
- Calls out Voxtral/Gemma fallback to Devanagari for unsupported languages (Odia, Punjabi, Gujarati) and Gemma's longer Tamil outputs inflating insertions.
- Concludes evaluator pairing looks sound since script-correct models (IndicConformer/OmniASR/M4T) have reasonable WER. Updated: `results.md`.

Next steps:
1. Add transliteration-normalized WER/CER reporting for script-mismatch cases.
2. Force language tokens for Whisper where supported and log mismatches in results.
3. Add a small script to flag script-dominant mismatches per run for quick sanity checks.


Added an analysis section to `results.md` explaining the multilingual WER spikes as script/language mismatches and LLM verbosity rather than evaluator errors.

Details:
- Notes Whisper’s auto-detect fallbacks (Odia to Devanagari, Malayalam to Gurmukhi) and Assamese vs Bengali orthography (high WER, lower CER) based on the sample comparisons.
- Calls out Voxtral/Gemma fallback to Devanagari on unsupported languages (Odia, Punjabi, Gujarati) and Gemma’s longer Tamil outputs inflating insertions.
- Concludes evaluator pairing looks sane because script-correct models (IndicConformer/OmniASR/M4T) stay in the 12–31 WER range.

Updated: `results.md`.

Next steps:
1. Add transliteration-normalized WER/CER reporting for script-mismatch cases.
2. Force language tokens for Whisper where supported and log mismatches in results.
3. Add a small script to flag script-dominant mismatches per run.


# Context from my IDE setup:

## Open tabs:
- results.md: results.md
- .env: .env
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json
- asr.py: maya_transcribe/asr.py
- Prompt.md: Prompt.md

## My request for Codex:
How can we fix this. i understand the limitations of LLM based transcriptions being more creative. Any prompt guidance, or adherance to pure transcriptions with temperature being 0.0 or something. how can we fix this.

- LLM-style models also add or paraphrase content. Gemma's Tamil hypotheses are ~1.46x the reference length on average (10% > 2x), which inflates insertions and pushes WER up even when the script matches.


**Suggesting LLM audio model decoding strategy**

Provide a strict prompt enforcing verbatim output in the original script with numeral preservation, apply forced language tokens or system prompts, and use length-based guards plus EOS for output control. Incorporate constrained decoding leveraging CTC/ASR anchors with fallback to a baseline ASR for robustness, and implement per-language script filters to reject or transliterate mismatches. Add repetition penalties and no-repeat n-gram constraints to reduce creativity, and evaluate with WER variants focused on preserving literal transcription.


**Recommending prompt, decoding, and gating refinements**

To reduce creative rewrites in LLM transcription, combine strict prompt instructions with controlled decoding and output validation. Use a verbatim transcription prompt that forbids additions, enforce greedy decoding with low temperature and repetition penalties, and impose length and script consistency checks post-generation with fallback to a baseline ASR if thresholds are exceeded.

Example prompt snippet for strict verbatim output:
```
You are an ASR system. Transcribe the audio verbatim in <LANGUAGE> script. Rules:
- Do not translate, summarize, or paraphrase.
- Do not add words not in audio.
- Preserve numbers and punctuation exactly.
- If unsure, output [inaudible].
Output only the transcript.
```

Key suggestions:
- Use decoding parameters like temperature=0, top_p=1, top_k=1 (greedy), repetition_penalty 1.1, no_repeat_ngram_size=3, and max_tokens tied to audio duration (e.g., max_tokens = min(int(duration_sec × 6) + 16, 512)).
- After generation, verify script match and length ratio (e.g., reject outputs if length differs >30% from expected or script mismatch is detected).
- Fall back to baseline ASR (e.g., Conformer or Whisper) if constraints fail or language unsupported.
- Optionally implement a two-pass approach where the LLM proofreads a baseline transcript with restricted editing (e.g., only fix clear errors within a small edit distance).
- Use CTC alignment scoring or lexicon constraints in the pipeline for output validation (refer to `maya_transcribe/ctc_fit.py` or `maya_transcribe/pipeline.py`).
- Avoid encouraging LLM expansion by strict prompt and decoding settings; add stop tokens and consider voice activity detection to trim silence edges.

Next steps:
1. Adopt strict verbatim prompt and decoding constraints in your pipeline for LLM transcription.
2. Implement post-generation gating for script consistency and output length in `maya_transcribe/pipeline.py` or `verify.py`.
3. Integrate fallback logic to baseline ASR when gating flags rerun or discard LLM output.
4. Optionally test two-pass proofreading with limited edit thresholds to balance correction and paraphrasing.
5. Evaluate improvements with relevant error metrics (CER, WER) focusing on literal transcription accuracy.

If desired, I can help implement gating and fallback mechanisms targeting specific models.


Temperature=0 is necessary but not sufficient for LLM ASR. The fastest wins come from hard constraints plus post‑checks, not just prompt tuning.

What to change:
- Tighten the prompt to “verbatim, same script, no paraphrase” and give a safe failure token. Example:
```text
You are an ASR system. Transcribe the audio verbatim in <LANGUAGE> script.
Rules:
- Do not translate, summarize, or paraphrase.
- Do not add or omit words.
- Preserve numbers and casing as spoken.
- If unsure, output [inaudible] for that word.
Output only the transcript.
```
- Constrain decoding harder: greedy (`temperature=0`, `top_p=1`, `top_k=1` if supported), small `repetition_penalty` (1.05–1.15), `no_repeat_ngram_size=3`, and a tighter length cap (e.g., `max_tokens = min(int(duration_sec * 5) + 16, 512)` instead of 12 tokens/sec). Avoid `min_tokens` which can force hallucinations.
- Add a post‑generation guard: reject or fall back if the output script is wrong or if length ratio is too large. A simple rule like “expected script >= 60% of alphabetic chars AND 0.7x–1.3x length” filters most creative drift.
- Use an anchor transcript: run a stable ASR first, then LLM as a proofreader. Accept LLM output only if edit distance is small (e.g., <= 15–20%) and script matches; otherwise keep the anchor. This avoids the “creative rewrite” failure mode.
- If the model doesn’t truly support a language/script, don’t force it. Route to a model that supports the script, or transliterate as a separate, explicit step. Prompting won’t fix missing language support.

If you want, I can implement a guard + fallback in `maya_transcribe/pipeline.py` and tighten decode limits for the LLM path. Let me know which models you want gated and what thresholds you prefer.

Next steps:
1. Pick the target model(s) to constrain and whether to add a script/length guard.
2. Decide the fallback (best ASR model per language or a single default).
3. Run a small 50‑sample A/B to validate WER impact before full rerun.