Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
## Benchmarks
## Table of contents
## Resources
### Download Training Datasets and Benchmarks
### Download Models
## Tutorials
### Setting up your environment
### Evaluating ASR models
### Inference using IndicWhisper
### Training on Vistaar Train Datasets
## License
## Contributors
## Contact
## Acknowledgements

--- Benchmarks section snippet ---
# Vistaar: Diverse Benchmarks and Training sets for Indian Language ASR

 Vistaar is a set of 59 benchmarks and training datasets across various language and domain combinations such as news, education, literature, tourism etc. The training datasets are avaialable for 12 Indian languages amounting to over 10,700 hours of labelled audio data. We also train IndicWhisper models by fine-tuning the Whisper models on the Vistaar train dataset and observe that it has the lowest WER on 39 out of 59 Vistaar benchmarks.

## Benchmarks
Vistaar consists of benchmarks from several public datasets - Kathbath, FLEURS, CommonVoice, IndicTTS, MUCS, GramVaani across 12 languages. We evaluate IndicWhisper on these benchmarks along with 3 publicly available ASR systems and 2 commercially available systems. Below mentioned are the results

| Datasets      | bn       | gu       | hi       | kn       | ml       | mr       | or       | pa       | sa   | ta       | te       | ur       | avg   |
|---------------|----------|----------|----------|----------|----------|----------|----------|----------|------|----------|----------|----------|-------|
| Kathbath      | 16.6     | 17.8     | **10.3** | **19.3** | **34.8** | **19.9** | 24.7     | 16.9     | 45.6 | **24.2** | 25       | **11.9** | 22.3  |
| Kathbath Hard | 19.4     | **20.6** | **12.0** | **22.2** | **38.4** | **22.1** | 29.1     | 19.7     | 50.5 | **27.5** | 27.8     | **14.7** | 25.3  |
| CommonVoice   | 24.7     | -        | **11.4** | -        | **44.5** | 22.8     | **35.2** | 22.4     | -    | **29.2** | -        | 31.7     | 27.8  |
| FLEURS        | **20.9** | 23.5     | **15.0** | **18.6** | **22.6** | **20.5** | **32.9** | **23.1** | -    | **25.2** | **25.4** | **19.2** | 22.5  |
| IndicTTS      | **18.8** | **19.1** | **7.6**  | **13.2** | **21.4** | **11.4** | **15.0** | -        | -    | **17.2** | 33.8     | -        | 17.5  |
| MUCS          | -        | 33.2     | **12.0** | -        | -        | **12.8** | **27.5** | -        | -    | 28.3     | 32.1     | -        | 24.3  |
| Gramvaani     | -        | -        | **26.8** | -        | -        | -        | -        | -        | -    | -        | -        | -        | 26.8  |
| Average       | 20.1     | 22.8     | 13.6     | 18.3     | 32.3     | 18.2     | 27.4     | 20.5     | 48   | 25.3     | 28.8     | 19.4     | 24.6  |

*Word error rates (%) of IndicWhisper on the Vistaar Benchmark. Values in bold indicates benchmarks where IndicWhisper has the lowest WER.*

| Model         | Kathbath | Kathbath-Hard | FLEURS   | CommonVoice | IndicTTS | MUCS         | Gramvaani | Average   |
|---------------|----------|---------------|----------|-------------|----------|--------------|-----------|-----------|
| Google STT    | 14.3     | 16.7          | 19.4     | 20.8        | 18.3     | 17.8         | 59.9      | 23.9      |
| IndicWav2vec  | 12.2     | 16.2          | 18.3     | 20.2        | 15       | 22.9         | 42.1      | 21        |
| Azure STT     | 13.6     | 15.1          | 24.3     | 14.6        | 15.2     | 15.1         | 42.3      | 20        |
| Nvidia-medium | 14       | 15.6          | 19.4     | 20.4        | 12.3     | 12.4         | 41.3      | 19.4      |
| Nvidia-large  | 12.7     | 14.2          | 15.7     | 21.2        | 12.2     | **11.8**     | 42.6      | 18.6      |
| IndicWhisper  | **10.3** | **12.0**      | **11.4** | **15.0**    | **7.6**  | 12           | **26.8**  | **13.6**  |

*Comparison of publicly-available models on the Hindi subset of the Vistaar benchmark*

## Table of contents
- [Vistaar](#vistaar-diverse-benchmarks-and-training-sets-for-indian-language-asr)
  - [Benchmarks](#benchmarks)
  - [Table of contents](#table-of-contents)
  - [Resources](#resources)
    - [Download Training Datasets and Benchmarks](#download-training-datasets-and-benchmarks)
    - [Download Models](#download-models)
  - [Tutorials](#tutorials)
    - [Setting up your environment](#setting-up-your-environment)
    - [Evaluating ASR models](#evaluating-asr-models)
      - [Manifest creation](#manifest-creation)
      - [Running evaluation](#running-evaluation)
    - [Inference using IndicWhisper](#inference-using-indicwhisper)
      - [Sample structure of manifest file](#sample-structure-of-manifest-file)
      - [Running inference](#running-infernece)
    - [Training on Vistaar Train Datasets](#training-on-vistaar-train-datasets)
      - [Manifest creation](#manifest-creation)
      - [Running training](#running-training)
  - [Cite](#cite)
  - [License](#license)
  - [Contributors](#contributors)
  - [Contact](#contact)
## Resources

### Download Training Datasets and Benchmarks
|Datasets | Training Datasets | Benchmarks |
| - | - | - |
| Kathbath | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/kathbath.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/kathbath.zip) |
| Kathbath Hard | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/kathbath_noisy.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/kathbath_noisy.zip) |
| CommonVoice | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/commonvoice.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/commonvoice.zip) |
| FLEURS | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/fleurs.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/fleurs.zip) |
| IndicTTS | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/indictts.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/indictts.zip) |
| MUCS | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/mucs.zip) | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/mucs.zip) |
| gramvaani | [link](https://indicwhisper.objectstore.e2enetworks.net/vistaar/gramvaani.zip) | [link]((https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks)/gramvaani.zip) |

### Download Models
|Language |Model Checkpoint |
| - | - |
| Bengali | [hf](https://indicwhisper.objectstore.e2enetworks.net/bengali_models.zip) |
| Gujarati | [hf](https://indicwhisper.objectstore.e2enetworks.net/gujarati_models.zip) |
| Hindi | [hf](https://indicwhisper.objectstore.e2enetworks.net/hindi_models.zip) |
| Kannada | [hf](https://indicwhisper.objectstore.e2enetworks.net/kannada_models.zip) |
| Malayalam | [hf](https://indicwhisper.objectstore.e2enetworks.net/malayalam_models.zip) |
| Marathi | [hf](https://indicwhisper.objectstore.e2enetworks.net/marathi_models.zip) |
| Odia | [hf](https://indicwhisper.objectstore.e2enetworks.net/odia_models.zip) |
| Punjabi | [hf](https://indicwhisper.objectstore.e2enetworks.net/punjabi_models.zip) |
| Sanskrit | [hf](https://indicwhisper.objectstore.e2enetworks.net/sanskrit_models.zip) |
| Tamil | [hf](https://indicwhisper.objectstore.e2enetworks.net/tamil_models.zip) |


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Summary: auto


**Summarizing Vistaar resources**

The Vistaar README is quite helpful, containing a WER table for IndicWhisper across datasets and languages, as well as a Hindi subset comparison table. There are resource links for downloading training datasets and model checkpoints, which are super valuable. We can use these zipped benchmark datasets, though I noticed some links might be broken, like the Gramvaani benchmark link with extra parentheses. Now, we can summarize this for another agent regarding our tasks and evaluation plans.


**Planning WER minimization strategy**

I want to establish an expected baseline to compare our pipeline's performance. Our goal is to minimize WER further through ensembling and CTC gating, so I’m curious if we can outperform IndicWhisper on the same benchmarks by integrating models like Whisper large v3. The benchmark zips probably contain audio segments with transcripts, which we can use in our pipeline by adjusting the segments_dir. However, I need to check the structure inside the zips, maybe even do a small HEAD request to assess file sizes without downloading the entire dataset.


**Instructions for benchmark processing**

To guide the other agent, we can have them download the zips, extract the contents, and look for the likely manifest files for each dataset and language. The Vistaar README offers tutorials for evaluating ASR models, but adapting it to our pipeline might be easier. It would be great if the other agent creates a script to convert the Vistaar benchmark manifest into our `segments_dir` and `refs.jsonl` format. For Assamese evaluation, we should use FLEURS and Common Voice directly. As for the robustness of our code for benchmarks across different languages, it seems promising, but there are some caveats regarding CTC scoring for MMS that might need attention.


**Highlighting code issues**

It seems there's a bit of confusion regarding the updates. They mentioned they added mapping and scripts, but I'm not seeing it quite yet. They stated they have "CTC scoring guardrails and alignment-coverage metrics," but I still don't see a set_target_lang. It's crucial to keep pointing this out. However, since the user wants to move forward, I'll treat the code as robust while noting this ongoing issue. The user also mentioned several fixes like stable segment IDs and improvements in Gemini parsing, which is encouraging.


**Planning benchmarking instructions**

The user is ready to move forward and thinks the code is solid. I agree it's mostly robust, but the main issue is the missing MMS target_lang for CTC scoring, which could lead to failures for non-English datasets. They want suggestions rather than more code changes, so I need to propose two approaches for benchmarking. The first is using Vistaar benchmark zips, which contains the precise benchmarks, but those might be more complex. The second approach is utilizing HuggingFace datasets like FLEURS and Common Voice, which offers easier access. For both, we can suggest evaluating a small sample of 100 segments per language and provide commands for downloading, unzipping, determining segments_dir, and generating refs.jsonl. I'll need to check the Vistaar README for specifics on "manifest creation" as part of this plan.


python3 - <<'PY'
import requests
text=requests.get('https://raw.githubusercontent.com/AI4Bharat/vistaar/master/README.md',timeout=20).text.splitlines()
# find the section for Evaluating ASR models
start=None
for i,line in enumerate(text):
    if line.strip().lower().startswith('### evaluating asr models'):
        start=i
        break
if start is None:
    print('not found')
    raise SystemExit
end=len(text)
for i in range(start+1,len(text)):
    if text[i].startswith('### ') and i!=start:
        end=i
        break
snippet='\n'.join(text[start:end])
print(snippet[:2000])
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
### Evaluating ASR models
  - Manifest creation
    - For each dataset Download and extract the benchmark data in a directory. The data should be extracted in such a way that each folder inside should contain data for a particular language i.e each language specific folder should contain train, valid and test folder and within them the audio + transcript.txt
    - Sample structure of folder tree:
    ```
      kathbath
          ├── bengali
          │   ├── audio
          │   └── transcript.txt
          │
          └── gujarti
              ├── audio
              └── transcript.txt
          .
          .
          .
          .
    ```
    - The manifest needs to be a Json file where each line is a dictionary with audio filepath, duration of audio and text transcript
    ```
     {"audio_filepath": <path to audio file 1>, "duration": <duration of audio file 1>, "text": <transcript of audio file 1>}
     {"audio_filepath": <path to audio file 2>, "duration": <duration of audio file 2>, "text": <transcript of audio file 2>}
     .
     .
     .
    ```
  - Running evaluation
    ```
    python evaluation.py --model_path=<model path> \
    --manifest_path=<manifest path in vistaar> \
    --manifest_name=<dataset name in vistaar> \
    --device=<gpu to use> \
    --batch_size=<batch size> \
    --language=<2 letter language code>
    ```


**Considering dataset suggestions**

I’m thinking about how to use the Vistaar dataset languages and whether to implement per-language CTC model mapping, possibly from AI4Bharat's `indicwav2vec`. I could disable CTC gating for initial WER measurement, but that might complicate things. To benchmark improvements, running with and without CTC could be insightful. Now, I should provide dataset suggestions and instructions, summarizing options available like Kathbath and Common Voice, along with their WER details for clarity.


**Discussing dataset options**

For English, I can suggest Librispeech for clean reads, but it won't have the Indian accent. For Indian English, Common Voice or other datasets might work. Another option is IndicSUPERB from Collabora, which includes audio and transcriptions, though it’s quite large. The user wants benchmarks for a few datasets to get accuracy scores, so I could mention Vistaar numbers like Hindi IndicWhisper at 10.3 on Kathbath and the average of 13.6. I also need to touch on possibly using a superior model like gemini-3-pro for other languages.


**Exploring model options**

I’m considering proposing the use of `gemini-3-pro-preview` as an additional hypothesis for handling code-switching and named entities. However, I need to emphasize that CTC verification is essential, and this model should not be the main reference. Since they asked for ideas, including Gemini-3-Pro as a model for challenging segments could be beneficial to evaluate improvement through selection gaps or WER. It’s important to maintain fairness, especially since Gemini might produce differently normalized transcripts. We should compare transcripts carefully, noting that many datasets like Common Voice and FLEURS use normalization. I'll keep my response structured yet concise, perhaps using bullet points or headings for clarity.


**Structuring the response clearly**

I’m thinking about using markdown with bullet lists to organize my response and possibly including headings for clarity. I know I should avoid citations like `【F:...】`, but plain links are allowed. No secrets can be included, so I'll keep it straightforward.

For example:
- **1) Code Readiness**: Yes, with some outstanding checks.
  - Confirm that the pipeline is capable of benchmarking and that the segment_id is stable for a good evaluation.
  - It’s important to mention that for CTC scoring with `facebook/mms-1b-all`, the `tokenizer.set_target_lang` needs to be set; otherwise, there will be many `too_many_unk_tokens` issues for non-English tokens.


**Suggesting implementation patches**

I’m considering suggesting some patches. First, I’ll mention dataset gating and the need for large downloads, which also involves the ffmpeg requirement. It’s vital to ensure model diversity, providing at least two models for a meaningful ROVER/oracle setup, while noting that the pipeline records `ensemble.warnings`.

For benchmark datasets, I’ll provide important findings from the AI4Bharat Vistaar repo, including WER numbers like an average of 24.6 and Hindi averages at 13.6. However, I’ll make sure to highlight that some numbers in the README differ from table values across versions, so I’ll recommend using the Vistaar repo for consistent references.


**Drafting benchmark execution instructions**

I want to provide structured guidance for benchmark execution. First, I'll suggest downloading dataset links from the README, like kathbath.zip, which contains training data and benchmarks. It's essential to note that these resources are directly zipped and stored, not hosted on HF.

For the Vistaar benchmark tasks, I'll guide users to run commands for downloading datasets, extracting files, and generating refs.jsonl. I'll also include an example Python snippet to ensure they can adequately parse the transcript file format.

Next, I plan to outline the steps for running the full evaluation pipeline using different models, highlighting the importance of setting the target language for accurate CTC scoring.

Finally, I’ll emphasize ensuring the correct dataset segregation and the need for consistency in evaluating models across languages with attention to known discrepancies in WER scores. Overall, my response will be concise yet detailed enough to cover the necessary instructions.


**Finalizing benchmark instructions**

I have to provide a Python snippet for generating the `refs.jsonl` file based on a transcript. This involves reading the lines, handling different delimiters, and storing the data in a structured format. After that, I'll guide the user on running the internal transcription pipeline with placeholders for directory paths and model IDs.

I want to highlight the importance of using a correct tokenizer setting for MMS models to avoid issues, so I’ll specify where to add it. Additionally, I'll mention relevant benchmark datasets like Vistaar and Kathbath, ensuring to acknowledge the availability of direct benchmark zips from AI4Bharat while keeping everything concise yet informative.


**1) Code readiness for open benchmarking**
- Overall yes: you now have reproducible IDs (`maya_transcribe/ids.py:16`), audit-friendly outputs (`maya_transcribe/pipeline.py:262`), and eval that reports WER+CER+tails+oracle gap (`maya_transcribe/eval.py:54`).
- One critical check before benchmarking with **MMS CTC**: `facebook/mms-1b-all` (and similar) needs `processor.tokenizer.set_target_lang(<iso3>)` before tokenizing text; otherwise Indic scripts become mostly `<unk>` and CTC scoring will silently fail via `too_many_unk_tokens`. I don’t see any `set_target_lang` call in `maya_transcribe/ctc_fit.py:101` yet. If you’re using MMS as the fallback CTC model, patch this (or ensure your `ctc_model_map` always points to monolingual CTC models that don’t require it).
- Benchmarking will work best only if you actually run ≥2 diverse ASR hypotheses; otherwise ROVER/CTC selection is meaningless (watch `ensemble.warnings` in `results.jsonl`).

**2) Benchmarks/datasets to use (internet findings)**
- **AI4Bharat Vistaar** (benchmarks + downloads + baseline WER tables): https://github.com/AI4Bharat/vistaar
  - Includes benchmark zips for **Kathbath, Kathbath-Hard, FLEURS, CommonVoice, IndicTTS, MUCS, GramVaani** and reports IndicWhisper WER (e.g., overall avg WER 24.6%; Hindi subset avg 13.6%) in the repo README.
  - Direct benchmark downloads are linked in their README (example): `https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/kathbath.zip` (same pattern for others).
- For **Assamese + English** (not covered in Vistaar’s 12-language table): use `google/fleurs` and Common Voice (HuggingFace `datasets`) and evaluate the same way.
- Optional additional benchmark: `collabora/indic-superb` (HF dataset; has `audio` + `transcription`, but is large).

**3) Handover: how to run benchmarks + analyze language patterns (no new files)**
1) **Download + unpack one benchmark** (Vistaar example):
   - `curl -L -o kathbath_bench.zip https://indicwhisper.objectstore.e2enetworks.net/vistaar_benchmarks/kathbath.zip`
   - `unzip kathbath_bench.zip -d bench/vistaar/kathbath`
2) **Create `refs.jsonl` for one language folder** (Vistaar structure is `…/<language>/audio/*.wav` + `…/<language>/transcript.txt` per their README). Run this from repo root:
```bash
python3 - <<'PY'
from pathlib import Path
import json
from maya_transcribe.ids import segment_identity

lang_dir = Path("bench/vistaar/kathbath/hindi")  # adjust
audio_dir = lang_dir / "audio"
txt_path = lang_dir / "transcript.txt"
out_path = lang_dir / "refs.jsonl"

def parse_line(line: str):
    line=line.strip()
    for sep in ("\t","|"," ",","):
        if sep in line:
            a,b=line.split(sep,1)
            return a.strip(), b.strip()
    return None

rows=[]
for line in txt_path.read_text(encoding="utf-8").splitlines():
    if not line.strip():
        continue
    parsed=parse_line(line)
    if not parsed:
        continue
    fname, text = parsed
    src = audio_dir / fname
    ident = segment_identity(src, root=audio_dir)
    rows.append({"segment_id": ident.segment_id, "text": text})

out_path.write_text("\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n", encoding="utf-8")
print("wrote", out_path, "rows", len(rows))
PY
```
3) **Run pipeline + eval** (start with no CTC to validate end-to-end; then enable CTC):
```bash
maya-transcribe run --segments-dir bench/vistaar/kathbath/hindi/audio --out bench_runs/kathbath_hi --whisper-model large-v3 \
  --transformers-whisper-model <indic_whisper_hf_id>

maya-transcribe eval --results bench_runs/kathbath_hi/results.jsonl --refs bench/vistaar/kathbath/hindi/refs.jsonl --out bench_runs/kathbath_hi/eval.json
```
4) **Pattern analysis to tune** (estimate human workload + where WER comes from):
```bash
python3 - <<'PY'
import json, collections
p="bench_runs/kathbath_hi/results.jsonl"
dec=collections.Counter(); reasons=collections.Counter()
for line in open(p,encoding="utf-8"):
    r=json.loads(line)
    v=r.get("verification",{})
    dec[v.get("decision")] += 1
    for reason in (v.get("reasons") or []):
        reasons[reason]+=1
print("decisions:",dec)
print("top reasons:",reasons.most_common(20))
PY
```
5) Repeat per language/dataset; use `eval.json` + oracle stats to decide:
- High `oracle_best_wer` ⇒ you need a better base ASR for that language/domain.
- High `selection_gap_wer_mean` ⇒ selection/gating (ROVER/CTC thresholds) needs tuning or the CTC model is mismatched.

If you tell me which exact datasets you want first (Vistaar-only vs Vistaar+FLEURS for Assamese/English), I can give a minimal “benchmark matrix” (languages × datasets × model configs) to run as ablations.


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <approval_policy>on-request</approval_policy>
  <sandbox_mode>workspace-write</sandbox_mode>
  <network_access>restricted</network_access>
  <shell>bash</shell>
</environment_context>

<environment_context>
  <approval_policy>never</approval_policy>
  <sandbox_mode>danger-full-access</sandbox_mode>
  <network_access>enabled</network_access>
  <shell>bash</shell>
</environment_context>

# Context from my IDE setup:

## Open tabs:
- Reference1.md: docs/Reference1.md
- Agents.md: Agents.md
- task.md: docs/task.md
- Reference5.md: docs/Reference5.md
- Reference6.md: docs/Reference6.md

## My request for Codex:
Task - [task.md](task.md)
Reference docs by other AI agents - /home/ubuntu/maya-transcribe/docs
Agents - [Agents.md](Agents.md)
secrets - /home/ubuntu/maya-transcribe/.env

# Context from my IDE setup:

## Open tabs:
- Agents.md: Agents.md
- task.md: task.md
- Reference1.md: docs/Reference1.md
- task.md: docs/task.md
- Reference5.md: docs/Reference5.md

## My request for Codex:
Can you now create the master.md in the root folder here along with agents.md with a block diagram and execution plan and why this current plan gives the minimal WER possible till date as we are generating transcriptions offline and works smartly and cleverly to minimize WER.

# Context from my IDE setup:

## Open tabs:
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md
- Reference1.md: docs/Reference1.md
- task.md: docs/task.md

## My request for Codex:
Can you create the architecture diagram as in like a block diagram, not tree so i have to put it in a visual editor. just create the diagram with lines and replace it with existing ## Pipeline block diagram with the new lines so i can directly understand the diagram without having to put it in a diff editor or compiler to view the diagram.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
can you check the implementation of my other coding agent and add reviews comments. maybe if you have comments, provide them in chat here and i shall carry them to the other agent. check for code robustness, and failure cases, edge cases in perspective of youtube audio segments and minimizing WER. let's also think of a sophisticated way of evaluating this pipeline. I can maybe help with telugu and hindi, but for other languages we need a robust handler like a superior model. what shall we consider for this. maybe a higher LLM like gemini-3-pro ?

# To run this code you need to install the following dependencies:
# pip install google-genai

import base64
import os
from google import genai
from google.genai import types

def generate():
    client = genai.Client(
        api_key=os.environ.get("GEMINI_API_KEY"),
    )

    model = "gemini-3-pro-preview"
    contents = [
        types.Content(
            role="user",
            parts=[
                types.Part.from_text(text="""INSERT_INPUT_HERE"""),
            ],
        ),
    ]
    generate_content_config = types.GenerateContentConfig(
        thinking_config=types.ThinkingConfig(
            thinking_level="HIGH",
        ),
        system_instruction=[
            types.Part.from_text(text="""Instruction: You are a strict, verbatim transcription engine for Indian languages
Input: Audio file attached
Primary audio language: telugu
Task:
Listen to the audio carefully
Identify the primary spoken language as specified above
Transcribe the speech exactly as spoken and produce four separate outputs as defined below
Sometimes you won't be clear about complete transcription but you know parts of it. From that you can fix the grammar. That's how you can create the best transcription.
You know the first word and the last word. You can create the best transcription. You know the middle word and the last word. You can create the best transcription.
You need to think in terms of what you know and how you can go to the end with 100% accuracy with predicting what comes here, what would be it, what is the right fit and attaching it all together and doing the combinations

STRICT GLOBAL RULES (apply to all outputs):
Verbatim only Include all repetitions filler words stammers false starts hesitations and colloquial expressions exactly as spoken
No normalization Do not correct grammar do not fix pronunciation do not standardize spellings and do not clean up dialect accent or mixed language usage
No inference Do not add meaning structure or emphasis that is not clearly audible
Script fidelity Follow the rules of each output strictly Do not translate paraphrase or summarize
OUTPUT 1: Proper native transcription
Write everything only in the native script of the primary language
Do not use any punctuation marks at all
Even if English words are spoken write them phonetically using the native script of the primary language
Preserve repetitions fillers and stammers exactly

OUTPUT FORMAT RULES:
Clearly label the four outputs as
Output 1: Proper native transcription
Do not add explanations notes or any extra English text beyond the labels"""),
        ],
    )

    for chunk in client.models.generate_content_stream(
        model=model,
        contents=contents,
        config=generate_content_config,
    ):
        print(chunk.text, end="")

if __name__ == "__main__":
    generate()

API key is in .env as GEMINI_KEY and include prompt robustness by understanding the prompt and giving it robust instructions as per [Prompt.md](Prompt.md)

First finish the code review, then suggest evaluations for WER check for our pipeline with a superior model.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
my coding agent says,

Fixes Applied

Prompt made non-hallucinatory + single-output: Prompt.md (line 6) (no “predict/fix grammar”, use <unk> if unsure; English stays Latin; no extra outputs).
Gemini integration added as an optional hypothesis: gemini_asr.py (line 126) + cli.py (line 178) (actually sends audio via types.Part.from_bytes(data=..., mime_type=...), reads GEMINI_KEY, temperature=0, retries/backoff, strict parsing/validation, stores raw output in meta).
Pipeline hardened: pipeline.py (line 44) + verify.py (line 31) (collision-safe segment_id, fail-fast MMS-LID by default, uses Whisper logprob/compression heuristics, optional Gemini, optional CTC-fit scoring that selects final transcript and can gate acceptance).
Eval upgraded: eval.py (line 50) (WER+CER, duration buckets, tail stats; maya-transcribe eval defaults to hypothesis=final).
Try It

maya-transcribe gemini --audio <file.wav> --language eng
wav2vec2-base-960h --gemini-model gemini-2.0-flash
Per-language CTC mapping: maya-transcribe run ... --ctc-model-map ctc_models.json (JSON { "eng": "<hf_id>", "hin": "<hf_id>", ... }).

can you do another round of review.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
Review Fixes Landed

Stable, reproducible segment_id (always base_id-uid, 64-bit hash): ids.py (line 16) (this will change IDs vs older runs, so older refs.jsonl won’t match).
Gemini language hint now uses human-friendly names/scripts for prompting (and “Unknown → auto-detect”): language.py (line 80), gemini_asr.py (line 153).
Gemini parsing is less brittle (salvage multi-line, strip punctuation, still records parse flags): gemini_asr.py (line 100).
CTC scoring guardrails + alignment-coverage metrics (unk-ratio filter, token coverage/score stats): ctc_fit.py (line 64).
Verification gate now uses CTC coverage metrics in addition to margin (flags ctc_low_coverage / ctc_low_token_score / ctc_span_mismatch): verify.py (line 31).
Eval now includes oracle stats (oracle-best WER/CER + selection gap): eval.py (line 50).
Metrics To Surface In UI (already in results.jsonl)

Primary decisioning: verification.decision, verification.reasons, verification.scores (pipeline.py (line 263)).
Audio/QC: src_ffprobe.duration_sec vs ffprobe.duration_sec (trim impact), qc.rms_dbfs, qc.peak_dbfs, qc.clipping_fraction.
LID: lid.predictions[] and verification.scores.lid.
Ensemble health: ensemble.unique_models, ensemble.warnings (low_model_diversity): pipeline.py (line 293).
Hypotheses + comparisons: asr[*].text, asr_summary, rover.text + rover.disagreements, ctc.scores, and final.text + final.source.

Addressed. can you now search the internet and find benchmarks on few datasets that we can actually use for in this scenario here and get some accuracy scores or WER scores on our pipeline ? how about datasets from ai4bharath any benchmarking datasets ? can you search and give ideas to my other agent ? and also check if the code is robust enough now for us to do a benchmark on few samples from various languages and analyse patterns between languages so we can tune accordingly.

1. Firstly, see if all code is robust and we are good to perform benchmarks on opensource data
2. give ideas and how we can perform these benchmarks, so we can get language-wise performance or statistics
3. handover script/instructions(inchat, dont create new files) to my other agent to do these benchmarks and analyse patterns to better conclude this before we move to a UI based diff analysis and see if human intervention is required at any scale or not.


# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
what all huggingface repo's do i need for us to get these datasets. i'd like to check if i have access first before letting the agent create benchmarking tests.


Summary: auto


**Clarifying dataset access**

I need to clarify that if you’re using Vistaar benchmark zips, you won't require access to Hugging Face datasets; instead, you just download from the object store. However, you might still need HF for certain models like Whisper.

To provide clarity, I'll create two categories:

1. **Recommended**: Use Vistaar benchmark zips (direct download) — no HF access required, just an internet connection.
2. **If you prefer HF datasets**: The identifiers like `google/fleurs`, `mozilla-foundation/common_voice_XX_0`, and `ai4bharat/indicvoices` may be necessary.

I’ll ensure I check Hugging Face API to get accurate dataset names!


**Querying Hugging Face datasets**

I can run a Python script to query `huggingface_hub`, but I'm not sure if it’s installed. Instead, I can use `curl` to call the Hugging Face API at `https://huggingface.co/api/datasets?search=...`, which returns a JSON list of datasets.

It's important to know if the user has Hugging Face access, and they should check this with `huggingface-cli whoami` or a quick Python command. I'll focus on listing dataset repo IDs and whether they're gated while querying the API for relevant dataset names, like "vistaar" or "kathbath." Let's go ahead and do that!


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=vistaar&limit=20" | python - <<'PY'
import sys, json
j=json.load(sys.stdin)
for d in j:
    print(d.get('id'))
PY


Output:
Exit code: 127
Wall time: 0.2 seconds
Output:
/bin/bash: line 2: python: command not found


I'm noticing that the Python command is missing, which is surprising! I wonder if Python isn't installed on this system or if the command should actually be `python3`. It seems like a good plan to try rerunning it with that command. It's moments like these that make me question what's really going on behind the scenes, but I'm focused on getting it sorted out. Let’s see if using `python3` resolves this issue!


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=vistaar&limit=20" | python3 - <<'PY'
import sys, json
j=json.load(sys.stdin)
for d in j:
    print(d.get('id'))
PY


Output:
Exit code: 1
Wall time: 0.2 seconds
Output:
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)


I think the curl output might be experiencing rate-limiting or some kind of issue. I should really take a closer look at the first bytes to understand what's going on. Inspecting those first bytes could reveal important information about the response. It’s a good idea to dig in and see if there’s a specific error or detail that can guide me in troubleshooting this. Let's get to the bottom of it!


set -euo pipefail
curl -s -D - "https://huggingface.co/api/datasets?search=vistaar&limit=5" | head


Output:
Exit code: 23
Wall time: 0.2 seconds
Output:
HTTP/2 200
content-type: application/json; charset=utf-8
content-length: 4391
date: Tue, 13 Jan 2026 06:43:36 GMT
etag: W/"1127-4bZgjtenmdDxSU7FQvyB8QvmiBo"
x-powered-by: huggingface-moon
x-request-id: Root=1-6965e998-0317257e0f4b54c20db2c96a
ratelimit: "api";r=497;t=60
ratelimit-policy: "fixed window";"api";q=500;w=300
cross-origin-opener-policy: same-origin


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=vistaar&limit=5" | head -c 200 && echo


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
[{"_id":"67407b1480c7c8e332c5d344","id":"iiBLACKii/Gujarati_Vistaar_DB","author":"iiBLACKii","disabled":false,"gated":false,"lastModified":"2024-11-22T12:49:04.000Z","likes":0,"trendingScore":0,"priva


**Troubleshooting JSON Parsing**

I'm trying to figure out the JSON parsing issue. It looks like there might be an error with my Python script since curl's output isn't being piped correctly. I realized that when I use `python3 - <<'PY'`, it reads the code from heredoc and not from stdin, which isn't what I want in this situation. I should consider using `python3 -c` or adjust the piping so that the JSON from curl is passed correctly to Python. It’s a bit tricky, but I'm working on it!


**Adjusting Python Command**

I've realized there's an issue with how I'm trying to pipe curl into Python. Using heredoc alongside the pipe isn't working as I expected. I need to decide whether to use `python3 -c` and read from stdin or use `python3 - <<'PY'` without the pipe. It looks like the better choice is to go with `python3 -c`, allowing me to fetch the JSON directly from stdin. I’ll write it as `curl -s ... | python3 -c 'import sys,json; j=json.load(sys.stdin); ...'`. Let's give this a shot!


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=vistaar&limit=20" | python3 -c 'import sys,json; j=json.load(sys.stdin); print("\n".join(d.get("id","") for d in j))'


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
iiBLACKii/Gujarati_Vistaar_DB
ekacare/vistaar_small_asr_eval
AdityK2409/vistaar_small_asr_eval
AdityK2409/vistaar_small_asr_eval_2


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=kathbath&limit=20" | python3 -c 'import sys,json; j=json.load(sys.stdin); print("\n".join(f"{d.get("id") } gated={d.get("gated")}" for d in j))'


Output:
Exit code: 1
Wall time: 0.2 seconds
Output:
  File "<string>", line 1
    import sys,json; j=json.load(sys.stdin); print("\n".join(f"{d.get("id") } gated={d.get("gated")}" for d in j))
                                                                       ^^
SyntaxError: f-string: unmatched '('


**Fixing quoting issues**

I'm encountering some quoting problems, and I need to adjust how I handle them. I think using single quotes inside a formatted string is the way to go here. Specifically, I'll implement it like this: `f"{d.get('id')} gated={d.get('gated')}"`. This adjustment should help me avoid those quoting issues and ensure everything works smoothly! Let's make this change and see how it goes.


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=kathbath&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
ai4bharat/Kathbath gated=auto
SayantanJoker/Kathbath-Hindi-Valid-Female gated=False
SayantanJoker/Kathbath-Hindi-Valid-Male gated=False
nickfuryavg/Kathbath-Audio-Subset-Batch-1 gated=False
nickfuryavg/Kathbath-Audio-Subset-Batch-2 gated=False
nickfuryavg/Kathbath-Audio-Subset-Batch-3 gated=False
nickfuryavg/Kathbath-Audio-Subset-Batch-4 gated=False
nickfuryavg/Kathbath-Audio-Subset-Batch-5 gated=False
nickfuryavg/Kathbath-Audio-Subset-1 gated=False
nickfuryavg/Kathbath-Audio-Subset-2 gated=False
nickfuryavg/Kathbath-Audio-Subset-3 gated=False
nickfuryavg/Kathbath-Audio-Batch-1 gated=False
nickfuryavg/Kathbath-Audio-Batch-2 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-28 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-37 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-29 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-67 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-73 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-79 gated=False
nickfuryavg/Kathbath-Audio-44100-Batch-91 gated=False


**Searching for datasets**

I'm noticing that the official dataset `ai4bharat/Kathbath` might be gated, which could limit access. There are other subsets that aren't gated but might be incomplete. I also brought up Svarah, Lahaja, and IndicVoices as options. Let’s explore these datasets further to see what we can find! I'll want to ensure I’m obtaining all relevant information without getting tangled up in availability issues.


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=ai4bharat%2F&limit=50" | python3 -c "import sys,json; j=json.load(sys.stdin);
for d in j:
    if d.get('author')=='ai4bharat':
        print(f\"{d.get('id')} gated={d.get('gated')}\")"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
ai4bharat/Aksharantar gated=False
ai4bharat/indicvoices_r gated=auto
ai4bharat/Kathbath gated=auto
ai4bharat/indic_glue gated=False
ai4bharat/MILU gated=auto
ai4bharat/IndicVoices-ST gated=auto
ai4bharat/Svarah gated=auto
ai4bharat/IndicVoices gated=auto
ai4bharat/samanantar gated=False
ai4bharat/IndicParaphrase gated=False
ai4bharat/IndicHeadlineGeneration gated=False
ai4bharat/IndicSentenceSummarization gated=False
ai4bharat/IndicWikiBio gated=False
ai4bharat/IndicQuestionGeneration gated=False
ai4bharat/indicwav2vec_noa_hf gated=False
ai4bharat/IndicQA gated=False
ai4bharat/IndicCOPA gated=False
ai4bharat/IndicXParaphrase gated=False
ai4bharat/IndicSentiment gated=False
ai4bharat/naamapadam gated=False
ai4bharat/Bhasha-Abhijnaanam gated=False
ai4bharat/IN22-Gen gated=auto
ai4bharat/IN22-Conv gated=auto
ai4bharat/IndicSentiment-Translated gated=False
ai4bharat/IndicXNLI-Translated gated=False
ai4bharat/IndicXParaphrase-Translated gated=False
ai4bharat/IndicCOPA-Translated gated=False
ai4bharat/IndicWikiBio-Translated gated=False
ai4bharat/IndicHeadlineGeneration-Translated gated=False
ai4bharat/ai2_arc-hi gated=False
ai4bharat/indic-instruct-data-v0.1 gated=False
ai4bharat/IndicQA-Translated gated=False
ai4bharat/human-eval gated=False
ai4bharat/boolq-translated gated=False
ai4bharat/hellaswag-translated gated=False
ai4bharat/sangraha gated=False
ai4bharat/indic-align gated=False
ai4bharat/INCLUDE gated=False
ai4bharat/FBI gated=False
ai4bharat/wiki-translate gated=False
ai4bharat/wiki-transliterate gated=False


Summary: auto


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=lahaja&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
ai4bharat/Lahaja gated=auto
Saranath07/test-lahaja gated=False
SayantanJoker/lahaja gated=False


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=gramvaani&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
theArif/gramvaani-reduplication-repetition-toy-dataset gated=False
TheAIchemist13/gramVaani-dataset-test gated=False
TheAIchemist13/gramVaani-dataset-train gated=False
TheAIchemist13/gramvaani_preprocessed_hi_train gated=False
TheAIchemist13/gramvaani_preprocessed_hi_test gated=False
satani/gramvaani_pseudo_labelled gated=False


**Searching for datasets**

I’m looking for the AI4Bharat Gramvaani dataset and I’m not finding an official one on Hugging Face, so I wonder if it might be spelled differently. I notice there’s an entry labeled `ai4bharat/Gramvaani`, so it seems like it could be what I need. Let’s search using "Gramvaani" in a way that doesn’t depend on case sensitivity, just to be sure I’m covering all bases!


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=Gramvaani&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
theArif/gramvaani-reduplication-repetition-toy-dataset gated=False
TheAIchemist13/gramVaani-dataset-test gated=False
TheAIchemist13/gramVaani-dataset-train gated=False
TheAIchemist13/gramvaani_preprocessed_hi_train gated=False
TheAIchemist13/gramvaani_preprocessed_hi_test gated=False
satani/gramvaani_pseudo_labelled gated=False


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=MUCS&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
AdityK2409/mucs gated=False


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=mucs&limit=50" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
AdityK2409/mucs gated=False


**Searching for IndicTTS**

I’m not seeing anything official, and I’m not sure if it’s available on Hugging Face. I think I should search for IndicTTS. But for ASR evaluation, I'm wondering whether it's reliant on transcripts. Wait, Vistaar uses the IndicTTS dataset for something, but I'm unclear about its application. It seems IndicTTS is mainly a TTS dataset, and might be used for synthetic evaluations. I should look into it further.


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=IndicTTS&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
SPRINGLab/IndicTTS-Hindi gated=False
MatrixSpeechAI/Indic_TTS_English_Female_Speaker gated=False
SPRINGLab/IndicTTS_Gujarati gated=False
SPRINGLab/IndicTTS_Marathi gated=False
SPRINGLab/IndicTTS_Odia gated=False
SPRINGLab/IndicTTS_Telugu gated=False
thennal/indic_tts_ml gated=False
kavyamanohar/indictts-ml gated=False
MatrixSpeechAI/Indic_TTS_HI_Male_Speaker_1 gated=False
MatrixSpeechAI/Indic_TTS_HI_Male_Speaker_2 gated=False
MatrixSpeechAI/Indic_TTS_HI_Female_Speaker_3 gated=False
MatrixSpeechAI/Indic_TTS_HI_Female_Speaker_4 gated=False
cmeraki/indic_tts gated=False
MatrixSpeechAI/Indic_TTS_English_Female_Speaker_stage_1 gated=False
MatrixSpeechAI/Indic_TTS_English_Female_Speaker_stage_2 gated=False
MatrixSpeechAI/Indic_TTS_English_Female_Speaker_stage_3 gated=False
SPRINGLab/IndicTTS-English gated=False
SPRINGLab/IndicTTS_Bengali gated=False
SPRINGLab/IndicTTS_Tamil gated=False
techkiksar/Indic_TTS_English_Female_Speaker gated=False


**Identifying datasets for ASR**

I see that there are IndicTTS datasets from SPRINGLab, but I need to focus on the user’s request for datasets from ai4bharath, specifically looking into benchmarking datasets. The Hugging Face repository has several relevant datasets like `ai4bharat/Kathbath`, `ai4bharat/IndicVoices`, and others, but I need to be careful to ensure they're appropriate. There’s a bit of ambiguity in the question about which datasets are necessary for benchmarking. I'll clarify these details further.


**Listing dataset access details**

I can list the datasets and note that some are "gated," meaning users need to accept access on Hugging Face. They can check by visiting each dataset page or using the `huggingface-cli` to test access. I’ll include commands like `huggingface-cli whoami` and a Python snippet for loading datasets to facilitate this.

For those following the Vistaar benchmark suite, they can download datasets directly from the object store. For Hugging Face datasets, I'll mention several useful IDs like `ai4bharat/Kathbath` and others that need access.


**Clarifying gated dataset access**

I see that some datasets are noted as "gated=auto," meaning users might need to accept a license to request access. There are non-AI4Bharat open datasets like `google/fleurs` and `mozilla-foundation/common_voice_17_0`, which are likely ungated, whereas `collabora/indic-superb` probably requires a check.

To help users quickly verify access, I can suggest running commands like `huggingface-cli login` with their token and a Python snippet to check a dataset while using `streaming=True`. It’s important to remind them that even with streaming, access is still needed. I'll ensure to keep the response straightforward!


**Confirming dataset gating statuses**

I want to ensure accuracy regarding the gating statuses of datasets, so I plan to confirm this via API calls instead of guessing. It’s essential to query each dataset with the Hugging Face API for precise information. I’ll check datasets like `google/fleurs`, `collabora/indic-superb`, and `mozilla-foundation/common_voice_17_0` by using their respective endpoints, like `https://huggingface.co/api/datasets/{repo_id}`.

I'll also use a command like `curl -s` to retrieve the information and process it with Python. This way, I can provide the most reliable details for each dataset!


set -euo pipefail
for id in \
  "google/fleurs" \
  "collabora/indic-superb" \
  "mozilla-foundation/common_voice_17_0" \
  "ai4bharat/Kathbath" \
  "ai4bharat/IndicVoices" \
  "ai4bharat/IndicVoices-ST" \
  "ai4bharat/indicvoices_r" \
  "ai4bharat/Lahaja" \
  "ai4bharat/Svarah" \
; do
  echo "--- $id"
  curl -s "https://huggingface.co/api/datasets/$id" | python3 -c "import sys, json; j=json.load(sys.stdin); print('gated=', j.get('gated')); print('private=', j.get('private')); print('sha=', j.get('sha'))"
done


Output:
Exit code: 0
Wall time: 1.9 seconds
Output:
--- google/fleurs
gated= False
private= False
sha= d7c758a6dceecd54a98cac43404d3d576e721f07
--- collabora/indic-superb
gated= False
private= False
sha= 25ef1078844ba3c20be31581020deadebc25579e
--- mozilla-foundation/common_voice_17_0
gated= False
private= False
sha= 11dc88355e899d1bf2df74f01b904a8544a17b33
--- ai4bharat/Kathbath
gated= auto
private= False
sha= 5b9e92849222026d9141acba4e8434fe816396bf
--- ai4bharat/IndicVoices
gated= auto
private= False
sha= c3c334ed5d659be63bcffa802852c45ac2b17853
--- ai4bharat/IndicVoices-ST
gated= auto
private= False
sha= 1d1774510ffd5097a1e711b6f713677cee6bdc9e
--- ai4bharat/indicvoices_r
gated= auto
private= False
sha= 5f4495c91d500742a58d1be2ab07d77f73c0acf8
--- ai4bharat/Lahaja
gated= auto
private= False
sha= d4ffd2ecbdd933e37c917ddcf620eef159ceb3a7
--- ai4bharat/Svarah
gated= auto
private= False
sha= ebbf7777fe771490696a3f7b007097606fa8c924


**Providing dataset access information**

I see the user wants to check access before allowing an agent to create benchmarking tests. For multi-lingual benchmarking, I can provide a minimal set of datasets. There are some ungated options like `google/fleurs`, `mozilla-foundation/common_voice_17_0`, and `collabora/indic-superb`. Then, for gated datasets from AI4Bharat, such as `ai4bharat/Kathbath` and others, I’ll mention they might need to request access. I'll also note that they're interested in ASR benchmarks with specific instructions for accessing HF UI.


**Checking dataset access via CLI**

I'm considering how to check dataset access through the command line interface. I could set the `HF_TOKEN` and then use a simple Python command to load a dataset, like `ai4bharat/Kathbath`. This command only reads metadata and streams, so if access is denied, I expect it to error out quickly. I'll need to provide similar commands for other datasets like IndicVoices and Lahaja, while being mindful of config names. I should check the dataset card for these names as they might use ISO codes. I can also suggest a command to fetch config names without downloading the data.


**Providing fallback access methods**

I’m thinking about how to provide a fallback access method. I can suggest using `curl` to access the README file for `ai4bharat/Kathbath`, but I’m unsure if that’s accessible if the dataset is gated. Usually, the dataset card is visible, but actual files might still require permission. To check access, the best approach is to try downloading an actual data file path. I can also guide the user to use the HF UI to check gating easily and suggest a list of dataset repo IDs for them to request access.


**Clarifying dataset access and requirements**

I can mention that the `ai4bharat/*` datasets are gated, and users need to click "Request access." On the other hand, FLEURS and Common Voice are ungated and accessible right away. For benchmarking, it's easiest to use Vistaar's zipped benchmarks since they don’t need HF gating. The user asked about dataset repos, so I should focus on those but I might also touch on relevant HF model repositories, especially since they inquired about necessary repos for these datasets. I could include a mention of IndicSUPERB too.


**Discussing dataset access and recommendations**

I'm considering the necessary datasets, like "OpenSLR" or "NTU," but I should stick to those we reviewed. We definitely need to avoid sharing secrets from the `.env` file, but I can mention the variable names `HF_TOKEN` or `HUGGINGFACE_TOKEN`. I can instruct the user to check access using this token without sharing it. I'll also highlight that some datasets, like Common Voice, are substantial, so for benchmarking, using `streaming=True` may help. The user has asked for a list of Hugging Face repos needed for these datasets, so I'll focus on that request and consider searching for "indicwhisper" and "vistaar_benchmarks."


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=indicwhisper&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([f\"{d.get('id')} gated={d.get('gated')}\" for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:


Summary: auto


set -euo pipefail
curl -s "https://huggingface.co/api/datasets?search=vistaar_benchmarks&limit=20" | python3 -c "import sys,json; j=json.load(sys.stdin); print('\\n'.join([d.get('id','') for d in j]))"


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:


set -euo pipefail
curl -s "https://huggingface.co/api/datasets/ai4bharat/Kathbath/tree/main?recursive=0" | head


**Exploring file paths for testing**

I think I can list some file paths for testing, like `README.md`, but gating might restrict reading. It’s probably better to try accessing an actual data file, such as `hindi/train.tsv`, to confirm functionality. I could also look into the directory structure to dig deeper. So, I’ll inspect the `hindi` directory with the API to see what files are available.


set -euo pipefail
curl -s "https://huggingface.co/api/datasets/ai4bharat/Kathbath/tree/main/hindi?recursive=0" | head


**Exploring AI4Bharat Datasets**

I’m looking into the AI4Bharat datasets, many of which are gated for access but typically auto-approve after accepting the license, although it can take some time. I want to check gating for datasets like `ai4bharat/IndicVoices` and also consider others like `ai4bharat/IN22-Conv` for conversational data. The user is interested in datasets we recommended, such as Vistaar and Kathbath, but I need to see what’s available on HF specifically.


**Checking Dataset Availability**

I’m considering providing disclaimers about the MUCS and Gramvaani datasets, noting they're not consistently official on HF. Vistaar does offer benchmark zips, and there are third-party uploads, like `AdityK2409/mucs`, which I need to clarify aren’t official. I might want to search for "mucs ai4bharat" and "Gramvaani ai4bharat." Previously, we found no official datasets for Gramvaani, but I'm curious if there's an `ai4bharat/Gramvaani` that was just hard to find. Let’s check that link to see what happens!


set -euo pipefail
for id in "ai4bharat/Gramvaani" "ai4bharat/MUCS"; do
  echo "--- $id"
  status=$(curl -s -o /tmp/hf_resp.json -w "%{http_code}" "https://huggingface.co/api/datasets/$id" || true)
  echo "http=$status"
  if [ "$status" = "200" ]; then
    cat /tmp/hf_resp.json | python3 -c "import sys,json; j=json.load(sys.stdin); print('gated=', j.get('gated')); print('private=', j.get('private'))"
  fi
done


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
--- ai4bharat/Gramvaani
http=401
--- ai4bharat/MUCS
http=401


set -euo pipefail
status=$(curl -s -o /dev/null -w "%{http_code}" "https://huggingface.co/api/datasets/ai4bharat/DoesNotExist" || true)
echo $status


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
401


**Exploring dataset errors**

Hmm, it seems like I'm getting a 401 error for a dataset that doesn't exist, which is an interesting response. I wonder if I can test this further with another dataset that definitely doesn't exist but has a different author. This could help me understand if the same error persists or changes. It’s always good to explore these things to get a clearer picture of how the system behaves!


set -euo pipefail
status=$(curl -s -o /dev/null -w "%{http_code}" "https://huggingface.co/api/datasets/thisauthorprobablydoesntexist12345/doesntexist" || true)
echo $status


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
401


**Clarifying dataset requirements**

The current task involves the HF dataset repository list, even though it’s not specific to my skills. The user’s asking which Hugging Face repositories they need to access certain datasets. I wonder if they’re also interested in zipped datasets from Vistaar. It seems they might want the dataset repository list plus a method to check for access, specifically focusing on datasets. However, they mentioned Hugging Face, so I’ll stick to that for now.


**Structuring access instructions**

I’m working on crafting an answer that includes categories and direct commands to check dataset access. I’ll provide a quick access check example without needing to download the entire dataset.

One idea is to use curl to check the README.md for various dataset IDs, but I have to consider that the README might still be accessible without a token. Instead, I could check access to an LFS file like a parquet by making a HEAD request. I'll guide the user to locate a small file using a tree API call first.


**Simplifying dataset instructions**

I think a simpler approach would be to use the `datasets` library for loading datasets. I can show how to use a Python snippet to load and stream the dataset. The config names seem to be in full language names, like "hindi" instead of just "hi."

To verify the config names, I can suggest running a check, though that might require access. I’ll instruct the user to run the `load_dataset` command with `streaming=True` for testing without creating new files. It’s good to mention that users may also be interested in datasets for languages like Telugu or Hindi.


**Adding important disclaimers**

It's important to include some disclaimers about AI4Bharat datasets, as many are gated and require accepting terms, sometimes through Google Forms. For users who can't access those, I can suggest that they benchmark using open datasets like FLEURS and Common Voice, or zipped datasets from Vistaar.

I also need to mention that the pipeline transcribes offline and evaluates with those datasets. Although the user hasn't asked, I might point out model repositories they may need for things like `wav2vec2` or CTC models, as some may require access.


**Providing dataset access instructions**

I think I’ll create a clear response for the user regarding Hugging Face dataset repositories. First, I’ll provide a list of recommended datasets while noting any gating.

For quick access checks, I can suggest two options:
- Option A: Open each dataset page in the browser while logged in to Hugging Face to see if "Access repository" is enabled.
- Option B: Use `huggingface-cli login` and run a Python command to load config names or stream a sample, checking for any access errors.

I’ll also mention a few specific datasets, like `ai4bharat/Kathbath`, `google/fleurs`, and `mozilla-foundation/common_voice`. Finally, I want to ensure the response is concise and actionable.


For the benchmark datasets we discussed, these are the Hugging Face dataset repos to check access for:

- **AI4Bharat (these are `gated=auto` → you must be logged in and click “Agree/Request access” on HF):**
  - `ai4bharat/Kathbath`
  - `ai4bharat/IndicVoices`
  - `ai4bharat/indicvoices_r`
  - `ai4bharat/Lahaja`
  - `ai4bharat/Svarah`
  - (optional, not ASR) `ai4bharat/IndicVoices-ST`

- **Open / ungated (should work without requesting access):**
  - `google/fleurs`
  - `mozilla-foundation/common_voice_17_0` (or whichever CV version you use)
  - `collabora/indic-superb`

- **Not needed on HF if you use Vistaar’s benchmark zips:** the Vistaar benchmark suite is downloadable directly (no HF dataset access required).

Quick “do I actually have access?” check (doesn’t download everything):
- `python3 -c "from datasets import get_dataset_config_names; print(get_dataset_config_names('ai4bharat/Kathbath'))"`
- then try streaming one split: `python3 -c "from datasets import load_dataset; ds=load_dataset('ai4bharat/Kathbath','hindi',split='validation',streaming=True); print(next(iter(ds)))"`


# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
Key Fixes

MMS CTC now calls tokenizer.set_target_lang(<iso3>) when available and scores all candidates from a single forward pass (much faster + avoids <unk> blowups): ctc_fit.py (line 30).
Odia no longer crashes faster-whisper ('or' is not a valid language code); it falls back to auto-detect, and I added urd/san mappings: language.py (line 3).
Benchmarking Harness

New CLI: maya-transcribe bench build|run|push to sample HF datasets → write audio/ + refs.jsonl → run pipeline → write eval.json: cli.py (line 298), bench.py (line 171).
maya-transcribe run/bench run accept --thresholds-json so you can tune LID/verify gating without code edits: cli.py (line 208).
Eval can group/normalize by reference language via --language-source refs: eval.py (line 54), cli.py (line 273).
Added optional deps for this: pyproject.toml (line 20).
What I Ran

Built corpora: data/bench/maya_hf_v0 (initial) and deduped data/bench/maya_hf_v2 (200 samples × 12 languages).
Ran baseline on v0: eval_refs.json (line 1) (WER mean ≈ 0.958); biggest blockers are weak base ASR for several Indic scripts + conservative LID gating (many segments run with no language hint).
HF Dataset

Created private dataset (manifest-only; no audio redistribution) on your org: maya-research/maya-transcribe-bench-v2 via maya-transcribe bench push (--include-audio exists, but only use it if licenses allow): bench.py (line 329), cli.py (line 414).
If you tell me which ASR checkpoint(s) you want as the next attempt (single multilingual vs per-language IDs), I can run maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 ... and iterate on WER with a proper ablation matrix.


Summary: auto


I see the user has shared updates about their coding agent, mentioning various fixes and a need for input on ASR checkpoints. They’re not explicitly asking for changes, but it seems like they want feedback on the updates and suggestions for an ablation plan to minimize WER. I need to keep in mind the previous context about approval policies and environment settings. It sounds like I should prepare to provide thoughtful commentary based on their input!


The user seems to be looking for recommendations on which ASR checkpoints to try next, like Whisper large-v3 or IndicWhisper. However, I'm noticing a significant issue with their WER of about 0.958, which indicates major alignment problems or script mismatches. This could be due to whitespace tokenization issues with Indic scripts or other factors like empty transcripts or incorrect models being used. I need to analyze the new changes in their code, including files like `bench.py` and `cli.py`, to ensure everything compiles and runs properly.


I need to follow the instructions carefully: finish the code review first, and then suggest evaluations. The user has asked for another review round on their coding agent's updates and mentioned changes made to the code with a high WER. It looks like they might want comments on this. Given the local repo, I can run commands to locate relevant files and ensure they are robust. While I should avoid creating new files unless specifically asked, I can modify the code if needed.


I need to clarify that the user hasn't specifically asked to modify the code; rather, they seem to be looking for a review and suggestions. They posted a summary of changes, and it sounds like they want feedback on those and recommendations for the next models. Given the WER of about 0.958, which raises suspicion, I should analyze potential issues like evaluation tokenization, especially with Indic scripts. It could get tricky if the tokens use spaces incorrectly or if the transcripts have discrepancies in scripts or punctuation.


There’s a lot to unpack here! The user has modified prompts to avoid hallucinations and has mentioned issues with ASR models like Whisper regarding punctuation. I need to analyze the code and configuration files, particularly `eval_refs.json`, to see if the baseline run references are incorrect. The WER of 0.958 raises red flags, potentially due to tokenization issues with Indic languages. I should explore relevant ASR checkpoints, considering larger multilingual models and evaluate whether to use native scripts or transliterations for better accuracy in comparisons. This is key to minimizing WER effectively.


It seems the user has modified their prompts to maintain English in Latin, but they might still be using ASR models like Gemini or Whisper. However, the high WER suggests potential issues with dataset references, such as uppercase letters or non-speech markers. I need to review how the benchmark builds the audio and references, possibly using the `datasets` library. The user is seeking to check Hugging Face repositories for access to necessary datasets before running tests. I should also ensure the bench harness handles all aspects like licensing, caching, and transcript formatting robustly while checking for any potential issues in the code.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 112
drwxrwxr-x  8 ubuntu ubuntu  4096 Jan 13 06:26 .
drwxr-x--- 43 ubuntu ubuntu  4096 Jan 13 04:01 ..
-rw-rw-r--  1 ubuntu ubuntu 41725 Jan 13 04:08 Agents.md
drwxrwxr-x  4 ubuntu ubuntu  4096 Jan 13 07:15 data
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 03:12 docs
-rw-rw-r--  1 ubuntu ubuntu  3704 Jan 13 05:07 .env
-rw-rw-r--  1 ubuntu ubuntu   144 Jan 13 04:16 .gitignore
-rw-rw-r--  1 ubuntu ubuntu  9299 Jan 13 04:09 master.md
drwxrwxr-x  3 ubuntu ubuntu  4096 Jan 13 07:11 maya_transcribe
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 04:29 maya_transcribe.egg-info
drwxrwxr-x 12 ubuntu ubuntu  4096 Jan 13 07:31 outputs
-rw-rw-r--  1 ubuntu ubuntu  1090 Jan 13 06:13 Prompt.md
-rw-rw-r--  1 ubuntu ubuntu  1140 Jan 13 07:12 pyproject.toml
-rw-rw-r--  1 ubuntu ubuntu  1280 Jan 13 05:39 README.md
-rw-rw-r--  1 ubuntu ubuntu  2627 Jan 13 02:35 task.md
drwxrwxr-x  2 ubuntu ubuntu  4096 Jan 13 06:26 .vscode


**Exploring directories**

I’m looking into the data and outputs directories, noting that the bench harness probably utilizes the data/bench folder. It’s essential to keep track of these components as I organize my workflow. I want to make sure everything is orderly, so I'll list the maya_transcribe directory next. It seems like a small yet necessary step in getting a clearer picture of the directory structure. Let’s see what’s inside!


set -euo pipefail
ls -la maya_transcribe


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 184
drwxrwxr-x 3 ubuntu ubuntu  4096 Jan 13 07:11 .
drwxrwxr-x 8 ubuntu ubuntu  4096 Jan 13 06:26 ..
-rw-rw-r-- 1 ubuntu ubuntu  2181 Jan 13 04:24 align.py
-rw-rw-r-- 1 ubuntu ubuntu  6663 Jan 13 04:21 asr.py
-rw-rw-r-- 1 ubuntu ubuntu  4147 Jan 13 04:18 audio.py
-rw-rw-r-- 1 ubuntu ubuntu 14114 Jan 13 09:41 bench.py
-rw-rw-r-- 1 ubuntu ubuntu 18799 Jan 13 09:38 cli.py
-rw-rw-r-- 1 ubuntu ubuntu  1258 Jan 13 04:16 config.py
-rw-rw-r-- 1 ubuntu ubuntu 13909 Jan 13 07:09 ctc_fit.py
-rw-rw-r-- 1 ubuntu ubuntu   387 Jan 13 04:17 dataset.py
-rw-rw-r-- 1 ubuntu ubuntu  1508 Jan 13 04:37 db.py
-rw-rw-r-- 1 ubuntu ubuntu 10860 Jan 13 09:37 eval.py
-rw-rw-r-- 1 ubuntu ubuntu  9033 Jan 13 06:06 gemini_asr.py
-rw-rw-r-- 1 ubuntu ubuntu   992 Jan 13 06:16 ids.py
-rw-rw-r-- 1 ubuntu ubuntu    50 Jan 13 04:16 __init__.py
-rw-rw-r-- 1 ubuntu ubuntu   667 Jan 13 04:19 io.py
-rw-rw-r-- 1 ubuntu ubuntu  2434 Jan 13 09:21 language.py
-rw-rw-r-- 1 ubuntu ubuntu  2272 Jan 13 04:35 lid.py
-rw-rw-r-- 1 ubuntu ubuntu 15966 Jan 13 06:14 pipeline.py
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 13 09:41 __pycache__
-rw-rw-r-- 1 ubuntu ubuntu  3887 Jan 13 04:32 r2.py
-rw-rw-r-- 1 ubuntu ubuntu  6414 Jan 13 04:37 rover.py
-rw-rw-r-- 1 ubuntu ubuntu  1120 Jan 13 04:17 tar.py
-rw-rw-r-- 1 ubuntu ubuntu  1448 Jan 13 04:25 textnorm.py
-rw-rw-r-- 1 ubuntu ubuntu  6613 Jan 13 06:08 verify.py


**Inspecting Python files**

I see that bench.py and cli.py were updated at 09:41, which is good. I want to read bench.py to understand the harness, so I’ll open the relevant parts. To view line numbers, I'll use `sed` or `nl -ba`, opting for `sed -n '1,200p'` for a clean output. While I prefer using `rg` for searching, I’ll stick with `sed` or `nl` for viewing. I’ll check if python3 is available, then proceed to inspect bench.py.


set -euo pipefail
nl -ba maya_transcribe/bench.py | sed -n '1,240p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	import json
     4	import os
     5	import random
     6	import re
     7	import wave
     8	from dataclasses import dataclass
     9	from pathlib import Path
    10	from typing import Any
    11
    12	import numpy as np
    13
    14	from maya_transcribe.ids import segment_identity
    15	from maya_transcribe.io import read_jsonl, write_jsonl
    16
    17
    18	@dataclass(frozen=True)
    19	class HFSource:
    20	    dataset: str
    21	    config: str | None
    22	    split: str
    23	    audio_key: str
    24	    text_key: str
    25	    id_key: str | None = None
    26	    trust_remote_code: bool = False
    27	    name: str | None = None
    28
    29
    30	_SAFE_NAME_RE = re.compile(r"[^a-zA-Z0-9._-]+")
    31
    32
    33	def _safe_name(value: str) -> str:
    34	    value = (value or "").strip()
    35	    value = value.replace("/", "_")
    36	    value = value.replace("\\", "_")
    37	    value = _SAFE_NAME_RE.sub("_", value)
    38	    value = value.strip("._-")
    39	    return value or "x"
    40
    41
    42	def _ensure_hf_token_env() -> None:
    43	    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
    44	    if token and not os.getenv("HUGGINGFACE_HUB_TOKEN"):
    45	        os.environ["HUGGINGFACE_HUB_TOKEN"] = token
    46
    47
    48	def _write_wav_mono16(path: Path, audio: np.ndarray, sampling_rate: int) -> None:
    49	    path.parent.mkdir(parents=True, exist_ok=True)
    50
    51	    wav = np.asarray(audio, dtype=np.float32)
    52	    if wav.ndim == 2:
    53	        wav = wav.mean(axis=1)
    54	    wav = np.clip(wav, -1.0, 1.0)
    55	    pcm16 = (wav * 32767.0).astype(np.int16)
    56
    57	    with wave.open(str(path), "wb") as wf:
    58	        wf.setnchannels(1)
    59	        wf.setsampwidth(2)
    60	        wf.setframerate(int(sampling_rate))
    61	        wf.writeframes(pcm16.tobytes())
    62
    63
    64	def _extract_audio(example: dict[str, Any], audio_key: str) -> tuple[np.ndarray, int] | None:
    65	    audio = example.get(audio_key)
    66	    if not isinstance(audio, dict):
    67	        return None
    68	    arr = audio.get("array")
    69	    sr = audio.get("sampling_rate")
    70	    if arr is None or sr is None:
    71	        return None
    72	    try:
    73	        sr_i = int(sr)
    74	    except (TypeError, ValueError):
    75	        return None
    76	    return np.asarray(arr, dtype=np.float32), sr_i
    77
    78
    79	def _extract_text(example: dict[str, Any], text_key: str) -> str | None:
    80	    text = example.get(text_key)
    81	    if not isinstance(text, str):
    82	        return None
    83	    text = text.strip()
    84	    return text or None
    85
    86
    87	def default_sources() -> dict[str, list[HFSource]]:
    88	    """
    89	    Default benchmark sources for the 12 target languages:
    90	    Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, English.
    91	    """
    92	    kathbath = "ai4bharat/Kathbath"
    93	    fleurs = "google/fleurs"
    94
    95	    # Canonical ISO-639-3 codes aligned to MMS-LID output (and language.py mappings).
    96	    return {
    97	        "hin": [HFSource(dataset=kathbath, config="hindi", split="valid", audio_key="audio_filepath", text_key="text")],
    98	        "tam": [HFSource(dataset=kathbath, config="tamil", split="valid", audio_key="audio_filepath", text_key="text")],
    99	        "tel": [HFSource(dataset=kathbath, config="telugu", split="valid", audio_key="audio_filepath", text_key="text")],
   100	        "kan": [HFSource(dataset=kathbath, config="kannada", split="valid", audio_key="audio_filepath", text_key="text")],
   101	        "mal": [
   102	            HFSource(dataset=kathbath, config="malayalam", split="valid", audio_key="audio_filepath", text_key="text")
   103	        ],
   104	        "ben": [HFSource(dataset=kathbath, config="bengali", split="valid", audio_key="audio_filepath", text_key="text")],
   105	        "guj": [
   106	            HFSource(dataset=kathbath, config="gujarati", split="valid", audio_key="audio_filepath", text_key="text")
   107	        ],
   108	        "pan": [
   109	            HFSource(dataset=kathbath, config="punjabi", split="valid", audio_key="audio_filepath", text_key="text")
   110	        ],
   111	        "mar": [
   112	            HFSource(dataset=kathbath, config="marathi", split="valid", audio_key="audio_filepath", text_key="text")
   113	        ],
   114	        "ory": [HFSource(dataset=kathbath, config="odia", split="valid", audio_key="audio_filepath", text_key="text")],
   115	        # Kathbath doesn't include Assamese/English; use FLEURS (requires trust_remote_code=True).
   116	        "asm": [
   117	            HFSource(
   118	                dataset=fleurs,
   119	                config="as_in",
   120	                split="test",
   121	                audio_key="audio",
   122	                text_key="transcription",
   123	                trust_remote_code=True,
   124	            ),
   125	            HFSource(
   126	                dataset=fleurs,
   127	                config="as_in",
   128	                split="validation",
   129	                audio_key="audio",
   130	                text_key="transcription",
   131	                trust_remote_code=True,
   132	            ),
   133	            HFSource(
   134	                dataset=fleurs,
   135	                config="as_in",
   136	                split="train",
   137	                audio_key="audio",
   138	                text_key="transcription",
   139	                trust_remote_code=True,
   140	            ),
   141	        ],
   142	        "eng": [
   143	            HFSource(
   144	                dataset=fleurs,
   145	                config="en_us",
   146	                split="test",
   147	                audio_key="audio",
   148	                text_key="transcription",
   149	                trust_remote_code=True,
   150	            ),
   151	            HFSource(
   152	                dataset=fleurs,
   153	                config="en_us",
   154	                split="validation",
   155	                audio_key="audio",
   156	                text_key="transcription",
   157	                trust_remote_code=True,
   158	            ),
   159	            HFSource(
   160	                dataset=fleurs,
   161	                config="en_us",
   162	                split="train",
   163	                audio_key="audio",
   164	                text_key="transcription",
   165	                trust_remote_code=True,
   166	            ),
   167	        ],
   168	    }
   169
   170
   171	def build_benchmark_corpus(
   172	    *,
   173	    out_dir: Path,
   174	    samples_per_lang: int = 200,
   175	    seed: int = 0,
   176	    shuffle: bool = True,
   177	    langs: list[str] | None = None,
   178	    min_duration_sec: float = 1.0,
   179	    max_duration_sec: float = 30.0,
   180	    force: bool = False,
   181	) -> dict[str, Any]:
   182	    """
   183	    Materialize a local benchmark corpus:
   184	      - `out_dir/audio/<lang>/*.wav`
   185	      - `out_dir/refs.jsonl` with `segment_id`, `text`, `lang`, `audio_relpath`, `source_*`
   186	      - `out_dir/manifest.json` with build config + counts
   187	    """
   188	    _ensure_hf_token_env()
   189
   190	    if out_dir.exists():
   191	        existing = [p for p in out_dir.iterdir()]
   192	        if existing and not force:
   193	            raise RuntimeError(f"Refusing to overwrite non-empty directory: {out_dir} (pass force=True)")
   194	        if force:
   195	            for p in sorted(out_dir.rglob("*"), reverse=True):
   196	                if p.is_file() or p.is_symlink():
   197	                    p.unlink()
   198	                elif p.is_dir():
   199	                    p.rmdir()
   200
   201	    out_dir.mkdir(parents=True, exist_ok=True)
   202	    audio_root = out_dir / "audio"
   203
   204	    sources_by_lang = default_sources()
   205	    target_langs = langs or sorted(sources_by_lang.keys())
   206	    target_langs = [l.strip().lower() for l in target_langs if l.strip()]
   207
   208	    try:
   209	        from datasets import load_dataset
   210	    except Exception as e:  # pragma: no cover
   211	        raise RuntimeError("Benchmark corpus build requires `datasets`. Install with: pip install -e '.[bench]'") from e
   212
   213	    rows: list[dict[str, Any]] = []
   214	    counts: dict[str, int] = {}
   215	    skipped: dict[str, int] = {}
   216
   217	    for lang in target_langs:
   218	        sources = sources_by_lang.get(lang)
   219	        if not sources:
   220	            continue
   221	        remaining = int(samples_per_lang)
   222	        written = 0
   223	        skipped_lang = 0
   224
   225	        for src in sources:
   226	            if remaining <= 0:
   227	                break
   228
   229	            ds = load_dataset(
   230	                src.dataset,
   231	                src.config,
   232	                split=src.split,
   233	                trust_remote_code=bool(src.trust_remote_code),
   234	            )
   235	            idxs = list(range(len(ds)))
   236	            if shuffle:
   237	                # Stable per-source shuffling (seeded by global seed + source identity).
   238	                s = f"{seed}|{lang}|{src.dataset}|{src.config}|{src.split}"
   239	                srng = random.Random(int.from_bytes(s.encode("utf-8"), "little") % (2**32))
   240	                srng.shuffle(idxs)


set -euo pipefail
nl -ba maya_transcribe/bench.py | sed -n '240,520p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   240	                srng.shuffle(idxs)
   241
   242	            for i in idxs:
   243	                if remaining <= 0:
   244	                    break
   245
   246	                ex = ds[int(i)]
   247	                if not isinstance(ex, dict):
   248	                    continue
   249
   250	                text = _extract_text(ex, src.text_key)
   251	                if not text:
   252	                    skipped_lang += 1
   253	                    continue
   254
   255	                audio_ex = _extract_audio(ex, src.audio_key)
   256	                if audio_ex is None:
   257	                    skipped_lang += 1
   258	                    continue
   259	                audio_arr, sr = audio_ex
   260	                dur = float(audio_arr.shape[0]) / float(sr) if sr else 0.0
   261	                if dur < float(min_duration_sec) or dur > float(max_duration_sec):
   262	                    skipped_lang += 1
   263	                    continue
   264
   265	                src_id = None
   266	                if src.id_key and isinstance(ex.get(src.id_key), (str, int)):
   267	                    src_id = str(ex.get(src.id_key))
   268	                elif isinstance(ex.get("id"), (str, int)):
   269	                    src_id = str(ex.get("id"))
   270	                elif isinstance(ex.get("fname"), str):
   271	                    src_id = ex.get("fname")
   272	                elif isinstance(ex.get("path"), str):
   273	                    src_id = ex.get("path")
   274
   275	                src_name = src.name or _safe_name(src.dataset.split("/")[-1])
   276	                src_key = _safe_name("__".join([src_name, src.config or "default", src.split]))
   277	                lang_dir = audio_root / lang
   278	                fname = _safe_name(f"{src_key}__{i}__{src_id or 'x'}")
   279	                wav_path = lang_dir / f"{fname}.wav"
   280
   281	                _write_wav_mono16(wav_path, audio_arr, sr)
   282	                ident = segment_identity(wav_path, root=audio_root)
   283
   284	                audio_relpath = wav_path.relative_to(audio_root).as_posix()
   285	                rows.append(
   286	                    {
   287	                        "segment_id": ident.segment_id,
   288	                        "text": text,
   289	                        "lang": lang,
   290	                        "audio_relpath": audio_relpath,
   291	                        "duration_sec": dur,
   292	                        "source_dataset": src.dataset,
   293	                        "source_config": src.config,
   294	                        "source_split": src.split,
   295	                        "source_id": src_id,
   296	                        "source_index": int(i),
   297	                    }
   298	                )
   299
   300	                remaining -= 1
   301	                written += 1
   302
   303	        counts[lang] = written
   304	        skipped[lang] = skipped_lang
   305
   306	    refs_path = out_dir / "refs.jsonl"
   307	    write_jsonl(refs_path, rows)
   308
   309	    manifest = {
   310	        "out_dir": str(out_dir),
   311	        "audio_root": str(audio_root),
   312	        "samples_per_lang": int(samples_per_lang),
   313	        "seed": int(seed),
   314	        "shuffle": bool(shuffle),
   315	        "langs": target_langs,
   316	        "min_duration_sec": float(min_duration_sec),
   317	        "max_duration_sec": float(max_duration_sec),
   318	        "counts": counts,
   319	        "skipped": skipped,
   320	        "total_rows": len(rows),
   321	        "sources_by_lang": {
   322	            k: [vars(s) for s in v] for k, v in sources_by_lang.items() if (not langs or k in set(target_langs))
   323	        },
   324	    }
   325	    (out_dir / "manifest.json").write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
   326	    return manifest
   327
   328
   329	def push_benchmark_to_hub(
   330	    *,
   331	    bench_dir: Path,
   332	    repo_id: str,
   333	    private: bool = True,
   334	    include_audio: bool = False,
   335	    split: str = "test",
   336	) -> str:
   337	    """
   338	    Push the benchmark corpus as a Hugging Face dataset repo.
   339
   340	    If `include_audio` is False, uploads only the JSONL manifest rows (no audio redistribution).
   341	    """
   342	    _ensure_hf_token_env()
   343	    token = os.getenv("HUGGINGFACE_HUB_TOKEN") or os.getenv("HF_TOKEN")
   344	    if not token:
   345	        raise RuntimeError("Missing HF auth token (set HF_TOKEN or HUGGINGFACE_HUB_TOKEN).")
   346
   347	    try:
   348	        from huggingface_hub import HfApi
   349	    except Exception as e:  # pragma: no cover
   350	        raise RuntimeError("Push-to-hub requires `huggingface_hub`. Install with: pip install -e '.[bench]'") from e
   351
   352	    try:
   353	        from datasets import Audio, Dataset, DatasetDict
   354	    except Exception as e:  # pragma: no cover
   355	        raise RuntimeError("Push-to-hub requires `datasets`. Install with: pip install -e '.[bench]'") from e
   356
   357	    refs_path = bench_dir / "refs.jsonl"
   358	    if not refs_path.exists():
   359	        raise RuntimeError(f"Missing refs.jsonl: {refs_path}")
   360	    rows = read_jsonl(refs_path)
   361
   362	    audio_root = bench_dir / "audio"
   363	    if include_audio and not audio_root.exists():
   364	        raise RuntimeError(f"Missing audio/ directory: {audio_root}")
   365
   366	    records: dict[str, list[Any]] = {
   367	        "segment_id": [],
   368	        "lang": [],
   369	        "text": [],
   370	        "audio_relpath": [],
   371	        "duration_sec": [],
   372	        "source_dataset": [],
   373	        "source_config": [],
   374	        "source_split": [],
   375	        "source_id": [],
   376	        "source_index": [],
   377	    }
   378	    audio_paths: list[str] = []
   379
   380	    for r in rows:
   381	        records["segment_id"].append(r.get("segment_id"))
   382	        records["lang"].append(r.get("lang"))
   383	        records["text"].append(r.get("text"))
   384	        records["audio_relpath"].append(r.get("audio_relpath"))
   385	        records["duration_sec"].append(r.get("duration_sec"))
   386	        records["source_dataset"].append(r.get("source_dataset"))
   387	        records["source_config"].append(r.get("source_config"))
   388	        records["source_split"].append(r.get("source_split"))
   389	        records["source_id"].append(r.get("source_id"))
   390	        records["source_index"].append(r.get("source_index"))
   391
   392	        if include_audio:
   393	            rel = str(r.get("audio_relpath") or "")
   394	            audio_paths.append(str(audio_root / rel))
   395
   396	    if include_audio:
   397	        records["audio"] = audio_paths
   398
   399	    ds = Dataset.from_dict(records)
   400	    if include_audio:
   401	        ds = ds.cast_column("audio", Audio())
   402
   403	    api = HfApi(token=token)
   404	    api.create_repo(repo_id=repo_id, repo_type="dataset", private=bool(private), exist_ok=True)
   405
   406	    dsd = DatasetDict({split: ds})
   407	    dsd.push_to_hub(repo_id=repo_id, token=token, private=bool(private))
   408	    return repo_id


**Examining CLI modifications**

I'm diving into the CLI modifications in `maya_transcribe/cli.py`, especially focusing on bench commands. It's interesting that while they use HfApi.create_repo, I'm noticing they don't include the dataset card or license, which I thought were important but apparently are optional. I wonder if that could lead to potential issues down the line or if it’s a common practice. I'll need to check this further!


set -euo pipefail
nl -ba maya_transcribe/cli.py | sed -n '1,260p'