Output:
Exit code: 0
Wall time: 0 seconds
Output:
# maya-transcribe

Offline, quality-first transcription pipeline for cleaned single-speaker audio segments (Indic + English).

## Install

```bash
python3 -m pip install -U pip
python3 -m pip install -e '.[full]'
```

If you only need R2 download + evaluation (no ASR), install the base deps:

```bash
python3 -m pip install -e .
```

## Configure

Create / update `.env` in the repo root. The CLI auto-loads it.

Minimum for R2:
- `R2_ENDPOINT_URL`
- `R2_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY`
- `R2_BUCKET` (defaults to `1-cleaned-data` if unset)
- `R2_PREFIX` (optional)

## Usage

```bash
maya-transcribe --help
maya-transcribe download --video-id <VIDEO_ID> --bucket 1-cleaned-data --out data/
maya-transcribe run --segments-dir data/<VIDEO_ID>/**/segments --out outputs/<VIDEO_ID> --whisper-model large-v3
```

Evaluation (requires a reference JSONL keyed by `segment_id`):

```bash
maya-transcribe eval --results outputs/<VIDEO_ID>/results.jsonl --refs refs.jsonl
```

Optional extras:
- Add additional Whisper-family checkpoints: `--transformers-whisper-model <hf_model_id>` (repeatable)
- Add Gemini as an extra hypothesis: `--gemini-model gemini-2.0-flash`
- Enable audio-grounded CTC fit scoring: `--ctc-model <hf_ctc_model_id>` (or `--ctc-model-map ctc_models.json`)


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Inspecting thresholds**

I need to check for bench information since it seems to be missing. I also have to consider thresholds in both `eval.py` and `verify.py`. It's important to inspect `verify.py` closely to figure out if there are any issues or specific parameters that need attention. I want to make sure I’m covering all my bases before moving forward! Let's dive into that and see what's going on in there.


sed -n '1,260p' maya_transcribe/verify.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from typing import Any

from maya_transcribe.lid import LIDResult

@dataclass(frozen=True)
class Verification:
    decision: str  # "accept" | "human" | "reject"
    reasons: list[str]
    scores: dict[str, Any]

def lid_confidence(lid: LIDResult | None) -> dict[str, Any]:
    if not lid or not lid.predictions:
        return {"available": False, "top1": None, "top1_prob": None, "margin": None, "confident": False}
    top1 = lid.predictions[0]
    top2_prob = lid.predictions[1].prob if len(lid.predictions) > 1 else 0.0
    margin = float(top1.prob) - float(top2_prob)
    return {
        "available": True,
        "top1": top1.code,
        "top1_prob": float(top1.prob),
        "margin": margin,
        "confident": False,
    }

def verify_segment(
    *,
    qc: dict[str, Any],
    lid: LIDResult | None,
    final_text: str,
    rover: dict[str, Any],
    asr_summaries: dict[str, Any],
    primary_asr_key: str | None = None,
    ctc: dict[str, Any] | None = None,
    thresholds: dict[str, Any] | None = None,
) -> Verification:
    """
    Conservative verification gate (precision-first).
    """
    thresholds = thresholds or {}
    min_duration = float(thresholds.get("min_duration_sec", 0.8))
    max_clipping = float(thresholds.get("max_clipping_fraction", 0.01))
    max_disagreement = float(thresholds.get("max_rover_disagreement_rate", 0.03))
    lid_min_prob = float(thresholds.get("lid_min_prob", 0.90))
    lid_min_margin = float(thresholds.get("lid_min_margin", 0.20))
    require_lid = bool(thresholds.get("require_lid", True))

    # Whisper-style failure heuristics (see OpenAI whisper fallback logic).
    min_avg_logprob = float(thresholds.get("min_avg_logprob_mean", -1.0))
    max_compression_ratio = float(thresholds.get("max_compression_ratio_mean", 2.4))

    # CTC fit (audio-grounded) gating.
    require_ctc = bool(thresholds.get("require_ctc", False))
    min_ctc_margin = float(thresholds.get("min_ctc_margin", 0.0))
    min_ctc_token_coverage = float(thresholds.get("min_ctc_token_coverage", 0.85))
    min_ctc_token_score_mean = float(thresholds.get("min_ctc_token_score_mean", -3.0))
    fail_on_ctc_span_mismatch = bool(thresholds.get("fail_on_ctc_span_mismatch", True))

    reasons: list[str] = []
    decision = "accept"

    duration = float(qc.get("duration_sec") or 0.0)
    clipping = float(qc.get("clipping_fraction") or 0.0)
    if duration < min_duration:
        decision = "reject"
        reasons.append("too_short")
    if clipping > max_clipping:
        decision = "human" if decision != "reject" else decision
        reasons.append("clipping")

    lidc = lid_confidence(lid)
    if lidc["available"]:
        lidc["confident"] = bool(lidc["top1_prob"] >= lid_min_prob and lidc["margin"] >= lid_min_margin)
        if not lidc["confident"]:
            decision = "human" if decision != "reject" else decision
            reasons.append("lid_uncertain")
    else:
        reasons.append("lid_missing")
        if require_lid:
            decision = "human" if decision != "reject" else decision

    disagreement_rate = float(rover.get("disagreement_rate") or 0.0)
    if disagreement_rate > max_disagreement:
        decision = "human" if decision != "reject" else decision
        reasons.append("asr_disagreement")

    final_text = (final_text or "").strip()
    if "<unk>" in final_text or "<inaudible>" in final_text:
        decision = "human" if decision != "reject" else decision
        reasons.append("contains_unk")

    # Whisper-style safety: flag if any decoder claims "no speech" with high probability.
    no_speech_probs = []
    for summary in asr_summaries.values():
        p = summary.get("no_speech_prob_mean")
        if p is None:
            continue
        no_speech_probs.append(float(p))
    if no_speech_probs and max(no_speech_probs) > 0.8:
        decision = "reject"
        reasons.append("no_speech")

    # Whisper-style hallucination heuristics: apply to the primary ASR (or rover skeleton fallback).
    asr_key_for_heuristics = primary_asr_key or str(rover.get("skeleton_key") or "")
    heur = asr_summaries.get(asr_key_for_heuristics, {}) if asr_key_for_heuristics else {}
    avg_lp = heur.get("avg_logprob_mean")
    comp = heur.get("compression_ratio_mean")
    if avg_lp is not None and float(avg_lp) < min_avg_logprob:
        decision = "human" if decision != "reject" else decision
        reasons.append("low_logprob")
    if comp is not None and float(comp) > max_compression_ratio:
        decision = "human" if decision != "reject" else decision
        reasons.append("high_compression_ratio")

    # Audio-grounded fit check (when available).
    if require_ctc and (ctc is None or ctc.get("best_loss") is None):
        decision = "human" if decision != "reject" else decision
        reasons.append("ctc_missing")
    if ctc is not None:
        margin = ctc.get("margin")
        if margin is not None and float(margin) < min_ctc_margin:
            decision = "human" if decision != "reject" else decision
            reasons.append("ctc_ambiguous")

        best_key = ctc.get("best_key")
        scores = ctc.get("scores")
        if isinstance(best_key, str) and isinstance(scores, dict):
            best = scores.get(best_key) or {}
            if isinstance(best, dict):
                cov = best.get("token_coverage")
                if cov is not None and float(cov) < min_ctc_token_coverage:
                    decision = "human" if decision != "reject" else decision
                    reasons.append("ctc_low_coverage")
                mean_tok = best.get("token_score_mean")
                if mean_tok is not None and float(mean_tok) < min_ctc_token_score_mean:
                    decision = "human" if decision != "reject" else decision
                    reasons.append("ctc_low_token_score")
                if fail_on_ctc_span_mismatch and bool(best.get("span_mismatch")):
                    decision = "human" if decision != "reject" else decision
                    reasons.append("ctc_span_mismatch")

    scores = {
        "duration_sec": duration,
        "clipping_fraction": clipping,
        "rover_disagreement_rate": disagreement_rate,
        "lid": lidc,
        "no_speech_prob_max": max(no_speech_probs) if no_speech_probs else None,
        "heuristics_asr_key": asr_key_for_heuristics or None,
        "avg_logprob_mean": avg_lp,
        "compression_ratio_mean": comp,
        "ctc": ctc,
    }

    return Verification(decision=decision, reasons=reasons, scores=scores)


Summary: auto


**Verifying WER segments**

I’m looking into verifying segments, which isn’t directly affecting Word Error Rate (WER). For benchmarking, we’ll evaluate WER based on final transcripts, independent of human decisions. The user wants minimal WER, so maybe we can adjust the pipeline to choose the best candidate using CTC scoring. I also think we might need to tweak text normalization to better handle punctuation for Indic languages. So now, I'll build a benchmark harness and inspect how evaluation works in `eval.py`.


nl -ba maya_transcribe/eval.py | sed -n '1,260p'


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 260
Output:
     1	from __future__ import annotations
     2
     3	import json
     4	from dataclasses import dataclass
     5	from pathlib import Path
     6	from typing import Any
     7
     8	from jiwer import cer, wer
     9
    10	from maya_transcribe.io import read_jsonl
    11	from maya_transcribe.textnorm import normalize_text
    12
    13
    14	@dataclass(frozen=True)
    15	class EvalRow:
    16	    segment_id: str
    17	    language: str | None
    18	    decision: str | None
    19	    duration_sec: float | None
    20	    duration_bucket: str | None
    21	    ref: str
    22	    hyp: str
    23	    wer: float
    24	    cer: float
    25	    oracle_best_wer: float | None = None
    26	    oracle_best_cer: float | None = None
    27	    oracle_best_key: str | None = None
    28	    selection_gap_wer: float | None = None
    29
    30
    31	def load_refs(refs_path: Path) -> dict[str, dict[str, Any]]:
    32	    """
    33	    Read a JSONL file of references.
    34	    Expected keys (any one): `text`, `transcript`, `reference`.
    35	    Required key: `segment_id` (or `id`).
    36	    """
    37	    refs: dict[str, dict[str, Any]] = {}
    38	    for row in read_jsonl(refs_path):
    39	        segment_id = str(row.get("segment_id") or row.get("id") or "").strip()
    40	        if not segment_id:
    41	            continue
    42	        refs[segment_id] = row
    43	    return refs
    44
    45
    46	def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    47	    for k in keys:
    48	        v = row.get(k)
    49	        if isinstance(v, str) and v.strip():
    50	            return v
    51	    return None
    52
    53
    54	def evaluate(
    55	    results_path: Path,
    56	    refs_path: Path,
    57	    out_path: Path | None = None,
    58	    hypothesis: str = "final",
    59	    oracle: bool = True,
    60	) -> dict[str, Any]:
    61	    refs = load_refs(refs_path)
    62	    results = read_jsonl(results_path)
    63
    64	    rows: list[EvalRow] = []
    65	    for r in results:
    66	        segment_id = str(r.get("segment_id") or "").strip()
    67	        if not segment_id or segment_id not in refs:
    68	            continue
    69
    70	        ref_row = refs[segment_id]
    71	        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
    72	        if ref_text is None:
    73	            continue
    74
    75	        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
    76	        if hyp_text is None:
    77	            continue
    78
    79	        lang = None
    80	        route = r.get("route") or {}
    81	        if isinstance(route, dict):
    82	            lang = route.get("language_hint")
    83
    84	        decision = None
    85	        ver = r.get("verification") or {}
    86	        if isinstance(ver, dict):
    87	            decision = ver.get("decision")
    88
    89	        duration_sec = None
    90	        qc = r.get("qc") or {}
    91	        if isinstance(qc, dict):
    92	            try:
    93	                duration_sec = float(qc.get("duration_sec")) if qc.get("duration_sec") is not None else None
    94	            except (TypeError, ValueError):
    95	                duration_sec = None
    96	        bucket = duration_bucket(duration_sec)
    97
    98	        ref_norm = normalize_text(ref_text, lang=lang)
    99	        hyp_norm = normalize_text(hyp_text, lang=lang)
   100	        w = float(wer(ref_norm, hyp_norm))
   101	        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))
   102
   103	        oracle_best_wer = None
   104	        oracle_best_cer = None
   105	        oracle_best_key = None
   106	        selection_gap_wer = None
   107
   108	        if oracle:
   109	            cand_wer: dict[str, float] = {}
   110	            cand_cer: dict[str, float] = {}
   111	            for key, text in extract_all_candidates(result_row=r).items():
   112	                ref_n = ref_norm
   113	                hyp_n = normalize_text(text, lang=lang)
   114	                cand_wer[key] = float(wer(ref_n, hyp_n))
   115	                cand_cer[key] = float(cer(ref_n.replace(" ", ""), hyp_n.replace(" ", "")))
   116
   117	            if cand_wer:
   118	                oracle_best_key, oracle_best_wer = min(cand_wer.items(), key=lambda kv: kv[1])
   119	                oracle_best_cer = cand_cer.get(oracle_best_key)
   120	                selection_gap_wer = float(w) - float(oracle_best_wer)
   121
   122	        rows.append(
   123	            EvalRow(
   124	                segment_id=segment_id,
   125	                language=lang,
   126	                decision=str(decision) if decision is not None else None,
   127	                duration_sec=duration_sec,
   128	                duration_bucket=bucket,
   129	                ref=ref_text,
   130	                hyp=hyp_text,
   131	                wer=w,
   132	                cer=c,
   133…1578 chars truncated…      "pairs_evaluated": len(rows),
   166	        "wer_mean": (sum(r.wer for r in rows) / len(rows)) if rows else None,
   167	        "cer_mean": (sum(r.cer for r in rows) / len(rows)) if rows else None,
   168	        "wer_p50": _percentile([r.wer for r in rows], 0.50),
   169	        "wer_p90": _percentile([r.wer for r in rows], 0.90),
   170	        "wer_p95": _percentile([r.wer for r in rows], 0.95),
   171	        "cer_p50": _percentile([r.cer for r in rows], 0.50),
   172	        "cer_p90": _percentile([r.cer for r in rows], 0.90),
   173	        "cer_p95": _percentile([r.cer for r in rows], 0.95),
   174	        "segment_error_rate": (sum(1 for r in rows if r.wer > 0.0) / len(rows)) if rows else None,
   175	        "pct_wer_gt_5pct": (sum(1 for r in rows if r.wer > 0.05) / len(rows)) if rows else None,
   176	        "pct_wer_gt_10pct": (sum(1 for r in rows if r.wer > 0.10) / len(rows)) if rows else None,
   177	        "wer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang.items())},
   178	        "wer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec.items())},
   179	        "wer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket.items())},
   180	        "cer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang_cer.items())},
   181	        "cer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec_cer.items())},
   182	        "cer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket_cer.items())},
   183	        "oracle": {
   184	            "enabled": bool(oracle),
   185	            "oracle_best_wer_mean": (sum(oracle_best_wers) / len(oracle_best_wers)) if oracle_best_wers else None,
   186	            "selection_gap_wer_mean": (sum(selection_gaps) / len(selection_gaps)) if selection_gaps else None,
   187	            "selection_gap_wer_p90": _percentile(selection_gaps, 0.90),
   188	        },
   189	    }
   190
   191	    if out_path is not None:
   192	        out_path.parent.mkdir(parents=True, exist_ok=True)
   193	        out_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
   194
   195	    return summary
   196
   197
   198	def extract_hypothesis_text(result_row: dict[str, Any], hypothesis: str) -> str | None:
   199	    """
   200	    hypothesis:
   201	      - "final": uses result_row["final"]["text"] (falls back to rover if missing)
   202	      - "rover" (default): uses result_row["rover"]["text"]
   203	      - "asr:<key>": uses result_row["asr"][<key>]["text"]
   204	    """
   205	    hypothesis = (hypothesis or "rover").strip()
   206	    if hypothesis == "final":
   207	        final = result_row.get("final") or {}
   208	        if isinstance(final, dict) and isinstance(final.get("text"), str):
   209	            return final["text"]
   210	        hypothesis = "rover"
   211	    if hypothesis == "rover":
   212	        rover = result_row.get("rover") or {}
   213	        if isinstance(rover, dict) and isinstance(rover.get("text"), str):
   214	            return rover["text"]
   215	        return None
   216
   217	    if hypothesis.startswith("asr:"):
   218	        key = hypothesis[len("asr:") :].strip()
   219	        asr = result_row.get("asr") or {}
   220	        if isinstance(asr, dict):
   221	            entry = asr.get(key) or {}
   222	            if isinstance(entry, dict) and isinstance(entry.get("text"), str):
   223	                return entry["text"]
   224	        return None
   225
   226	    return None
   227
   228
   229	def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:
   230	    """
   231	    Collect all text candidates available in a pipeline result row.
   232	    Keys are stable and namespaced (final/rover/asr:<key>).
   233	    """
   234	    candidates: dict[str, str] = {}
   235
   236	    final = extract_hypothesis_text(result_row, "final")
   237	    if final is not None:
   238	        candidates["final"] = final
   239
   240	    rover = extract_hypothesis_text(result_row, "rover")
   241	    if rover is not None:
   242	        candidates["rover"] = rover
   243
   244	    asr = result_row.get("asr") or {}
   245	    if isinstance(asr, dict):
   246	        for key, entry in asr.items():
   247	            if not isinstance(key, str):
   248	                continue
   249	            if not isinstance(entry, dict):
   250	                continue
   251	            text = entry.get("text")
   252	            if isinstance(text, str) and text.strip():
   253	                candidates[f"asr:{key}"] = text
   254
   255	    # Also include normalized hypotheses if present (useful for debugging).
   256	    normalized = result_row.get("normalized") or {}
   257	    if isinstance(normalized, dict):
   258	        for key, text in normalized.items():
   259	            if not isinstance(key, str) or not isinstance(text, str):
   260	                continue


nl -ba maya_transcribe/asr.py | sed -n '1,280p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
     1	from __future__ import annotations
     2
     3	from dataclasses import dataclass
     4	from pathlib import Path
     5	from typing import Any
     6
     7	from maya_transcribe.language import to_whisper_lang
     8
     9
    10	@dataclass(frozen=True)
    11	class ASRSegment:
    12	    start: float
    13	    end: float
    14	    text: str
    15	    avg_logprob: float | None = None
    16	    no_speech_prob: float | None = None
    17	    compression_ratio: float | None = None
    18
    19
    20	@dataclass(frozen=True)
    21	class ASRResult:
    22	    model: str
    23	    text: str
    24	    language: str | None
    25	    language_probability: float | None
    26	    segments: list[ASRSegment]
    27	    meta: dict[str, Any]
    28
    29
    30	class FasterWhisperASR:
    31	    def __init__(
    32	        self,
    33	        model_size_or_path: str = "large-v3",
    34	        device: str | None = None,
    35	        compute_type: str | None = None,
    36	    ) -> None:
    37	        try:
    38	            from faster_whisper import WhisperModel
    39	        except Exception as e:  # pragma: no cover
    40	            raise RuntimeError(
    41	                "faster-whisper is not installed. Install with: pip install -e '.[asr]'"
    42	            ) from e
    43
    44	        if device is None:
    45	            try:
    46	                import torch
    47
    48	                device = "cuda" if torch.cuda.is_available() else "cpu"
    49	            except Exception:
    50	                device = "cpu"
    51
    52	        if compute_type is None:
    53	            compute_type = "float16" if device == "cuda" else "float32"
    54
    55	        self.model_size_or_path = model_size_or_path
    56	        self.device = device
    57	        self.compute_type = compute_type
    58	        self._model = WhisperModel(model_size_or_path, device=device, compute_type=compute_type)
    59
    60	    @property
    61	    def name(self) -> str:
    62	        return f"faster-whisper:{self.model_size_or_path}"
    63
    64	    def transcribe(
    65	        self,
    66	        audio_path: Path,
    67	        language: str | None = None,
    68	        beam_size: int = 5,
    69	        best_of: int = 5,
    70	        temperature: float = 0.0,
    71	        vad_filter: bool = False,
    72	    ) -> ASRResult:
    73	        lang = to_whisper_lang(language)
    74
    75	        segments_iter, info = self._model.transcribe(
    76	            str(audio_path),
    77	            language=lang,
    78	            beam_size=beam_size,
    79	            best_of=best_of,
    80	            temperature=temperature,
    81	            vad_filter=vad_filter,
    82	        )
    83
    84	        segments: list[ASRSegment] = []
    85	        texts: list[str] = []
    86	        for s in segments_iter:
    87	            texts.append(s.text)
    88	            segments.append(
    89	                ASRSegment(
    90	                    start=float(s.start),
    91	                    end=float(s.end),
    92	                    text=str(s.text),
    93	                    avg_logprob=getattr(s, "avg_logprob", None),
    94	                    no_speech_prob=getattr(s, "no_speech_prob", None),
    95	                    compression_ratio=getattr(s, "compression_ratio", None),
    96	                )
    97	            )
    98
    99	        return ASRResult(
   100	            model=self.name,
   101	            text="".join(texts).strip(),
   102	            language=getattr(info, "language", None),
   103	            language_probability=getattr(info, "language_probability", None),
   104	            segments=segments,
   105	            meta={
   106	                "device": self.device,
   107	                "compute_type": self.compute_type,
   108	                "beam_size": beam_size,
   109	                "best_of": best_of,
   110	                "temperature": temperature,
   111	                "vad_filter": vad_filter,
   112	            },
   113	        )
   114
   115
   116	class TransformersWhisperASR:
   117	    """
   118	    Whisper-family ASR using HuggingFace Transformers (useful for fine-tuned checkpoints like IndicWhisper).
   119	    """
   120
   121	    def __init__(
   122	        self,
   123	        model_id: str,
   124	        device: str | None = None,
   125	        dtype: str | None = None,
   126	    ) -> None:
   127	        try:
   128	            import torch
   129	            from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
   130	        except Exception as e:  # pragma: no cover
   131	            raise RuntimeError(
   132	                "TransformersWhisperASR requires the optional dependencies: torch + transformers. "
   133	                "Install with: pip install -e '.[torch]'"
   134	            ) from e
   135
   136	        if device is None:
   137	            device = "cuda" if torch.cuda.is_available() else "cpu"
   138	        if dtype is None:
   139	            dtype = "float16" if (device == "cuda") else "float32"
   140
   141	        torch_dtype = torch.float16 if dtype == "float16" else torch.float32
   142
   143	        self.model_id = model_id
   144	        self.device = device
   145	        self.dtype = dtype
   146	        self._torch = torch
   147	        self._processor = AutoProcessor.from_pretrained(model_id)
   148	        self._model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype).to(device)
   149	        self._model.eval()
   150
   151	    @property
   152	    def name(self) -> str:
   153	        return f"transformers:{self.model_id}"
   154
   155	    def transcribe(
   156	        self,
   157	        audio_path: Path,
   158	        language: str | None = None,
   159	        max_new_tokens: int = 256,
   160	        num_beams: int = 5,
   161	    ) -> ASRResult:
   162	        from maya_transcribe.audio import decode_pcm16
   163
   164	        wav, sr = decode_pcm16(audio_path, sample_rate=16000, mono=True)
   165	        if wav.size == 0:
   166	            return ASRResult(
   167	                model=self.name,
   168	                text="",
   169	                language=language,
   170	                language_probability=None,
   171	                segments=[],
   172	                meta={"max_new_tokens": max_new_tokens, "num_beams": num_beams},
   173	            )
   174
   175	        inputs = self._processor(wav, sampling_rate=sr, return_tensors="pt")
   176	        input_features = inputs.get("input_features")
   177	        if input_features is None:
   178	            raise RuntimeError("Processor did not return input_features")
   179	        input_features = input_features.to(self.device)
   180
   181	        forced_decoder_ids = None
   182	        whisper_lang = to_whisper_lang(language)
   183	        if whisper_lang:
   184	            get_ids = getattr(self._processor, "get_decoder_prompt_ids", None)
   185	            if callable(get_ids):
   186	                forced_decoder_ids = get_ids(language=whisper_lang, task="transcribe")
   187
   188	        with self._torch.no_grad():
   189	            generated_ids = self._model.generate(
   190	                input_features,
   191	                forced_decoder_ids=forced_decoder_ids,
   192	                max_new_tokens=max_new_tokens,
   193	                num_beams=num_beams,
   194	            )
   195
   196	        text = self._processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
   197	        return ASRResult(
   198	            model=self.name,
   199	            text=text.strip(),
   200	            language=whisper_lang or language,
   201	            language_probability=None,
   202	            segments=[],
   203	            meta={
   204	                "device": self.device,
   205	                "dtype": self.dtype,
   206	                "max_new_tokens": max_new_tokens,
   207	                "num_beams": num_beams,
   208	            },
   209	        )


**Planning benchmark tests**

I see that the pipeline already runs at least two hypotheses with faster-whisper. Since the user seeks minimal Word Error Rate (WER), I could add additional transformer models, like IndicWhisper or `sanchit-gandhi/whisper-medium-hi`. It sounds like we might need to build a benchmark matrix with optional models per language, but which model is best? I'm unsure if searching for that is possible, but I might explore using the Hugging Face API. The user also requested benchmarks with several hundred samples for each language, ideally pulling from varied sources to reach a total of 1000-2000 samples.


**Designing dataset collection**

I’m thinking we should create a dataset subset using multiple sources for various languages, including Malayalam, Hindi, Punjabi, Tamil, Kannada, Gujarati, Telugu, Bengali, Odia, Marathi, Assamese, and English. Some potential sources are ai4bharat/Kathbath, which has many configurations, and ai4bharat/Svarah, though I’m not entirely sure about its focus — possibly Sanskrit or Hindi? We might also consider Shrutilipi and google/fleurs for languages like Assamese and English. Common Voice and indic-superb could be useful too. Since the user has access to Kathbath, we can start there and verify dataset config names by checking our environment with Python.


python3 -c "import sys; print(sys.version)"


Output:
Exit code: 0
Wall time: 0 seconds
Output:
3.10.12 (main, Nov  4 2025, 08:48:33) [GCC 11.4.0]


**Checking installed packages**

I think the best step is to check the installed packages, so I might run a command like `python3 -c "import datasets"` to see if it works. This will help confirm whether or not the datasets package is actually installed. Just a simple command should do the trick! Hopefully, that will give me the information I need to move forward. Let's see what I uncover!


python3 -c "import datasets, huggingface_hub, torch, transformers, faster_whisper"


Output:
Exit code: 0
Wall time: 2.9 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]


Hmm, I need to run the `maya-transcribe` command, which will load the necessary environment variables. But for my Python command to check the dataset, I'll also need to load the `.env` file. I’m wondering if the `HF_TOKEN` environment variable is already in the shell. I could just check by running `echo $HF_TOKEN`. It seems like a straightforward approach, but I want to make sure I’m not missing anything here. Let's see what it returns!


python3 -c "import os; print('HF_TOKEN' in os.environ, 'HUGGINGFACE_HUB_TOKEN' in os.environ)"


Output:
Exit code: 0
Wall time: 0 seconds
Output:
False False


**Loading environment variables**

It looks like the environment variables aren’t loaded, so I need to load the `.env` file or export it. My benchmarking script should handle this like the CLI does. I’ll make sure to integrate that properly. It’s also crucial not to leak any tokens, so I must avoid printing them. Now, I’ll focus on querying the dataset config names for ai4bharat/Kathbath and run the script with `load_dotenv` and the dataset config.


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
# mirror HF_TOKEN -> HUGGINGFACE_HUB_TOKEN
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import get_dataset_config_names

for repo in ['ai4bharat/Kathbath','ai4bharat/Svarah','google/fleurs']:
    try:
        names = get_dataset_config_names(repo)
        print(repo, 'configs', len(names))
        print('  sample:', names[:10])
    except Exception as e:
        print(repo, 'ERROR', type(e).__name__, e)
PY


Output:
Exit code: 0
Wall time: 4.7 seconds
Output:
ai4bharat/Kathbath configs 12
  sample: ['bengali', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'sanskrit', 'tamil']
ai4bharat/Svarah configs 1
  sample: ['default']
The repository for google/fleurs contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/google/fleurs.
You can avoid this prompt in future by passing the argument `trust_remote_code=True`.

Do you wish to run the custom code? [y/N] google/fleurs ERROR ValueError The repository for google/fleurs contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/google/fleurs.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.


**Designing dataset subsets**

I need to design a dataset subset that will work for each language except for "as" and "en." One idea is to use the ai4bharat/Kathbath validation for this purpose. So, I’ll look into the dataset splits first. I think it would be a good start to use the function to get the dataset info, then load one configuration and check the splits to ensure everything is structured properly. Let’s see what I find!


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset

d = load_dataset('ai4bharat/Kathbath','hindi',split=None)
print(d)
for split,ds in d.items():
    print(split, ds)
    print('columns', ds.column_names)
    break
PY


Output:
Exit code: 124
Wall time: 100.1 seconds
Output:
command timed out after 100115 milliseconds

Downloading data:   0%|          | 0/32 [00:00<?, ?files/s]
Downloading data:   3%|▎         | 1/32 [00:03<01:40,  3.26s/files]
Downloading data:   6%|▋         | 2/32 [00:05<01:21,  2.71s/files]
Downloading data:   9%|▉         | 3/32 [00:09<01:28,  3.04s/files]
Downloading data:  12%|█▎        | 4/32 [00:11<01:21,  2.90s/files]
Downloading data:  16%|█▌        | 5/32 [00:14<01:14,  2.75s/files]
Downloading data:  19%|█▉        | 6/32 [00:16<01:09,  2.68s/files]
Downloading data:  22%|██▏       | 7/32 [00:18<01:02,  2.52s/files]
Downloading data:  25%|██▌       | 8/32 [00:21<00:59,  2.47s/files]
Downloading data:  28%|██▊       | 9/32 [00:23<00:57,  2.52s/files]
Downloading data:  31%|███▏      | 10/32 [00:26<00:53,  2.45s/files]
Downloading data:  34%|███▍      | 11/32 [00:28<00:48,  2.33s/files]
Downloading data:  38%|███▊      | 12/32 [00:30<00:45,  2.27s/files]
Downloading data:  41%|████      | 13/32 [00:32<00:42,  2.24s/files]
Downloading data:  44%|████▍     | 14/32 [00:34<00:40,  2.25s/files]
Downloading data:  47%|████▋     | 15/32 [00:36<00:37,  2.22s/files]
Downloading data:  50%|█████     | 16/32 [00:39<00:35,  2.25s/files]
Downloading data:  53%|█████▎    | 17/32 [00:41<00:33,  2.25s/files]
Downloading data:  56%|█████▋    | 18/32 [00:43<00:31,  2.28s/files]
Downloading data:  59%|█████▉    | 19/32 [00:46<00:29,  2.25s/files]
Downloading data:  62%|██████▎   | 20/32 [00:48<00:26,  2.22s/files]
Downloading data:  66%|██████▌   | 21/32 [00:50<00:24,  2.27s/files]
Downloading data:  69%|██████▉   | 22/32 [00:53<00:23,  2.32s/files]
Downloading data:  72%|███████▏  | 23/32 [00:54<00:19,  2.14s/files]
Downloading data:  75%|███████▌  | 24/32 [00:57<00:17,  2.24s/files]
Downloading data:  78%|███████▊  | 25/32 [00:59<00:16,  2.32s/files]
Downloading data:  81%|████████▏ | 26/32 [01:01<00:13,  2.18s/files]
Downloading data:  84%|████████▍ | 27/32 [01:03<00:10,  2.18s/files]
Downloading data:  88%|████████▊ | 28/32 [01:06<00:09,  2.45s/files]
Downloading data:  91%|█████████ | 29/32 [01:09<00:07,  2.58s/files]
Downloading data:  94%|█████████▍| 30/32 [01:13<00:05,  2.84s/files]
Downloading data:  97%|█████████▋| 31/32 [01:16<00:02,  2.90s/files]
Downloading data: 100%|██████████| 32/32 [01:19<00:00,  2.99s/files]
Downloading data: 100%|██████████| 32/32 [01:19<00:00,  2.48s/files]

Generating valid split:   0%|          | 0/3151 [00:00<?, ? examples/s]
Generating valid split:  10%|▉         | 300/3151 [00:00<00:01, 2139.15 examples/s]
Generating valid split:  29%|██▊       | 900/3151 [00:00<00:00, 3538.08 examples/s]
Generating valid split:  48%|████▊     | 1500/3151 [00:00<00:00, 4249.66 examples/s]
Generating valid split:  63%|██████▎   | 1976/3151 [00:00<00:00, 4175.82 examples/s]
Generating valid split:  79%|███████▊  | 2476/3151 [00:00<00:00, 3997.10 examples/s]
Generating valid split:  98%|█████████▊| 3076/3151 [00:00<00:00, 4000.96 examples/s]
Generating valid split: 100%|██████████| 3151/3151 [00:00<00:00, 3927.90 examples/s]

Generating train split:   0%|          | 0/91752 [00:00<?, ? examples/s]
Generating train split:   1%|          | 600/91752 [00:00<00:18, 4881.47 examples/s]
Generating train split:   1%|▏         | 1300/91752 [00:00<00:16, 5591.68 examples/s]
Generating train split:   2%|▏         | 2000/91752 [00:00<00:19, 4579.08 examples/s]
Generating train split:   3%|▎         | 2600/91752 [00:00<00:18, 4832.23 examples/s]
Generating train split:   4%|▎         | 3368/91752 [00:00<00:18, 4875.51 examples/s]
Generating train split:   4%|▍         | 4068/91752 [00:00<00:16, 5337.32 examples/s]
Generating train split:   5%|▌         | 4668/91752 [00:00<00:16, 5291.39 examples/s]
Generating train split:   6%|▌         | 5268/91752 [00:01<00:16, 5354.96 examples/s]
Generating train split:   6%|▋         | 5836/91752 [00:01<00:17, 4987.21 examples/s]
Generating train split:   7%|▋         | 6636/91752 [00:01<00:15, 5634.13 examples/s]
Generating train split:   8%|▊         | 7436/91752 [00:01<00:15, 5320.29 examples/s]
Generating train split:   9%|▉         | 8036/91752 [00:01<00:15, 5277.73 examples/s]
Generating train split:   9%|▉         | 8704/91752 [00:01<00:16, 5129.70 examples/s]
Generating train split:  10%|█         | 9404/91752 [00:01<00:15, 5351.65 examples/s]
Generating train split:  11%|█         | 10204/91752 [00:01<00:14, 5711.39 examples/s]
Generating train split:  12%|█▏        | 10904/91752 [00:02<00:14, 5770.78 examples/s]
Generating tra…1908 chars truncated…ng train split:  31%|███       | 28611/91752 [00:04<00:10, 6235.98 examples/s]
Generating train split:  32%|███▏      | 29478/91752 [00:05<00:10, 5859.84 examples/s]
Generating train split:  33%|███▎      | 30278/91752 [00:05<00:10, 6047.95 examples/s]
Generating train split:  34%|███▍      | 31078/91752 [00:05<00:09, 6219.21 examples/s]
Generating train split:  35%|███▍      | 31945/91752 [00:05<00:10, 5808.34 examples/s]
Generating train split:  36%|███▌      | 32745/91752 [00:05<00:09, 6024.21 examples/s]
Generating train split:  37%|███▋      | 33645/91752 [00:05<00:09, 5850.00 examples/s]
Generating train split:  37%|███▋      | 34245/91752 [00:05<00:10, 5613.66 examples/s]
Generating train split:  38%|███▊      | 34812/91752 [00:06<00:10, 5313.14 examples/s]
Generating train split:  39%|███▉      | 35612/91752 [00:06<00:10, 5607.89 examples/s]
Generating train split:  40%|███▉      | 36312/91752 [00:06<00:09, 5805.59 examples/s]
Generating train split:  40%|████      | 37112/91752 [00:06<00:08, 6072.86 examples/s]
Generating train split:  41%|████      | 37779/91752 [00:06<00:09, 5895.53 examples/s]
Generating train split:  42%|████▏     | 38579/91752 [00:06<00:08, 6124.66 examples/s]
Generating train split:  43%|████▎     | 39379/91752 [00:06<00:08, 6235.62 examples/s]
Generating train split:  44%|████▍     | 40146/91752 [00:06<00:07, 6565.02 examples/s]
Generating train split:  45%|████▍     | 41046/91752 [00:07<00:08, 5846.08 examples/s]
Generating train split:  45%|████▌     | 41746/91752 [00:07<00:08, 5709.85 examples/s]
Generating train split:  46%|████▋     | 42546/91752 [00:07<00:08, 5981.15 examples/s]
Generating train split:  47%|████▋     | 43513/91752 [00:07<00:08, 5657.13 examples/s]
Generating train split:  48%|████▊     | 44313/91752 [00:07<00:07, 6015.53 examples/s]
Generating train split:  49%|████▉     | 45113/91752 [00:07<00:07, 6303.06 examples/s]
Generating train split:  50%|████▉     | 45813/91752 [00:07<00:07, 6358.93 examples/s]
Generating train split:  51%|█████     | 46480/91752 [00:07<00:07, 5918.70 examples/s]
Generating train split:  52%|█████▏    | 47280/91752 [00:08<00:07, 6184.48 examples/s]
Generating train split:  52%|█████▏    | 48080/91752 [00:08<00:07, 6227.14 examples/s]
Generating train split:  53%|█████▎    | 48847/91752 [00:08<00:07, 5980.38 examples/s]
Generating train split:  54%|█████▍    | 49647/91752 [00:08<00:06, 6169.17 examples/s]
Generating train split:  55%|█████▍    | 50447/91752 [00:08<00:06, 6319.82 examples/s]
Generating train split:  56%|█████▌    | 51247/91752 [00:08<00:06, 6315.82 examples/s]
Generating train split:  57%|█████▋    | 52114/91752 [00:08<00:06, 6099.09 examples/s]
Generating train split:  58%|█████▊    | 52914/91752 [00:09<00:06, 5701.10 examples/s]
Generating train split:  59%|█████▊    | 53714/91752 [00:09<00:07, 5337.89 examples/s]
Generating train split:  59%|█████▉    | 54481/91752 [00:09<00:06, 5654.32 examples/s]
Generating train split:  60%|██████    | 55281/91752 [00:09<00:06, 5302.08 examples/s]
Generating train split:  61%|██████    | 55881/91752 [00:09<00:06, 5347.21 examples/s]
Generating train split:  62%|██████▏   | 56681/91752 [00:09<00:06, 5608.19 examples/s]
Generating train split:  62%|██████▏   | 57281/91752 [00:09<00:06, 5493.59 examples/s]
Generating train split:  63%|██████▎   | 57848/91752 [00:09<00:06, 5097.69 examples/s]
Generating train split:  64%|██████▎   | 58448/91752 [00:11<00:27, 1225.92 examples/s]
Generating train split:  65%|██████▍   | 59448/91752 [00:11<00:17, 1857.32 examples/s]
Generating train split:  66%|██████▌   | 60315/91752 [00:11<00:13, 2401.34 examples/s]
Generating train split:  66%|██████▋   | 61015/91752 [00:11<00:11, 2754.55 examples/s]
Generating train split:  67%|██████▋   | 61715/91752 [00:11<00:09, 3147.43 examples/s]
Generating train split:  68%|██████▊   | 62515/91752 [00:12<00:07, 3764.42 examples/s]
Generating train split:  69%|██████▉   | 63182/91752 [00:12<00:06, 4115.15 examples/s]
Generating train split:  70%|██████▉   | 63982/91752 [00:12<00:05, 4690.95 examples/s]
Generating train split:  71%|███████   | 64782/91752 [00:12<00:05, 5208.18 examples/s]
Generating train split:  71%|███████▏  | 65582/91752 [00:12<00:04, 5726.67 examples/s]
Generating train split:  72%|███████▏  | 66449/91752 [00:12<00:04, 5651.90 examples/s]
Generating train split:  73%|███████▎  | 67149/91752 [00:12<00:04, 5808.48 examples/s]


**Analyzing dataset loading issues**

I'm facing a timeout issue while trying to download or generate a dataset. It was still generating the training split at 73%, which indicates that loading the full dataset is quite slow. To optimize this, I should avoid loading the entire dataset for sampling. Instead, using `load_dataset(..., split="validation", streaming=True)` will let me iterate without downloading everything. However, for audio files, we need to download each file anyway. I should check the actual split names to ensure proper use.


**Deciding on dataset splits**

I’ve settled on using the "valid" split for sampling, and I could also consider the "train" split. For benchmarks, it seems better to use "test" or "valid." The "valid" split has 3151 examples for Hindi, which is manageable. However, using `load_dataset(..., split="valid")` still downloads dataset files, resulting in 32 files. If it takes around 80 seconds per language, maybe I can adjust the timeout. Streaming could help; it might just download necessary metadata, which is promising. Let’s test the streaming for "valid" quickly!


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset

ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid',streaming=True)
ex = next(iter(ds))
print(ex.keys())
print({k: type(v).__name__ for k,v in ex.items()})
# audio might be dict with path
print('audio', ex.get('audio'))
print('text sample', ex.get('text') or ex.get('sentence') or ex.get('transcript') or ex.get('transcription'))
PY


Output:
Exit code: 0
Wall time: 19.2 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
dict_keys(['fname', 'text', 'audio_filepath', 'lang', 'duration', 'gender', 'speaker_id'])
{'fname': 'str', 'text': 'str', 'audio_filepath': 'dict', 'lang': 'str', 'duration': 'float', 'gender': 'str', 'speaker_id': 'int'}
audio None
text sample हमने ओबीसी तबके के बच्चों के लिए उच्च शिक्षा में आरक्षण दिया


Summary: auto


**Exploring dataset structure**

I'm getting some intriguing details about the Kathbath dataset streaming. It returns keys like 'fname', 'text', and 'audio_filepath', but not 'audio' directly. It seems that 'audio_filepath' might be some kind of dictionary. I should check how the structure looks. Understanding this structure is important because it will help me know how to work with the audio data efficiently. Let’s print the audio_filepath structure and see what we find!


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset

ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid',streaming=True)
ex = next(iter(ds))
print(ex['audio_filepath'])
PY


Output:
Exit code: 134
Wall time: 75 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
{'path': '844424930524026-252-f.flac', 'array': array([ 0.        ,  0.        ,  0.        , ..., -0.0368253 ,
       -0.04417396, -0.04884875]), 'sampling_rate': 16000}
Fatal Python error: PyGILState_Release: thread state 0x7340a40ebb10 must be current when releasing
Python runtime state: finalizing (tstate=0x000063af79e5f620)

Thread 0x0000734143a5c000 (most recent call first):
  <no Python frame>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _brotli, backports.zstd._zstd, simplejson._speedups, charset_normalizer.md, yaml._yaml, _cffi_backend, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, xxhash._xxhash, pyarrow._json, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, scipy.signal._sigtools, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy._lib._uarray._uarray, scipy.signal._max_len_seq_inner, scipy.signal._upfirdn_apply, scipy.signal._spline, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, scipy.signal._sosfilt, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.signal._peak_finding_utils, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.typing.builtins.itertools, numba.cpython.builtins.math, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, msgpack._cmsgpack, psutil._psutil_linux, psutil._psutil_posix, numba.cpython.mathimpl.math, numba.cpython.mathimpl.sys, numba.cpython.hashing.math, numba.cpython.hashing.sys, numba.cpython.numbers.math, numba.np.arraymath.math, numba.core.typing.cmathdecl.cmath, numba.core.typing.mathdecl.math (total: 208)
/bin/bash: line 14: 26131 Aborted                 (core dumped) python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset

ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid',streaming=True)
ex = next(iter(ds))
print(ex['audio_filepath'])
PY


**Investigating a crash issue**

I’m looking into a python process crashing with a fatal error related to PyGILState_Release. This seems to be concerning and might stem from some extension libraries due to streaming datasets with audio and torch. I noticed a warning printed by torch, and I’m wondering if the crash could be linked to threading issues when finalizing these processes. Maybe avoiding importing torch could help, but it seems like it was imported indirectly through another library. I should dig deeper to understand this!


**Exploring torch import issues**

I’m reflecting on why there was no crash in the previous script despite not importing torch. The warning indicating that "torch/cuda/__init__.py" was imported suggests torch got pulled in somehow, possibly due to `faster-whisper` or another dependency. Maybe `datasets` uses torch for audio decoding when installed. The crash is serious and could be related to bugs in `datasets`, `torch`, or `numba`. To avoid future crashes, I might need to consider running it as a persistent process, even though it might complicate longer benchmarks.


**Investigating environmental bugs**

I’m considering whether this environment has a bug causing Python processes to abort. To potentially avoid issues, I could refrain from importing torch or set an environment variable. Using `datasets` with `streaming=True` may employ `soundfile`, which relies on `numba`. The crash stack includes many modules like `scipy`, `numba`, and `torch`, suggesting that huggingface's audio decode might be involved. It seems the fatal crash is related to threads spawned by torch during finalization. Calling `os._exit(0)` is an option, but it isn't ideal. I want to ensure we don’t crash during benchmarking.


**Exploring crash causes in datasets**

I’m thinking that the crash might be tied to using `datasets` with audio arrays, which involves `torch`, `librosa`, and possibly triggers `numba`. To avoid this, I could use `decode=False` to prevent heavy libraries from loading. It’s worth testing the streaming dataset in this way. In the Kathbath dataset, the column has 'audio_filepath' for the audio path and array, but it seems it's not leveraging the Audio feature of datasets. It also looks like the dataset might be decoding audio automatically, so I'll need to figure out how to work around that.


**Trying to cast audio column**

I’m noticing that in the output, the `audio_filepath` is a dictionary that includes the path, array, and sampling rate. This hints that it should be using the `Audio` feature even though it’s named `audio_filepath`. I think I could cast it to `datasets.Audio(decode=False)` to retrieve just the audio path. So, let's give it a shot with the following code snippet to see how it works out! I'm curious to check the result!


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset, Audio

ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid',streaming=True)
ds = ds.cast_column('audio_filepath', Audio(decode=False))
ex = next(iter(ds))
print(ex['audio_filepath'])
print(type(ex['audio_filepath']))
PY


Output:
Exit code: 134
Wall time: 6.1 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
{'bytes': b'fLaC\x00\x00\x00"\x04\x80\x04\x80\x00\x04M\x00\n\x12\x03\xe8\x01p\x00\x00\xfb\x11\x8f\xe1b\xd5\xdd\xd5\x1c\x8d\xe4<\xccVV\x1e\x7f \x04\x00\x00\x8d\r\x00\x00\x00Lavf58.29.100\x05\x00\x00\x00\x10\x00\x00\x00major_brand=mp42\x0f\x00\x00\x00minor_version=0\x1a\x00\x00\x00compatible_brands=isommp42\x16\x00\x00\x00com.android.version=11\x15\x00\x00\x00encoder=Lavf58.29.100\x81\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00…403606 chars truncated…6(\xf4\xdc2\xe5(\'\x94)yd\x0b\xf5\x87(\xb4y\xf6\xa0\xde\xd4WlC\xd8\x8co\xa8\x94\'*F\x16\xc2\x83\x0e\xbd\xda\x9fd\x1b\x0flH\xa8\x08\xe5\xc8\x15\x95\x1aor\x8e\xfbB\rg\x9c9\x05\xb5R\x07\xee\xd4r\xa0=\x91\x99\xe9rTi\x1f\x88P\x92(Qe)\x97\xe0Hj?\x9b\x9d\x87V\xe2\x95\x01S\xdc\x8a\xbf\xc6g\x92\xb8\xc6\xdc\xf7\xbc\x0bZa\x9c\xb7\x9f\xeb\xcfv\xb7?\x8a\xa4h\xc2~e\xf3r\xa7\'9\x88\xa1Zb\x8dP\xb6\xdaqo\xb6\xcd@\x15\xdb\xfc\xa4\xe2\nf\xa3u\x13\xaa\x91\xcb^\xa0\xbd\xc2\xe82\x989\xe7w\xae\xc9\xc9F\xcb\xc6#\xf0o\x9cj\xa3%NW\xe2\x989\xbe\xfa\xbd\xf6\x89n\r\xfe_\x05\xf7T\xd2\'\x86\xef\xd4\\\x04\xd2T.v\x16v\xb2Ms7P\xf0\x86z\x9b{\x87\xfc\xefn\x93\xf9Eo\xbe5\x95\x1a^\nC!>T\xb5\xde\xc4NK\x7fA\x1d\xae\xd8\x15#L\xd8\xfb\\l\x9a\x8dE*Z\xcc\xa8\xd5\xb1o\xbbn]\xc3\xb4\x81K\xcf\x80\x85-;\xdc\xc8\xb4\xe4HR\xc6Hb\xcc\xb59\x13\x80J\xce\x9b#w\x17\xfd\x89\xac\x83\xc7\x95\xf9\x07\xa2\x83KBU\xa7YP\xad$J\x19(\xd7\xa8&?\x10', 'path': '844424930524026-252-f.flac'}
<class 'dict'>
Fatal Python error: PyGILState_Release: thread state 0x7bfe3c0e9de0 must be current when releasing
Python runtime state: finalizing (tstate=0x000060c8815dc620)

Thread 0x00007bfee0255000 (most recent call first):
  <no Python frame>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _brotli, backports.zstd._zstd, simplejson._speedups, charset_normalizer.md, yaml._yaml, _cffi_backend, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, xxhash._xxhash, pyarrow._json, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special (total: 127)
/bin/bash: line 16: 26984 Aborted                 (core dumped) python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset, Audio

ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid',streaming=True)
ds = ds.cast_column('audio_filepath', Audio(decode=False))
ex = next(iter(ds))
print(ex['audio_filepath'])
print(type(ex['audio_filepath']))
PY


**Testing for issues**

I’m noticing a crash during finalization, which might be due to `torch` being imported or a thread that hasn't been cleaned up properly. To troubleshoot, I should run a simple script that only imports the datasets and exits right after. If this works without crashing, it could help narrow down the problem to something related to `torch` or thread management. Let’s see if this leads to any clues!


python3 - <<'PY'
import datasets
print('datasets imported')
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
datasets imported


**Analyzing a crash issue**

It looks like an abort happened after some output, with a core dump and exit code 134, which could break automation. I’m thinking the crash might be triggered by something related to atexit or a destructor in a library during interpreter finalization. I could try calling `import os; os._exit(0)` at the end to avoid cleanup, but that seems hacky. It’s better to figure out which library is causing the crash, possibly related to the `torch` destructor when accessing audio data. Running the script in an interactive environment could also change things.


**Exploring benchmark functionality**

I think we can integrate benchmark functionality into the `maya-transcribe` CLI, which outputs results after a long-running process. However, exiting might lead to similar crashes if we're using datasets and Torch. While using `os._exit` could prevent issues, it's not ideal. The problem may stem from Python 3.10 or a known bug with `torch` and datasets in streaming mode, potentially linked to `multiprocessing` or `aiohttp`. To mitigate this, I might set `TOKENIZERS_PARALLELISM=false`. I’ll try reproducing a minimal script using `datasets.load_dataset` without accessing the audio field initially to check for crashes. However, I need to access audio to create a benchmark dataset, so maybe I'll use normal loading with `split='valid'` and `streaming=False`.


**Testing dataset loading**

I think downloading the dataset and storing audio files in a local cache might help reduce crashes. I want to test loading the dataset in non-streaming mode while decoding audio, but I'm worried that might still lead to a crash at exit. To minimize issues, I’ll set a small limit and avoid printing audio arrays. Previously, loading the dataset in non-streaming mode took longer but didn't crash—it just timed out. So, I'll try loading it with `split='valid'` and `download_mode='reuse_cache_if_exists'`, running it with a high timeout of about 300 seconds.


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(dotenv_path=Path('.env'), override=False)
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN'] = os.environ['HF_TOKEN']

from datasets import load_dataset

# load only valid split non-streaming
print('loading...')
ds = load_dataset('ai4bharat/Kathbath','hindi',split='valid')
print('loaded', ds)
print('first keys', ds[0].keys())
print('audio type', type(ds[0]['audio_filepath']))
# access audio bytes? maybe.
a = ds[0]['audio_filepath']
print('audio keys', a.keys())
print('path', a.get('path'))
print('array shape', getattr(a.get('array'), 'shape', None), 'sr', a.get('sampling_rate'))
PY


Output:
Exit code: 0
Wall time: 35.3 seconds
Output:

Downloading data:   0%|          | 0/32 [00:00<?, ?files/s]loading...

Downloading data:   3%|▎         | 1/32 [00:00<00:04,  6.90files/s]
Downloading data: 100%|██████████| 32/32 [00:00<00:00, 210.15files/s]

Generating valid split:   0%|          | 0/3151 [00:00<?, ? examples/s]
Generating valid split:  10%|▉         | 300/3151 [00:00<00:01, 2416.85 examples/s]
Generating valid split:  22%|██▏       | 700/3151 [00:00<00:00, 3072.76 examples/s]
Generating valid split:  38%|███▊      | 1200/3151 [00:00<00:00, 3493.96 examples/s]
Generating valid split:  53%|█████▎    | 1676/3151 [00:00<00:00, 2953.72 examples/s]
Generating valid split:  66%|██████▌   | 2076/3151 [00:00<00:00, 3094.29 examples/s]
Generating valid split:  79%|███████▊  | 2476/3151 [00:00<00:00, 3090.22 examples/s]
Generating valid split:  91%|█████████▏| 2876/3151 [00:00<00:00, 3025.28 examples/s]
Generating valid split: 100%|██████████| 3151/3151 [00:01<00:00, 3100.73 examples/s]

Generating train split:   0%|          | 0/91752 [00:00<?, ? examples/s]
Generating train split:   0%|          | 100/91752 [00:00<02:13, 687.30 examples/s]
Generating train split:   1%|          | 600/91752 [00:00<00:40, 2267.53 examples/s]
Generating train split:   1%|          | 1100/91752 [00:00<00:30, 3015.62 examples/s]
Generating train split:   2%|▏         | 1500/91752 [00:00<00:28, 3166.53 examples/s]
Generating train split:   2%|▏         | 1900/91752 [00:00<00:27, 3319.99 examples/s]
Generating train split:   3%|▎         | 2300/91752 [00:00<00:26, 3390.72 examples/s]
Generating train split:   3%|▎         | 2700/91752 [00:00<00:25, 3529.51 examples/s]
Generating train split:   3%|▎         | 3068/91752 [00:01<00:40, 2199.54 examples/s]
Generating train split:   4%|▍         | 3868/91752 [00:01<00:27, 3160.31 examples/s]
Generating train split:   5%|▍         | 4268/91752 [00:01<00:26, 3266.43 examples/s]
Generating train split:   5%|▌         | 4668/91752 [00:01<00:25, 3397.10 examples/s]
Generating train split:   6%|▌         | 5068/91752 [00:01<00:24, 3501.81 examples/s]
Generating train split:   6%|▌         | 5668/91752 [00:01<00:23, 3677.27 examples/s]
Generating train split:   7%|▋         | 6536/91752 [00:02<00:29, 2901.36 examples/s]
Generating train split:   8%|▊         | 7236/91752 [00:02<00:25, 3305.23 examples/s]
Generating train split:   8%|▊         | 7636/91752 [00:02<00:25, 3290.98 examples/s]
Generating train split:   9%|▉         | 8036/91752 [00:02<00:26, 3205.30 examples/s]
Generating train split:   9%|▉         | 8604/91752 [00:02<00:23, 3607.42 examples/s]
Generating train split:  10%|▉         | 9004/91752 [00:03<00:33, 2464.94 examples/s]
Generating train split:  11%|█         | 10004/91752 [00:03<00:22, 3689.28 examples/s]
Generating train split:  12%|█▏        | 10604/91752 [00:03<00:21, 3690.33 examples/s]
Generating train split:  12%|█▏        | 11204/91752 [00:03<00:22, 3597.35 examples/s]
Generating train split:  13%|█▎        | 11772/91752 [00:03<00:22, 3551.79 examples/s]
Generating train split:  14%|█▎        | 12472/91752 [00:03<00:26, 3005.33 examples/s]
Generating train split:  14%|█▍        | 13072/91752 [00:04<00:23, 3409.15 examples/s]
Generating train split:  15%|█▍        | 13572/91752 [00:04<00:22, 3518.62 examples/s]
Generating train split:  15%|█▌        | 14072/91752 [00:04<00:21, 3663.07 examples/s]
Generating train split:  16%|█▌        | 14640/91752 [00:04<00:23, 3345.88 examples/s]
Generating train split:  17%|█▋        | 15340/91752 [00:04<00:26, 2920.96 examples/s]
Generating train split:  17%|█▋        | 15940/91752 [00:04<00:23, 3235.57 examples/s]
Generating train split:  18%|█▊        | 16340/91752 [00:05<00:24, 3091.74 examples/s]
Generating train split:  18%|█▊        | 16740/91752 [00:05<00:24, 3101.25 examples/s]
Generating train split:  19%|█▉        | 17208/91752 [00:05<00:22, 3330.11 examples/s]
Generating train split:  19%|█▉        | 17708/91752 [00:05<00:23, 3158.66 examples/s]
Generating train split:  20%|█▉        | 18308/91752 [00:05<00:26, 2755.62 examples/s]
Generating train split:  21%|██        | 18908/91752 [00:05<00:22, 3282.40 examples/s]
Generating train split:  21%|██        | 19408/91752 [00:06<00:21, 3430.27 examples/s]
Generating train split:  22%|██▏       | 19808/91752 [00:06<00:20, 3493.01 examples/s]
Generating train split:  22%|██▏       | 20476/91752 [00:06<00:21, 3372.01 examples/s]
Generating train split:  23%|██▎       | 21276/91752 [00:06<00:23, 2975.39 examples/s]
Generating train split:  24%|██▍       | 21876/91752 [00:06<00:21, 3316.38 examples/s]
Generating train split:  24%|██▍       | 22376/91752 [00:06<00:20, 3425.42 examples/s]
Generating train split:  25%|██▌       | 23044/91752 [00:07<00:20, 3319.01 …8485 chars truncated…███▏ | 74850/91752 [00:23<00:05, 3119.64 examples/s]
Generating train split:  82%|████████▏ | 75250/91752 [00:23<00:05, 3214.29 examples/s]
Generating train split:  82%|████████▏ | 75650/91752 [00:23<00:05, 3144.95 examples/s]
Generating train split:  83%|████████▎ | 76050/91752 [00:23<00:04, 3178.35 examples/s]
Generating train split:  83%|████████▎ | 76450/91752 [00:23<00:04, 3121.25 examples/s]
Generating train split:  84%|████████▍ | 76850/91752 [00:23<00:05, 2975.30 examples/s]
Generating train split:  84%|████████▍ | 77250/91752 [00:24<00:04, 2961.97 examples/s]
Generating train split:  85%|████████▍ | 77717/91752 [00:24<00:05, 2721.65 examples/s]
Generating train split:  85%|████████▌ | 78117/91752 [00:24<00:04, 2767.00 examples/s]
Generating train split:  86%|████████▌ | 78517/91752 [00:24<00:04, 2690.13 examples/s]
Generating train split:  86%|████████▌ | 78917/91752 [00:24<00:04, 2731.79 examples/s]
Generating train split:  86%|████████▋ | 79317/91752 [00:24<00:04, 2755.71 examples/s]
Generating train split:  87%|████████▋ | 79717/91752 [00:24<00:04, 2680.29 examples/s]
Generating train split:  87%|████████▋ | 80117/91752 [00:25<00:04, 2751.36 examples/s]
Generating train split:  88%|████████▊ | 80584/91752 [00:25<00:04, 2538.55 examples/s]
Generating train split:  88%|████████▊ | 80984/91752 [00:25<00:04, 2684.94 examples/s]
Generating train split:  89%|████████▊ | 81384/91752 [00:25<00:03, 2734.18 examples/s]
Generating train split:  89%|████████▉ | 81784/91752 [00:25<00:03, 2755.83 examples/s]
Generating train split:  90%|████████▉ | 82184/91752 [00:25<00:03, 2655.22 examples/s]
Generating train split:  90%|█████████ | 82584/91752 [00:26<00:03, 2603.07 examples/s]
Generating train split:  90%|█████████ | 82984/91752 [00:26<00:03, 2616.76 examples/s]
Generating train split:  91%|█████████ | 83251/91752 [00:26<00:03, 2444.81 examples/s]
Generating train split:  91%|█████████ | 83651/91752 [00:26<00:03, 2512.65 examples/s]
Generating train split:  92%|█████████▏| 84051/91752 [00:26<00:02, 2607.01 examples/s]
Generating train split:  92%|█████████▏| 84451/91752 [00:26<00:02, 2635.41 examples/s]
Generating train split:  92%|█████████▏| 84851/91752 [00:26<00:02, 2674.13 examples/s]
Generating train split:  93%|█████████▎| 85251/91752 [00:27<00:02, 2756.91 examples/s]
Generating train split:  93%|█████████▎| 85651/91752 [00:27<00:02, 2798.48 examples/s]
Generating train split:  94%|█████████▍| 86118/91752 [00:27<00:02, 2527.06 examples/s]
Generating train split:  94%|█████████▍| 86518/91752 [00:27<00:01, 2645.28 examples/s]
Generating train split:  95%|█████████▍| 86918/91752 [00:27<00:01, 2626.66 examples/s]
Generating train split:  95%|█████████▌| 87318/91752 [00:27<00:01, 2735.98 examples/s]
Generating train split:  96%|█████████▌| 87718/91752 [00:27<00:01, 2774.55 examples/s]
Generating train split:  96%|█████████▌| 88118/91752 [00:28<00:01, 2739.47 examples/s]
Generating train split:  96%|█████████▋| 88518/91752 [00:28<00:01, 2754.45 examples/s]
Generating train split:  97%|█████████▋| 88985/91752 [00:28<00:01, 2469.68 examples/s]
Generating train split:  97%|█████████▋| 89385/91752 [00:28<00:00, 2494.76 examples/s]
Generating train split:  98%|█████████▊| 89785/91752 [00:28<00:00, 2501.84 examples/s]
Generating train split:  98%|█████████▊| 90185/91752 [00:28<00:00, 2545.51 examples/s]
Generating train split:  99%|█████████▊| 90585/91752 [00:29<00:00, 2631.85 examples/s]
Generating train split:  99%|█████████▉| 90985/91752 [00:29<00:00, 2610.22 examples/s]
Generating train split: 100%|█████████▉| 91385/91752 [00:29<00:00, 2634.25 examples/s]
Generating train split: 100%|██████████| 91752/91752 [00:29<00:00, 2801.99 examples/s]
Generating train split: 100%|██████████| 91752/91752 [00:29<00:00, 3111.42 examples/s]
loaded Dataset({
    features: ['fname', 'text', 'audio_filepath', 'lang', 'duration', 'gender', 'speaker_id'],
    num_rows: 3151
})
first keys dict_keys(['fname', 'text', 'audio_filepath', 'lang', 'duration', 'gender', 'speaker_id'])
audio type <class 'dict'>
audio keys dict_keys(['path', 'array', 'sampling_rate'])
path 844424930524026-252-f.flac
array shape (64273,) sr 16000


**Analyzing dataset loading**

I’m excited that the script succeeded without crashing and printed the first audio array successfully. It seems like the crash issue is related to the streaming mode; avoiding `streaming=True` might be the way to go. So, I’ll focus on loading datasets non-streaming, perhaps limiting splits and number of examples. There’s a bit of confusion since loading `split='valid'` still generated a train split—this might be a quirk of how the dataset builder operates. Even when selecting a specific split, it can prepare others.


**Considering dataset preparation**

It feels a bit wasteful that the dataset generated a train split alongside the valid one, but it may be necessary due to the underlying data structure. The train split had 91,752 rows, which took about 29 seconds, which isn't too bad. However, for 12 languages, generating a separate train for each might get heavy. Caching could help since subsequent loads may not require re-generation. Still, we only need 1,000 to 2,000 samples, so using `load_dataset` with `split='valid'` while managing memory sounds like a better option, despite the train split still being generated.


**Exploring dataset management**

It seems like using local caching could help reduce costs, but I wonder if there's a `download_mode='reuse_cache_if_exists'` to aid in reusing resources. Initially, there might be a cost for each language, but I want to avoid unnecessary train generation by focusing on `split='valid'`. I could also load other languages with `streaming=False` while utilizing previous downloads. I'll test a smaller language config to check the dataset length. Since the user only needs 1,000 to 2,000 samples, I could sample from multiple datasets to create a curated dataset for languages like Assamese and English.


**Planning dataset construction**

The user has access to "shrutilipi," which might include more data, but I think starting with Kathbath, FLEURS, and Common Voice is a good plan. However, building a dataset from Common Voice could be quite large, and while I could load a small subset using `load_dataset('mozilla-foundation/common_voice_17_0', 'hi', split='test')`, I’m concerned about potential crashes with streaming. It might be better to avoid streaming altogether and consider using the FLEURS dataset, which supports many languages and is more manageable, while ensuring we address the `trust_remote_code` requirement.


**Developing language benchmark**

I'm considering using FLEURS for Assamese and English, as well as other languages not in Kathbath. For languages in Kathbath, they might be a better fit, but if the goal is 2,000 examples, the Kathbath valid set has about 3,000 available. FLEURS might offer around 1,000 to 3,000 examples, and I need to map BCP-47 language codes like 'as_in' and 'en_us' to our requirements. I'll focus on building a benchmarking system that includes performance evaluations per language, using both Kathbath and FLEURS for robust analysis and adjustable tweaks for better results.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import unicodedata

_ZERO_WIDTH = {
    "\u200b",  # zero width space
    "\u200c",  # zero width non-joiner
    "\u200d",  # zero width joiner
    "\ufeff",  # byte order mark
}

def normalize_text(text: str, lang: str | None = None) -> str:
    """
    Conservative normalization for hypothesis alignment/comparison (not for publishing).
    - Unicode NFKC
    - Drop punctuation
    - Normalize all Unicode digits to ASCII
    - Collapse whitespace
    - Lowercase for English only
    """
    if not text:
        return ""

    text = unicodedata.normalize("NFKC", text)

    out_chars: list[str] = []
    for ch in text:
        if ch in _ZERO_WIDTH:
            continue

        cat = unicodedata.category(ch)
        if cat.startswith("P"):
            continue

        if cat == "Zs" or ch in ("\t", "\n", "\r"):
            out_chars.append(" ")
            continue

        if cat == "Nd":
            try:
                out_chars.append(str(unicodedata.digit(ch)))
                continue
            except (TypeError, ValueError):
                pass

        out_chars.append(ch)

    norm = "".join(out_chars)
    norm = " ".join(norm.split())

    if (lang or "").lower() in {"en", "eng"}:
        norm = norm.lower()

    return norm

def tokenize_words(text: str) -> list[str]:
    text = " ".join((text or "").split())
    if not text:
        return []
    return text.split(" ")


Summary: auto


**Exploring normalization challenges**

Normalization seems quite straightforward, yet it could be better suited for Indic languages—like possibly removing diacritics, but maybe not. For English, lowercase works fine, but for Indic, it doesn't align. I notice that `jiwer.wer` uses whitespace tokenization, which is acceptable. Customizing with `normalize_text` could fit for both reference and hypothesis, but I worry about being too aggressive by removing punctuation. Implementing a new benchmarking tool with `maya-transcribe` could help create and analyze local datasets too.


**Implementing benchmark commands**

I’m thinking about implementing a top-level command called `maya-transcribe benchmark` with useful subcommands using Typer. It's a great way to keep things organized. I’ll set up `bench_app` under the main app, which should help keep the code concise. The user needs to create a new benchmark dataset linked to their Hugging Face ID using credentials from a `.env` file, so I’ll add a `maya-transcribe bench create-hf` command for that purpose. Plus, I’ll need another command to sample sources for the dataset, starting with `maya-transcribe bench build`.


**Planning benchmark operations**

I’m looking at how to implement `maya-transcribe bench run` next, which should take inputs like `--bench-dir bench/<name>` and run the pipeline on different language subsets, writing outputs accordingly. For `maya-transcribe bench upload`, inputs will include `--bench-dir bench/<name>` and the dataset repo ID to create and push the dataset. The user wants benchmarks now, but creating a full dataset may take time. I think I could start with running scripts for one language, like Hindi, to ensure that the pipeline works—but I’m pondering if they can run it later, given their computing resources. I’ll check if the environment has a GPU with `nvidia-smi`.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-ada3fc23-5709-2bd4-dd15-24f3b0c3cd2d)


**Evaluating GPU capabilities for tasks**

I’m excited about having the A100 80GB GPU! It should be powerful enough to run whisper large-v3, but I need to confirm that faster-whisper uses CTranslate2 and utilizes GPU with float16 compute type. I might benchmark across multiple languages, aiming for 200 segments per language across 12 languages, totaling about 2400 segments. Each segment will likely be around 5 seconds, leading to roughly 3.3 hours total. If it runs at 2x real-time, we could finish in around 1.6 hours, though I’ll need to manage resources carefully!


**Managing execution time and resources**

I realize we’re in an interactive environment where I can run long commands, but they need to fit within user requests. Typical codex sessions might have timeouts, and while I know I can extend our tool calls, there could be a limit of around 10 minutes. The `shell_command` tool has a `timeout_ms` parameter that I can adjust, possibly up to an hour. However, running large benchmarks in one go might not be feasible. Instead, I could implement and run smaller subsets, maybe around 20 per language. Also, using `load_dataset` might cache large audio files, so I need to check for enough disk space!


**Preparing for dataset management**

I need to make sure we use a HF token for accessing gated datasets, and I can set that as an environment variable. Additionally, it’s important for our benchmarking harness to include robust dataset caching, so we can reuse downloaded subsets efficiently. Before we dive in, I should definitely check the available disk space to ensure we have enough room for everything. It’s all about making sure we’re set up for success!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
/dev/sda2       984G  497G  447G  53% /


**Creating dataset repo**

I see we've got plenty of disk space, so creating a local dataset subset and running the pipeline is feasible. The user is looking for a new benchmark dataset under their HF ID, which might involve uploading sizable audio files, but that's manageable since we have network capabilities.

However, I realize the token in `.env` seems like a real HF token. My environment is ephemeral, but I can use it while ensuring the dataset repo remains private. I'll explore using `huggingface_hub.whoami` to get the username for the dataset creation. It might take some time to upload, so I could create the dataset repo first without the full audio files, just to set things up.


**Creating benchmark dataset**

The user has specifically asked me to create a new benchmark dataset under their huggingface ID, using the key in `.env`, and ensure it's private. This means I need to push to the hub. I could start with a small dataset subset, like 50 per language, to keep the size manageable, and then scale up as needed.

They also mentioned wanting 1000-2000 per language, which is doable, but the upload could be huge—potentially tens of gigabytes. That might take a while, but it could be feasible in the A100 environment if bandwidth is good. I might suggest creating the dataset incrementally and supporting resume features. Given the network capabilities here, starting with 200 per language might work well, totaling around 2400 segments. If those segments are around 3 hours of audio and stored as FLAC, the size could be about 1-2GB, making the upload more manageable.


**Considering dataset licensing**

Building the dataset involves downloading audio from original datasets using the `datasets` library. Once I have that, I can export the audio and push it. However, I must ensure that we’re following license restrictions.

It’s particularly concerning that the dataset might be derived from the AI4Bharat dataset, which could have licensing prohibiting redistribution. Even if the user suggests creating a benchmark dataset privately under their HF ID, I need to be cautious, as redistribution—even if private—might still breach license terms. It's crucial to respect these policies.


**Addressing licensing and dataset creation**

As I consider the importance of licensing and gating restrictions, I must ensure that the user has the right to redistribute any datasets. In the coding environment, I can implement a tool to create a dataset from sources, but I should treat it as for internal use and keep it private to avoid potential license violations.

An alternative approach would be to create a Hugging Face dataset that only includes references to source dataset indices and transcripts. This way, I could use the `datasets` library to stream audio from the original dataset during runtime, ensuring we aren't redistributing audio. I could create a "manifest dataset" containing metadata with links to the source dataset, allowing for reproducibility while respecting the licensing. However, the user might prefer including the actual audio to bypass gating.


**Creating a safe benchmark dataset**

If legally redistributing audio isn't an option, I should focus on creating a manifest dataset. The idea is to develop a private dataset that contains only source information and specific subset indices or transcripts, without including the audio files. This would allow the user to download the original audio locally using their HF token, satisfying the requirement for a benchmark dataset.

They mentioned wanting to include both audio and text, but I think sticking with just metadata and hashed segment IDs is safer. While they want an easy process, it's important to stay cautious and examine the licensing for the AI4Bharat dataset. It seems there might be open licenses available, but I’ll avoid copying audio unless I'm absolutely sure it's allowed.

I’ll provide a flag `--include-audio` for cases where they wish to embed audio while defaulting to a "manifest-only" dataset for safety. Finally, we can sample across datasets using the ai4bharat/Kathbath dataset for several languages, making sure to account for all configurations.


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN']=os.environ['HF_TOKEN']
from datasets import get_dataset_config_names
print(get_dataset_config_names('ai4bharat/Kathbath'))
PY


Output:
Exit code: 0
Wall time: 2.7 seconds
Output:
['bengali', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'sanskrit', 'tamil', 'telugu', 'urdu']


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN']=os.environ['HF_TOKEN']
from datasets import get_dataset_config_names
try:
    names = get_dataset_config_names('google/fleurs', trust_remote_code=True)
    print('configs', len(names))
    print(names[:50])
except TypeError as e:
    print('TypeError', e)
PY


Output:
Exit code: 0
Wall time: 1.8 seconds
Output:
configs 103
['af_za', 'am_et', 'ar_eg', 'as_in', 'ast_es', 'az_az', 'be_by', 'bg_bg', 'bn_in', 'bs_ba', 'ca_es', 'ceb_ph', 'ckb_iq', 'cmn_hans_cn', 'cs_cz', 'cy_gb', 'da_dk', 'de_de', 'el_gr', 'en_us', 'es_419', 'et_ee', 'fa_ir', 'ff_sn', 'fi_fi', 'fil_ph', 'fr_fr', 'ga_ie', 'gl_es', 'gu_in', 'ha_ng', 'he_il', 'hi_in', 'hr_hr', 'hu_hu', 'hy_am', 'id_id', 'ig_ng', 'is_is', 'it_it', 'ja_jp', 'jv_id', 'ka_ge', 'kam_ke', 'kea_cv', 'kk_kz', 'km_kh', 'kn_in', 'ko_kr', 'ky_kg']


python3 - <<'PY'
from dotenv import load_dotenv
from pathlib import Path
load_dotenv(Path('.env'))
import os
if os.getenv('HF_TOKEN') and not os.getenv('HUGGINGFACE_HUB_TOKEN'):
    os.environ['HUGGINGFACE_HUB_TOKEN']=os.environ['HF_TOKEN']

from datasets import load_dataset

print('loading fleurs as_in test')
ds = load_dataset('google/fleurs','as_in',split='test',trust_remote_code=True)
print(ds)
print('columns', ds.column_names)
print('first', {k: type(v).__name__ for k,v in ds[0].items()})
print('text', ds[0].get('transcription') or ds[0].get('text') or ds[0].get('sentence'))
print('audio', ds[0].get('audio'))
PY