Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
configs 28
first 30 configs: ['malayalam', 'marathi', 'nepali', 'sindhi', 'tamil', 'telugu', 'assamese', 'bengali', 'bodo', 'dogri', 'gujarati', 'hindi', 'kannada', 'kashmiri', 'konkani', 'maithili', 'malayalam', 'manipuri', 'marathi', 'nepali', 'odia', 'punjabi', 'sanskrit', 'santali', 'sindhi', 'tamil', 'telugu', 'urdu']

 ai4bharat/Svarah configs 1
first 30: ['default']

 ai4bharat/Kathbath configs 12
first 30: ['bengali', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'sanskrit', 'tamil', 'telugu', 'urdu']

 ai4bharat/IndicVoices-ST configs 1
first 30: ['indic2en']


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Assessing dataset options**

I’m considering IndicVoices, which has configurations for multiple languages, and then there's Kathbath with 12 languages. I'll need to see the splits and fields and check the dataset information. I can use the `datasets` library to load these, but it seems it’s not currently installed. I should suggest installing both `datasets` and maybe `soundfile`. The user mentioned searching for benchmarks, so I can guide their other agent on that search.


**Gathering WER resources**

I can’t produce the WER pipeline right now, but I can offer baseline WER scores for models based on certain datasets from research papers. Additionally, AI4Bharat's Vistaar benchmark suite provides WER for baseline ASR models in 12 languages, which could be referenced for replication. I also need to check the license for the `ai4bharat/IndicVoices` dataset, which might not be fully open. Its Hugging Face metadata should clarify the license, so I'll check that along with Kathbath.


python3 - <<'PY'
import requests
for ds in ['ai4bharat/IndicVoices','ai4bharat/Kathbath','ai4bharat/Svarah','google/fleurs','mozilla-foundation/common_voice_17_0']:
    j=requests.get(f'https://huggingface.co/api/datasets/{ds}',timeout=20).json()
    card=j.get('cardData') or {}
    print(ds,'license',card.get('license'))
PY


Output:
Exit code: 0
Wall time: 1.1 seconds
Output:
ai4bharat/IndicVoices license cc-by-4.0
ai4bharat/Kathbath license cc-by-4.0
ai4bharat/Svarah license None
google/fleurs license ['cc-by-4.0']
mozilla-foundation/common_voice_17_0 license None


Summary: auto


python3 - <<'PY'
import requests
for query in ['MUCS', 'code switch hindi english speech', 'mucs hindi english']:
    url=f'https://huggingface.co/api/datasets?search={requests.utils.quote(query)}'
    items=requests.get(url,timeout=20).json()
    print('\nquery:',query,'count',len(items))
    for it in items[:10]:
        print(' -',it.get('id'))
PY


Output:
Exit code: 0
Wall time: 0.6 seconds
Output:

query: MUCS count 1
 - AdityK2409/mucs

query: code switch hindi english speech count 0

query: mucs hindi english count 0


**Evaluating language normalization**

I’m considering the need for language-specific normalization; the current `normalize_text` function removes punctuation, which might distort Word Error Rate (WER) for scripts. For evaluation, it seems wise to separate normalization for WER and Character Error Rate (CER).

We should evaluate "raw" transcripts against normalized ones. Standardized tokenization is a must for diverse languages since some, like Indic languages, may not use spaces effectively.

I'll look into using the default from `jiwer` that splits by whitespace and evaluate code modifications related to `CTCScore`.


**Checking CTC score metrics**

I'm looking at the `rank_candidates_by_ctc` function, which returns scores based on metrics like `token_coverage`, `token_score_mean`, and `span_mismatch`. I need to ensure that these fields are returned correctly.

Since we only saw partial outputs due to truncation, I'll check the entire `ctc_fit.py` file, particularly from around line 100 onward, to compute `token_coverage` and the other metrics fully. I'll make sure to verify that everything is working well!


nl -ba maya_transcribe/ctc_fit.py | sed -n '100,360p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   100
   101	        # Tokenize targets.
   102	        tokenizer = getattr(self._processor, "tokenizer", None)
   103	        if tokenizer is None:
   104	            raise ValueError("missing_tokenizer")
   105	        tokenized = tokenizer(text_norm, add_special_tokens=False)
   106	        target_ids = tokenized.get("input_ids")
   107	        if not target_ids:
   108	            raise ValueError("empty_targets")
   109
   110	        unk_id = getattr(tokenizer, "unk_token_id", None)
   111	        unk_ratio: float | None = None
   112	        if unk_id is not None:
   113	            unk_count = sum(1 for tid in target_ids if int(tid) == int(unk_id))
   114	            unk_ratio = float(unk_count) / float(len(target_ids))
   115	            if unk_ratio > max_unk_ratio:
   116	                raise ValueError(f"too_many_unk_tokens:{unk_ratio:.3f}")
   117
   118	        # torch ctc_loss expects (T,N,C)
   119	        log_probs_tnc = log_probs.transpose(0, 1)
   120	        device = log_probs_tnc.device
   121	        input_lengths = self._torch.tensor([log_probs_tnc.size(0)], dtype=self._torch.long, device=device)
   122	        targets = self._torch.tensor(target_ids, dtype=self._torch.long, device=device)
   123	        target_lengths = self._torch.tensor([targets.numel()], dtype=self._torch.long, device=device)
   124
   125	        # Reduction=mean normalizes by target length; good for comparability.
   126	        loss = F.ctc_loss(
   127	            log_probs_tnc,
   128	            targets,
   129	            input_lengths,
   130	            target_lengths,
   131	            blank=self.blank_id,
   132	            reduction="mean",
   133	            zero_infinity=True,
   134	        )
   135	        loss_value = float(loss.detach().cpu().item())
   136
   137	        # Forced alignment token scores (coverage/fit). Best-effort: alignment may fail for short audio vs long text.
   138	        token_score_mean = None
   139	        token_score_min = None
   140	        token_coverage = None
   141	        span_count = None
   142	        span_mismatch = None
   143
   144	        try:
   145	            import torchaudio
   146
   147	            with warnings.catch_warnings():
   148	                warnings.filterwarnings("ignore", message=".*forced_align.*deprecated.*", category=UserWarning)
   149	                aligned_tokens, aligned_scores = torchaudio.functional.forced_align(
   150	                    log_probs,
   151	                    targets.unsqueeze(0),
   152	                    input_lengths=input_lengths,
   153	                    target_lengths=target_lengths,
   154	                    blank=self.blank_id,
   155	                )
   156
   157	            spans = torchaudio.functional.merge_tokens(
   158	                aligned_tokens[0].detach().cpu(), aligned_scores[0].detach().cpu(), blank=self.blank_id
   159	            )
   160	            span_scores = [float(s.score) for s in spans]
   161	            span_count = len(span_scores)
   162	            span_mismatch = (span_count != int(target_lengths.item()))
   163	            if span_scores:
   164	                token_score_mean = sum(span_scores) / float(len(span_scores))
   165	                token_score_min = min(span_scores)
   166	                token_coverage = sum(1 for s in span_scores if s >= token_logp_threshold) / float(len(span_scores))
   167	        except Exception:
   168	            pass
   169
   170	        return CTCScore(
   171	            model_id=self.model_id,
   172	            loss=loss_value,
   173	            text=text,
   174	            unk_ratio=unk_ratio,
   175	            target_len=int(target_lengths.item()),
   176	            token_score_mean=token_score_mean,
   177	            token_score_min=token_score_min,
   178	            token_coverage=token_coverage,
   179	            token_logp_threshold=token_logp_threshold,
   180	            span_count=span_count,
   181	            span_mismatch=span_mismatch,
   182	        )
   183
   184
   185	class CTCFitScorer:
   186	    """
   187	    Cache-loaded CTC scoring models for audio-grounded transcript fit.
   188	    """
   189
   190	    def __init__(
   191	        self,
   192	        model_map: dict[str, str] | None = None,
   193	        default_model: str | None = None,
   194	        *,
   195	        max_unk_ratio: float = 0.2,
   196	        token_logp_threshold: float = -3.0,
   197	    ) -> None:
   198	        self.model_map = model_map or {}
   199	        self.default_model = default_model
   200	        self.max_unk_ratio = float(max_unk_ratio)
   201	        self.token_logp_threshold = float(token_logp_threshold)
   202	        self._cache: dict[str, _CTCModel] = {}
   203
   204	    def _get_model_id(self, lang: str | None) -> str | None:
   205	        if lang and lang in self.model_map:
   206	            return self.model_map[lang]
   207	        return self.default_model
   208
   209	    def score(self, audio_path: Path, text: str, lang: str | None) -> CTCScore | None:
   210	        model_id = self._get_model_id(lang)
   211	        if not model_id:
   212	            return None
   213
   214	        model = self._cache.get(model_id)
   215	        if model is None:
   216	            model = _CTCModel(model_id=model_id)
   217	            self._cache[model_id] = model
   218
   219	        return model.score(
   220	            audio_path=audio_path,
   221	            text=text,
   222	            lang=lang,
   223	            max_unk_ratio=self.max_unk_ratio,
   224	            token_logp_threshold=self.token_logp_threshold,
   225	        )
   226
   227
   228	def rank_candidates_by_ctc(
   229	    scorer: CTCFitScorer, audio_path: Path, candidates: dict[str, str], lang: str | None
   230	) -> dict[str, Any]:
   231	    scored: list[tuple[str, CTCScore]] = []
   232	    errors: dict[str, str] = {}
   233
   234	    for key, text in candidates.items():
   235	        text = (text or "").strip()
   236	        if not text:
   237	            continue
   238	        try:
   239	            s = scorer.score(audio_path=audio_path, text=text, lang=lang)
   240	            if s is None:
   241	                continue
   242	            scored.append((key, s))
   243	        except Exception as e:
   244	            errors[key] = f"{type(e).__name__}:{e}"
   245
   246	    scored.sort(key=lambda kv: kv[1].loss)
   247	    best = scored[0] if scored else None
   248
   249	    # Find the best *different-text* alternative, so duplicates don't create a fake "zero margin".
   250	    second = None
   251	    if best:
   252	        best_text = best[1].text
   253	        for item in scored[1:]:
   254	            if item[1].text != best_text:
   255	                second = item
   256	                break
   257
   258	    margin = (second[1].loss - best[1].loss) if (best and second) else None
   259
   260	    return {
   261	        "model_id": best[1].model_id if best else None,
   262	        "best_key": best[0] if best else None,
   263	        "best_loss": best[1].loss if best else None,
   264	        "second_best_key": second[0] if second else None,
   265	        "second_best_loss": second[1].loss if second else None,
   266	        "margin": float(margin) if margin is not None else None,
   267	        "scores": {
   268	            k: {
   269	                "loss": s.loss,
   270	                "model_id": s.model_id,
   271	                "unk_ratio": s.unk_ratio,
   272	                "target_len": s.target_len,
   273	                "token_score_mean": s.token_score_mean,
   274	                "token_score_min": s.token_score_min,
   275	                "token_coverage": s.token_coverage,
   276	                "token_logp_threshold": s.token_logp_threshold,
   277	                "span_count": s.span_count,
   278	                "span_mismatch": s.span_mismatch,
   279	            }
   280	            for k, s in scored
   281	        },
   282	        "errors": errors,
   283	    }


**Searching for datasets**

The user wants me to search the internet for benchmarks and datasets. I can start by gathering dataset suggestions and baseline WER numbers from reliable sources. I'm thinking of the AI4Bharat Vistaar paper and possibly dataset cards. I might try to download the arXiv PDF to search for "WER" and extract relevant tables. It could be challenging, but with the `pdftotext` tool installed, I can at least pull out some lines with WER values by checking if it's available first.


Output:
Exit code: 0
Wall time: 0 seconds
Output:


python3 - <<'PY'
import importlib
for m in ['pypdf','PyPDF2','pdfminer','pdfminer.high_level']:
    try:
        importlib.import_module(m)
        print(m,'available')
    except Exception as e:
        print(m,'not available')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
pypdf not available
PyPDF2 not available
pdfminer not available
pdfminer.high_level not available


**Searching for model evaluation results**

I’m thinking about fetching model cards for `ai4bharat/indicwhisper` to get the word error rate (WER) results. Many model cards on Hugging Face include evaluation results, so querying their API for the raw model card would be a good idea. It seems easier than dealing with a PDF. I’ll search Hugging Face models for "indicwhisper" to find out if this model card lists the WER per language. Let’s get started!


python3 - <<'PY'
import requests
url='https://huggingface.co/api/models?search=indicwhisper'
items=requests.get(url,timeout=20).json()
print('models',len(items))
for it in items[:20]:
    print('-',it.get('modelId'))
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
models 18
- sawradip/indicwhisper-m-hi-float16-ct2
- parthiv11/indic_whisper_hi_multi_gpu
- parthiv11/indic_whisper_nodcil
- Vignesh-M/Indic-whisper
- HariprasathSB/indic-whisper-vulnerable
- HariprasathSB/indic-whisper-vulnerable1
- HariprasathSB/indic-whisper-vulnerable2
- vrclc/indicwhisper-Mal
- kavyamanohar/AI4B-Indicwhisper-ml
- kavyamanohar/AI4B-Indicwhisper-or
- kritianandan/indic-whisper-kn-reverse-mft-1-0-1-8
- kritianandan/indic-whisper-kn-reverse-mft-1-0-1-9
- kritianandan/indic-whisper-kn-reverse-mft-1-0-1-10
- kritianandan/indic-whisper-reverse-ml-mft-1-0-1-11
- kritianandan/indic-whisper-reverse-ml-mft-1-0-1-13
- kritianandan/indic-whisper-reverse-ml-mft-1-0-1-14
- Koosh0610/indic-whisper-reverse-ml-mft-1-0-1-15
- adalat-ai/ct2-AI4B-Indicwhisper-ml


**Searching for model names**

I'm contemplating why I'm not finding the official `ai4bharat/indicwhisper-...`. It might be due to a naming difference. So, I'll go ahead and search specifically for `ai4bharat/indicwhisper` instead. Using HF search with the phrase "ai4bharat indicwhisper" could lead me to the information I need. Let’s see what that yields!


python3 - <<'PY'
import requests
url='https://huggingface.co/api/models?search=ai4bharat%2Findicwhisper'
items=requests.get(url,timeout=20).json()
print('models',len(items))
for it in items[:30]:
    print('-',it.get('modelId'))
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
models 11
- ai4bharat/indicwav2vec_v1_gujarati
- ai4bharat/indicwav2vec_v1_hindi
- ai4bharat/indicwav2vec_v1_marathi
- ai4bharat/indicwav2vec_v1_nepali
- ai4bharat/indicwav2vec_v1_odia
- ai4bharat/indicwav2vec_v1_sinhala
- ai4bharat/indicwav2vec_v1_tamil
- ai4bharat/indicwav2vec_v1_telugu
- ai4bharat/indicwav2vec_v1_bengali
- ai4bharat/indicwav2vec-hindi
- ai4bharat/indicwav2vec-odia


python3 - <<'PY'
import requests
items=requests.get('https://huggingface.co/api/models?search=ai4bharat%20whisper',timeout=20).json()
print('models',len(items))
for it in items[:40]:
    print('-',it.get('modelId'))
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
models 1
- soketlabs/whisper_indic_hi_ai4bharat


python3 - <<'PY'
import requests
for ds in ['ai4bharat/IndicVoices','ai4bharat/Kathbath','ai4bharat/Svarah']:
    url=f'https://huggingface.co/datasets/{ds}/raw/main/README.md'
    r=requests.get(url,timeout=20)
    print('\n',ds,'status',r.status_code,'len',len(r.text))
    print(r.text[:400].replace('\n',' ')+'...')
PY


Output:
Exit code: 0
Wall time: 0.6 seconds
Output:

 ai4bharat/IndicVoices status 401 len 131
Access to dataset ai4bharat/IndicVoices is restricted. You must have access to it and be authenticated to access it. Please log in....

 ai4bharat/Kathbath status 401 len 128
Access to dataset ai4bharat/Kathbath is restricted. You must have access to it and be authenticated to access it. Please log in....

 ai4bharat/Svarah status 401 len 126
Access to dataset ai4bharat/Svarah is restricted. You must have access to it and be authenticated to access it. Please log in....


**Identifying benchmark datasets**

The user asked for benchmarks on datasets we can use in this scenario, so I can suggest a few options. FLEURS is good for all languages, while Common Voice is useful but has some noise. Kathbath/IndicVoices offers more Indian speech, though it's gated. We could also consider Svarah for accents but it's gated too. OpenSLR has potential datasets like SLR64 for Marathi, and I'll need to search for Punjabi options. Ultimately, I can propose benchmarking with FLEURS, Common Voice, and AI4Bharat, while finding accuracy scores or WER scores.


**Accessing FLEURS dataset card**

I need to find the dataset card for the FLEURS dataset to check if it has baseline WER metrics for some models. It's open, so I can read through the first part for information. I’ll fetch the README from the provided URL to see what baseline metrics are documented. It should be straightforward to access, so let's go ahead and retrieve that data to see what insights we can uncover!


python3 - <<'PY'
import requests
url='https://huggingface.co/datasets/google/fleurs/raw/main/README.md'
r=requests.get(url,timeout=20)
print('status',r.status_code,'len',len(r.text))
text=r.text
# Print sections containing 'WER' or 'baseline'
import re
for m in re.finditer(r'(?i)wer', text):
    pass
# Just show lines with WER
lines=[ln for ln in text.splitlines() if 'WER' in ln or 'wer' in ln]
print('lines with wer:',len(lines))
for ln in lines[:40]:
    print(ln)
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 200 len 13336
lines with wer: 2
 'raw_transcription': 'Dit is nog nie huidiglik bekend watter aantygings gemaak sal word of wat owerhede na die seun gelei het nie maar jeugmisdaad-verrigtinge het in die federale hof begin',
 'transcription': 'dit is nog nie huidiglik bekend watter aantygings gemaak sal word of wat owerhede na die seun gelei het nie maar jeugmisdaad-verrigtinge het in die federale hof begin',


**Considering WER metrics**

I'm noticing that the dataset card doesn't provide WER metrics, so I might look at the FLEURS paper for details. Alternatively, searching for "OpenAI Whisper FLEURS WER" could yield results. The user might just want dataset suggestions; they might run a pipeline to compute WER themselves. They requested benchmarks and accuracy scores for their pipeline. It’s a bit ambiguous, so I could suggest open-source evaluation benchmarks with typical scores from model cards, along with instructions for a "benchmark harness."


**Evaluating the pipeline**

I’m considering whether we should run the pipeline on a small subset of FLEURS across languages to compute WER. That might need downloading the dataset and could require heavy computation, but maybe I can take a sample of about 10 per language. However, the user just wants to check whether the code is robust and get instructions. Still, running a small sample might validate the code. Using a smaller Whisper model like Tiny could help test the pipeline, although it won't compute WER. I could run a minimal benchmark for one language to ensure everything works.


**Addressing dataset requirements**

Running the pipeline requires the `datasets` library, which might not be installed, but I can pip install it since we’re network enabled, though it might take some time. The user mentioned not wanting to create new files, but we could run the code and create temporary directories if needed. The environment allows for downloading the dataset, and I could quickly attempt a small benchmark to ensure the pipeline functions. However, since the code is robust, I should just confirm that. I’ll highlight any caveats for the other agent about benchmark execution.


**Providing dataset and benchmarking options**

I’ve now fixed everything. I’ll suggest dataset options like FLEURS, Common Voice, and AI4Bharat's Kathbath/IndicVoices/Svarah, as well as OpenSLR corpora and MUCS for code-switching. For each dataset, I’ll include details such as relevant languages, style (read vs. conversational), licensing, accessibility, and where to access them along with approximate WER difficulty.

For benchmarking, I’ll outline an approach: First, choose datasets for each language; next, sample N clips from the test split, convert them to a structured directory, and create a refs.jsonl. Then, I’ll use the pipeline `maya-transcribe run` with the appropriate models and set CTC models per language.


**Mapping models for languages**

I’m considering model options from Hugging Face, like `ai4bharat/indicwav2vec_v1_hindi`, which is likely a CTC model that we can use for scoring. I'll also check for similar models for other Indian languages, though not all languages have dedicated models. For English, I’ll propose using `facebook/wav2vec2-base-960h`.

For Indian languages, a better approach could be `ai4bharat/indicwav2vec_v1_{language}` models for specific languages. If that's not available, I might suggest `facebook/mms-1b-all`, which is multilingual. However, for quick benchmarks, I can recommend using `facebook/mms-1b-all` as the default scoring model, keeping in mind that we need to verify script compatibility for evaluation. I’ll check if there’s a model for Malayalam, too, and use the HF API for that.


python3 - <<'PY'
import requests
for q in ['ai4bharat/indicwav2vec_v1_malayalam','indicwav2vec_v1_malayalam','indicwav2vec_v1_telugu','indicwav2vec_v1_kannada','indicwav2vec_v1_punjabi','indicwav2vec_v1_assamese']:
    items=requests.get('https://huggingface.co/api/models?search='+requests.utils.quote(q),timeout=20).json()
    print('\nquery',q,'count',len(items))
    for it in items[:10]:
        print(' -',it.get('modelId'))
PY


Output:
Exit code: 0
Wall time: 1.1 seconds
Output:

query ai4bharat/indicwav2vec_v1_malayalam count 11
 - ai4bharat/indicwav2vec_v1_gujarati
 - ai4bharat/indicwav2vec_v1_hindi
 - ai4bharat/indicwav2vec_v1_marathi
 - ai4bharat/indicwav2vec_v1_nepali
 - ai4bharat/indicwav2vec_v1_odia
 - ai4bharat/indicwav2vec_v1_sinhala
 - ai4bharat/indicwav2vec_v1_tamil
 - ai4bharat/indicwav2vec_v1_telugu
 - ai4bharat/indicwav2vec_v1_bengali
 - ai4bharat/indicwav2vec-hindi

query indicwav2vec_v1_malayalam count 1
 - ai4bharat/indicwav2vec_v1_marathi

query indicwav2vec_v1_telugu count 2
 - ai4bharat/indicwav2vec_v1_tamil
 - ai4bharat/indicwav2vec_v1_telugu

query indicwav2vec_v1_kannada count 0

query indicwav2vec_v1_punjabi count 0

query indicwav2vec_v1_assamese count 0


Summary: auto


**Exploring language models**

It looks like only a few languages have v1 models available: Gujarati, Hindi, Marathi, Nepali, Odia, Sinhala, Tamil, Telugu, and Bengali. Unfortunately, Malayalam, Kannada, Punjabi, and Assamese aren't included, which is a bit disappointing. For Kannada, I'm thinking we might consider `ai4bharat/indicwav2vec_v1_kannada`, but it seems it might not exist. Another option is the `ai4bharat/indicwav2vec_noa_hf`, which could support multiple languages. Let's check out that model type!


python3 - <<'PY'
import requests, json
model='ai4bharat/indicwav2vec_noa_hf'
url=f'https://huggingface.co/api/models/{model}'
j=requests.get(url,timeout=20).json()
print('id',j.get('modelId'))
print('pipeline_tag',j.get('pipeline_tag'))
print('tags sample', (j.get('tags') or [])[:20])
print('config', j.get('config',{}).get('model_type'))
print('library_name', j.get('library_name'))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
id None
pipeline_tag None
tags sample []
config None
library_name None


python3 - <<'PY'
import requests
model='ai4bharat/indicwav2vec_noa_hf'
url=f'https://huggingface.co/api/models/{model}'
r=requests.get(url,timeout=20)
print('status',r.status_code)
print(r.text[:200])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 401
{"error":"Invalid username or password."}


python3 - <<'PY'
import requests
model='facebook/mms-1b-all'
r=requests.get(f'https://huggingface.co/api/models/{model}',timeout=20)
print('status',r.status_code)
j=r.json()
print('pipeline_tag',j.get('pipeline_tag'))
print('library_name',j.get('library_name'))
print('model_type', (j.get('config') or {}).get('model_type'))
print('tags contains wav2vec2?', any('wav2vec2' in t for t in (j.get('tags') or [])))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
status 200
pipeline_tag automatic-speech-recognition
library_name transformers
model_type wav2vec2
tags contains wav2vec2? True


python3 - <<'PY'
from transformers import AutoProcessor
proc=AutoProcessor.from_pretrained('facebook/mms-1b-all')
tok=getattr(proc,'tokenizer',None)
print('tokenizer class', tok.__class__.__name__ if tok else None)
print('has set_target_lang', hasattr(tok,'set_target_lang'))
print('has lang2id', hasattr(tok,'lang2id'))
if hasattr(tok,'lang2id'):
    print('lang2id size', len(tok.lang2id))
    # show some keys for Indian languages if present
    for key in ['eng','hin','tam','tel','kan','mal','ben','asm','pan','guj','mar','ori','ory','or']:
        if key in tok.lang2id:
            print('lang2id contains', key)
PY


Output:
Exit code: 0
Wall time: 10.6 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 06:29:26.402920: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
tokenizer class Wav2Vec2CTCTokenizer
has set_target_lang True
has lang2id False


python3 - <<'PY'
from transformers import AutoProcessor
import inspect
proc=AutoProcessor.from_pretrained('facebook/mms-1b-all')
tok=proc.tokenizer
print('set_target_lang sig', inspect.signature(tok.set_target_lang))
print('target_lang', getattr(tok,'target_lang',None))
PY


Output:
Exit code: 0
Wall time: 9.9 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 06:29:43.205151: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
set_target_lang sig (target_lang: str)
target_lang eng


python3 - <<'PY'
from transformers import AutoProcessor
proc=AutoProcessor.from_pretrained('facebook/mms-1b-all')
tok=proc.tokenizer

def unk_ratio(text):
    ids=tok(text, add_special_tokens=False).get('input_ids')
    unk=tok.unk_token_id
    if not ids:
        return None
    return sum(1 for i in ids if i==unk)/len(ids)

sample_hi='यह हिंदी है'
print('default target_lang', tok.target_lang)
print('unk_ratio default:', unk_ratio(sample_hi))

tok.set_target_lang('hin')
print('target_lang after set', tok.target_lang)
print('unk_ratio hin:', unk_ratio(sample_hi))

sample_te='ఇది తెలుగు'
tok.set_target_lang('eng')
print('telugu unk_ratio eng:', unk_ratio(sample_te))

tok.set_target_lang('tel')
print('telugu unk_ratio tel:', unk_ratio(sample_te))
PY


Output:
Exit code: 0
Wall time: 9.7 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 06:30:02.465819: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
default target_lang eng
unk_ratio default: 0.8181818181818182
target_lang after set hin
unk_ratio hin: 0.0
telugu unk_ratio eng: 0.9
telugu unk_ratio tel: 0.0


**Addressing CTC scoring issues**

I need to ensure that our CTC scorer with mms-1b-all sets the target_lang to the correct language code; otherwise, the unknown token ratio will exceed the threshold, leading to candidate rejection. Currently, the CTC scorer doesn't set target_lang, which is critical for mms models before tokenization. The AutoProcessor in `ctc_fit.py` uses a default English tokenizer, causing a high unknown token ratio for non-English scripts. This leads to ValueErrors and empty scored entries, which can disrupt pipeline function.


**Reviewing multi-lingual scoring adjustments**

For our multi-lingual scoring, the scorer needs to set target_lang for mms models, or alternatively use per-language CTC model mapping that doesn't require it. We want to maintain the mms model as default. I think I should update the code to check if the processor's tokenizer has a method for setting the target language and call it with the right code before tokenization.

The user claims this issue is resolved, mentioning "CTC scoring guardrails" but didn't address target_lang. The code indeed doesn't set it, which is a key point to share. If they opt for monolingual models, then target_lang wouldn't be necessary, but for languages without those models, we still need it for mms.


**Communicating CTC gating requirements**

I need to let them know that while the new CTC gating is effective, it’s essential to set `processor.tokenizer.set_target_lang` to the language code before tokenization for MMS CTC models. The `lang` variable from the pipeline uses an MMS-LID label code, like `'eng'`, `'hin'`, and `'tel'`, which seems to match our language codes.

I should confirm the correct MMS-LID labels for languages like Hindi (`hin`), Bengali (`ben`), Telugu (`tel`), and more. The `route_lang` derived from LID will be utilized for Whisper language hints, CTC scoring, and possibly with Gemini too. For CTC scoring, calling `set_target_lang(route_lang)` should work since tokenizers may expect three-letter codes.


**Updating language model considerations**

There seems to be a discrepancy between LID codes and what the models expect; for instance, Punjabi might use 'pan.' I think we should update the code accordingly, but since the user wants review comments rather than code changes right now, I can highlight this as critical.

The `text_norm` in the CTC scorer uses `normalize_text` to remove punctuation and normalize digits, which is beneficial. However, for MMS languages, we might want to maintain script integrity. I also need to check `_force_uppercase`, which benefits English but may not suit Indian scripts. Additionally, the gate relies on CTC coverage and token scores, which will require proper target_lang setting in the tokenizer for MMS models.


**Searching for benchmark datasets**

I see a potential issue with `span_count != target_len` for CTC, where `span_mismatch` gets flagged as true, indicating a mismatch, and then requires human verification if we skip. So, I need to look for benchmark datasets, potentially referencing Vistaar.

I think I should inspect AI4Bharat's Vistaar for an official repository or dataset list, possibly on GitHub with an evaluation script. Using `requests` to search GitHub seems fine, but it might be better to explore their website directly for dataset names. I'll quickly check `https://ai4bharat.iitm.ac.in/areas/asr` for the user’s requested benchmark datasets and parse that information.


python3 - <<'PY'
import requests, re
url='https://ai4bharat.iitm.ac.in/areas/asr'
html=requests.get(url,timeout=20).text
print('status ok, len',len(html))
# Find mentions of datasets keywords
keywords=['Vistaar','IndicVoices','Kathbath','Svarah','MUCS','Bhashini','Common Voice','FLEURS']
for kw in keywords:
    if kw.lower() in html.lower():
        print('found',kw)

# Print a few lines around 'Vistaar'
idx=html.lower().find('vistaar')
if idx!=-1:
    snippet=html[max(0,idx-200):idx+400]
    snippet=re.sub('<[^>]+>',' ',snippet)
    print('\nSnippet around Vistaar:')
    print(' '.join(snippet.split())[:600])
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
status ok, len 61359
found Vistaar
found Svarah

Snippet around Vistaar:
like YouTube. This extensive dataset empowers us to address the complexities of India's linguistic landscape effectively. Our focus on building robust benchmarks is exemplified by our work with Vistaar, IndicSUPERB, Lahaja, and Svarah, which have set new standards in ASR evaluation. Our state-of-the-art models include IndicWav2Vec, IndicWhisper, and IndicConformer, with our latest model supporting all 22 languages and demonstrating our commitment to technological excellence. Moving forward, we aim to enhance our models to handle 8KZ telephony data, adapt them for specific domains and dem


**Searching for datasets**

The page mentions IndicSUPERB and Lahaja as benchmarks for our needs. I think IndicSUPERB is a benchmark suite for Indic languages, similar to another suite called SUPERB. Lahaja might refer to a dataset. I can look for specific links on the page related to these names. My plan involves parsing the HTML to find any 'href' attributes that include either IndicSUPERB or Lahaja. Time to get started on that!


python3 - <<'PY'
import requests,re
html=requests.get('https://ai4bharat.iitm.ac.in/areas/asr',timeout=20).text
for kw in ['Vistaar','IndicSUPERB','Lahaja','Svarah']:
    print('\n--',kw,'--')
    # find first 3 occurrences with nearby href
    pattern=re.compile(r'.{0,200}'+re.escape(kw)+r'.{0,200}', re.I|re.S)
    m=pattern.search(html)
    if not m:
        print('not found snippet')
        continue
    sn=m.group(0)
    # extract links in snippet
    links=re.findall(r'href="([^"]+)"', sn)
    print('links:', links[:5])
    text=re.sub('<[^>]+>',' ',sn)
    print('text snippet:', ' '.join(text.split())[:300])
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:

-- Vistaar --
links: []
text snippet: like YouTube. This extensive dataset empowers us to address the complexities of India's linguistic landscape effectively. Our focus on building robust benchmarks is exemplified by our work with Vistaar, IndicSUPERB, Lahaja, and Svarah, which have set new standards in ASR evaluation. Our state-o

-- IndicSUPERB --
links: []
text snippet: Tube. This extensive dataset empowers us to address the complexities of India's linguistic landscape effectively. Our focus on building robust benchmarks is exemplified by our work with Vistaar, IndicSUPERB, Lahaja, and Svarah, which have set new standards in ASR evaluation. Our state-of-the-ar

-- Lahaja --
links: []
text snippet: tensive dataset empowers us to address the complexities of India's linguistic landscape effectively. Our focus on building robust benchmarks is exemplified by our work with Vistaar, IndicSUPERB, Lahaja, and Svarah, which have set new standards in ASR evaluation. Our state-of-the-art models incl

-- Svarah --
links: []
text snippet: set empowers us to address the complexities of India's linguistic landscape effectively. Our focus on building robust benchmarks is exemplified by our work with Vistaar, IndicSUPERB, Lahaja, and Svarah, which have set new standards in ASR evaluation. Our state-of-the-art models include IndicWav


**Searching for Svarah links**

It seems there aren't any links in the snippet, which might mean the relevant dataset pages aren't directly linked. I'll need to search for "Svarah" in the broader HTML to find any anchor tags that reference it. Since the snippet didn't include them, I think it's best to look for sections with links. So, I'll search specifically for "Svarah" and "href" together to find more context. Let's proceed with that search!


python3 - <<'PY'
import requests,re
html=requests.get('https://ai4bharat.iitm.ac.in/areas/asr',timeout=20).text
kw='Svarah'
for m in re.finditer(kw, html):
    start=max(0,m.start()-400)
    end=min(len(html), m.end()+400)
    sn=html[start:end]
    links=re.findall(r'href="([^"]+)"', sn)
    if links:
        print('found links near occurrence:', links[:10])
        text=re.sub('<[^>]+>',' ',sn)
        print('text:', ' '.join(text.split())[:300])
        break
else:
    print('no links')
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
no links


Summary: auto


**Searching for datasets**

I’m thinking the AI4Bharat site might be using JavaScript and doesn’t include links. Still, I can propose some benchmarks: Vistaar, IndicSUPERB, Lahaja, and Svarah. However, it’s crucial that I find an open-source dataset or code to utilize. I should search Hugging Face for these names, starting with "Lahaja" and "IndicSUPERB." Let’s dive in and see what I can discover!


python3 - <<'PY'
import requests
for term in ['IndicSUPERB','Lahaja','Svarah','Vistaar']:
    items=requests.get('https://huggingface.co/api/datasets?search='+requests.utils.quote(term),timeout=20).json()
    print('\nsearch',term,'datasets',len(items))
    for it in items[:15]:
        print(' -',it.get('id'))
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:

search IndicSUPERB datasets 5
 - makaveli10/indic-superb-whisper
 - collabora/indic-superb
 - alekya/IndicSUPERB
 - alekya/indic_superb
 - makaveli10/indic-superb-wds

search Lahaja datasets 3
 - ai4bharat/Lahaja
 - Saranath07/test-lahaja
 - SayantanJoker/lahaja

search Svarah datasets 11
 - ai4bharat/Svarah
 - Bhargav0044/svarah1
 - saisundaresan01/svarah
 - saisundaresan01/svarah-eval
 - blastman7/svarah_use
 - amrithanandini/svarah
 - KingNish/svarah_snac_codes
 - ahamedddd/svarah_processed
 - tinisoft/svarah-with-asl-gloss
 - tanushm/svarah_neucoded
 - Sannidhya3/word_phoneme_aligned_svarah

search Vistaar datasets 4
 - iiBLACKii/Gujarati_Vistaar_DB
 - ekacare/vistaar_small_asr_eval
 - AdityK2409/vistaar_small_asr_eval
 - AdityK2409/vistaar_small_asr_eval_2


I'm looking into datasets like `ai4bharat/Lahaja` and `ai4bharat/Svarah`, which seems promising. I’ve also noted that the IndicSUPERB dataset exists under `collabora/indic-superb`, which is great! Now, I need to verify whether `ai4bharat/Lahaja` is gated. I’ll do this by checking through the HF API to make sure I have access to everything I need. Let’s get started on that!


python3 - <<'PY'
import requests
for ds in ['ai4bharat/Lahaja','collabora/indic-superb','alekya/IndicSUPERB','mozilla-foundation/common_voice_17_0','google/fleurs']:
    url=f'https://huggingface.co/api/datasets/{ds}'
    r=requests.get(url,timeout=20)
    print(ds,'status',r.status_code)
    if r.status_code==200:
        card=r.json().get('cardData') or {}
        print(' license',card.get('license'))
        configs=(card.get('configs') or [])
        if configs:
            print(' configs',len(configs),'sample',[c.get('config_name') for c in configs[:10]])
PY


Output:
Exit code: 0
Wall time: 1.9 seconds
Output:
ai4bharat/Lahaja status 200
 license mit
 configs 1 sample ['default']
collabora/indic-superb status 200
 license None
 configs 1 sample ['default']
alekya/IndicSUPERB status 200
 license None
mozilla-foundation/common_voice_17_0 status 200
 license None
google/fleurs status 200
 license ['cc-by-4.0']


**Exploring dataset access**

Alright, so the `ai4bharat/Lahaja` dataset is accessible, which is great! It seems to have an MIT license, but I need to confirm since it’s a dataset, not code. I’m curious about whether it includes audio and transcripts. I should check the `configs` to see if there’s a default setup. I think I’ll try fetching the README file raw to see what's available. Let's continue with that!


python3 - <<'PY'
import requests
url='https://huggingface.co/datasets/ai4bharat/Lahaja/raw/main/README.md'
r=requests.get(url,timeout=20)
print('status',r.status_code,'len',len(r.text))
print(r.text[:600])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 401 len 126
Access to dataset ai4bharat/Lahaja is restricted. You must have access to it and be authenticated to access it. Please log in.


python3 - <<'PY'
import requests
url='https://huggingface.co/datasets/collabora/indic-superb/raw/main/README.md'
r=requests.get(url,timeout=20)
print('status',r.status_code,'len',len(r.text))
print(r.text[:600])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 200 len 642
---
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
dataset_info:
  features:
  - name: audio
    dtype: audio
  - name: transcription
    dtype: string
  - name: duration
    dtype: float64
  splits:
  - name: train
    num_bytes: 46776721194.64
    num_examples: 24872
  - name: test
    num_bytes: 1592137067.0
    num_examples: 872
  download_size: 46065024050
  dataset_size: 48368858261.64
---
# Dataset Card for "indic-superb"

[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.


**Evaluating paper parsing options**

I’m considering how to fetch the Vistaar paper and parse its tables, possibly using `pandoc` or `latex`. Installing `pypdf` is also an option, especially for extracting the WER table from the PDF. Since my environment allows for pip installations, that could work, but it might take some time. The user specifically asked for benchmarks and scores, so I’ll focus on extracting relevant numbers. I’ll look into what WER ranges the Vistaar paper includes.


**Installing PDF parsing library**

I plan to run `python3 -m pip install pypdf -q` since my network is enabled and I can proceed with this installation. Even though the user didn't specifically ask for any changes to the environment, installing `pypdf` will allow for text extraction from PDFs. This package is lightweight, which is good because I want to avoid anything too big. After installing, I can download the Vistaar PDF and look for specific tables related to WER, though I might also consider using arXiv's LaTeX source for better reliability.


**Downloading and extracting source files**

I’m considering fetching the ArXiv e-print source to extract WER values, as it might be simpler than dealing with the PDF. I can download the tar.gz file and extract it to a temporary folder like /tmp to keep things tidy. Using the `tar` command will be sufficient without needing additional libraries. I'll download the Vistaar source from `https://arxiv.org/e-print/2305.15386` using either `curl -L` or `requests`, then extract the contents to look for WER patterns.


set -euo pipefail
mkdir -p /tmp/vistaar_src
cd /tmp/vistaar_src
rm -f 2305.15386.tar.gz
curl -L -o 2305.15386.tar.gz https://arxiv.org/e-print/2305.15386
ls -lh 2305.15386.tar.gz
file 2305.15386.tar.gz
tar -tzf 2305.15386.tar.gz | head


Output:
Exit code: 141
Wall time: 0.5 seconds
Output:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0  894k    0  8259    0     0  20306      0  0:00:45 --:--:--  0:00:45 20292
100  894k  100  894k    0     0  1746k      0 --:--:-- --:--:-- --:--:-- 1746k
-rw-rw-r-- 1 ubuntu ubuntu 895K Jan 13 06:32 2305.15386.tar.gz
2305.15386.tar.gz: gzip compressed data, last modified: Thu Aug  3 00:20:52 2023, from Unix, original size modulo 2^32 1146880
camera_ready.pdf
figure1.pdf
figure1.svg
figure.pdf
for_review.pdf
IEEEtran.bst
INTERSPEECH2023.cls
mybib.bib
plot1_backup.png
plot1.png


**Analyzing command outcomes**

I see that command exit code 141 means the pipe was broken. The `tar -tzf ... | head` command produced 10 lines, but head closing the pipe caused `tar` to receive a SIGPIPE resulting in that exit code. That's okay. The extracted listing shows `camera_ready.pdf` but not any .tex files? Maybe the source is in PDF format, which is common for many arXiv entries. I should check further by running `tar -tzf ... | rg '\\.tex$'` to search for any .tex files.


cd /tmp/vistaar_src
# list tex files
tex=$(tar -tzf 2305.15386.tar.gz | rg -n '\.tex$' || true)
printf '%s\n' "$tex" | head
# list top-level files
ls -la


Output:
Exit code: 0
Wall time: 0 seconds
Output:
12:styles.tex
14:template.tex
total 944
drwxrwxr-x   2 ubuntu ubuntu   4096 Jan 13 06:32 .
drwxrwxrwt 121 root   root    40960 Jan 13 06:32 ..
-rw-rw-r--   1 ubuntu ubuntu 915608 Jan 13 06:32 2305.15386.tar.gz


**Extracting tex files**

I need to extract the `.tex` files, specifically `styles.tex` and `template.tex`. To do that, I’ll use the command `tar -xzf` to extract them to the `/tmp/vistaar_src` directory. It seems like I should only extract the `.tex` and `.bib` files for now. Let's go ahead with this plan! I hope everything goes smoothly as I take these steps.


set -euo pipefail
cd /tmp/vistaar_src
rm -rf extracted
mkdir extracted
cd extracted
tar -xzf ../2305.15386.tar.gz styles.tex template.tex mybib.bib || true
ls -la
wc -l *.tex *.bib || true
sed -n '1,120p' template.tex


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 80
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 13 06:32 .
drwxrwxr-x 3 ubuntu ubuntu  4096 Jan 13 06:32 ..
-rw-rw-r-- 1 ubuntu ubuntu 18543 Aug  2  2023 mybib.bib
-rw-rw-r-- 1 ubuntu ubuntu 12645 Aug  2  2023 styles.tex
-rw-rw-r-- 1 ubuntu ubuntu 36006 Aug  2  2023 template.tex
  418 styles.tex
  460 template.tex
  464 mybib.bib
 1342 total
\documentclass{INTERSPEECH2023}

% 2023-01-06 modified by Simon King (Simon.King@ed.ac.uk)

% **************************************
% *    DOUBLE-BLIND REVIEW SETTINGS    *
% **************************************
% Comment out \interspeechcameraready when submitting the
% paper for review.
% If your paper is accepted, uncomment this to produce the
%  'camera ready' version to submit for publication.
\interspeechcameraready

\usepackage{wasysym}
\usepackage{hyperref}
% \hypersetup{
%     colorlinks=true,
%     linkcolor=blue,
%     filecolor=magenta,
%     urlcolor=cyan,
%     }

% **************************************
% *                                    *
% *      STOP !   DO NOT DELETE !      *
% *          READ THIS FIRST           *
% *                                    *
% * This template also includes        *
% * important INSTRUCTIONS that you    *
% * must follow when preparing your    *
% * paper. Read it BEFORE replacing    *
% * the content with your own work.    *
% **************************************

\title{Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR}
\name{Kaushal Santosh Bhogale$^1$*, Sai Sundaresan$^2$*\dag \thanks{* The first two authors have contributed equally.}\thanks{\dag Work was done when Sai was at AI4Bharat}, Abhigyan Raman$^3$, Tahir Javed$^4$, Mitesh M. Khapra$^5$, Pratyush Kumar$^6$}
%The maximum number of authors in the author list is 20. If the number of contributing authors is more than this, they should be listed in a footnote or the acknowledgement section.
\address{
$^{1,2,3,4,5,6}$AI4Bharat, India \\
  $^{1,4,5,6}$Indian Institute of Technology Madras, India\\
  $^{2}$Indian Institute of Technology Kharagpur, India\\
  $^{6}$Microsoft, India
  }
\email{cs22d006@cse.iitm.ac.in, {tahirjmakhdoomi,saisundaresan01,ramanabhigyan}@gmail.com, {miteshk,pratyush}@cse.iitm.ac.in}
\begin{document}

\maketitle

\begin{abstract}
% 1000 characters. ASCII characters only. No citations.
Improving ASR systems is necessary to make new LLM-based use-cases accessible to people across the globe.
In this paper, we focus on Indian languages, and make the case that diverse benchmarks are required to evaluate and improve ASR systems for Indian languages.
To address this, we collate Vistaar as a set of 59 benchmarks across various language and domain combinations, on which we evaluate 3 publicly available ASR systems and 2 commercial systems.
We also train IndicWhisper models by fine-tuning the Whisper models on publicly available training datasets across 12 Indian languages totalling to 10.7K hours.
We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark.
Indeed, IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER.
We open-source all datasets, code and models - \url{https://github.com/AI4Bharat/vistaar}

% Recent advances in ASR have shown that training on a large and diverse weakly-supervised dataset can lead to robust systems with zero-shot transfer capabilities. This enables new possibilities for low-resource Indian languages where labeled data is scarce and collecting new datasets is expensive, but effective mining techniques can be adopted to created weakly-supervised datasets from publicly available sources. We leverage this to horizontally scale ASR ecosystem for Indian languages on three axes; training datasets, benchmarks and models. First, we collate data from multiple sources across 12 Indian languages. Second, we evaluate our models across 59 benchmarks consisting of diverse domains, speaking styles and audio quality. Third, we train large-scale models and compare it against 14 publicly available models and 2 commercial services. Our work establishes the effectiveness of training on weakly-supervised datasets, and the need for better benchmarks for Indian languages.

\end{abstract}
% \noindent\textbf{Index Terms}: speech recognition, weakly-supervised, low-resources

\section{Introduction}

Large Language Models such as GPT3\cite{brown2020language} have demonstrated emergent behaviors which unlock new use-cases for deployment of language AI.
For instance, an Indian villager could interact with a LLM-based system to learn about rights and entitlements on existing government schemes.
To make such use-cases accessible to a large population, it is essential to make them voice-enabled, which requires support for speech recognition models across languages.
Specifically, for Indian languages, given the large linguistic diversity (60 languages with over a million speakers) and the large print illiterate population (over 300 million people \cite{census}), accurate ASR systems have significant societal impact.

Improvement of Machine Learing systems like ASR are dependent on various choices of training architectures, algorithms, datasets, and metrics.
Systematic evaluation of these choices is critically dependent on representative benchmarks.
Several works, over many years, have contributed publicly available benchmarks of specific languages and specific domains/types of data.
For instance, Gramvaani \cite{gramvaani}, a Hindi dataset, contains data from agriculture domain; Vakyasancayah \cite{vakyasancayah}, a Sanskrit dataset, contains data from literature domain.
Often these benchmarks are accompanied by leaderboards which earmark performance of different ASR systems.
We make the case that such narrow comparison incentivizes model optimization to over-fit for the benchmark's characteristics.
For instance, tuning of language models to the dev set of benchmarks have been shown to reduce WER scores by over 6.5 points averaged across 8 languages \cite{indicwav2vec}.
This suggests that narrow benchmarking may incentivize such over-fitting and thereby preclude more robust comparison of models.

To address some of these challenges, we propose collation of benchmarks across languages and domains/types of data.
We call this Vistaar (meaning broad in Hindi) and it comprises of publicly available benchmarks across 12 languages, leading to 59 computed WER values across benchmarks and languages.
We show in Figure~\ref{fig:vistaar} the diversity of the languages and domains/types covered by these benchmarks.
We will host these datasets in a public resource and release source-code to evaluate an ASR system across these benchmarks.

\begin{figure}[t!]
    \centering
    \includegraphics[width=0.8\columnwidth]{figure1.pdf}
    \caption{The diversity domains/types covered by the Vistaar benchmark. (CV: CommonVoice;  VS: Vakyasancayah)}
    \label{fig:vistaar}
\end{figure}

We then evaluate various ASR systems on Vistaar.
Specifically, we consider 3 publicly released ASR systems and 2 commercial systems from Google and Microsoft.
We report the results across models and find large variance across benchmarks, languages, and models.
Specifically, we show that comparing models based on one benchmark may be highly uncorelated with results on another benchmark.
We hope that the Vistaar benchmark, and easy access to it, enables robust evaluation of ASR systems.

Based on our insight that ASR systems' performance varies across domains, we recognize value in training ASR systems with a wide set of training sets.
To this end, we curate all publicly available training sets for 12 Indian languages amounting over 10,700 hours of audio.
We call this the Vistaar-train dataset.
We train the Whisper ASR model \cite{whisper} with Vistaar-train for each of the 12 languages, to create a family of IndicWhisper models.
When evaluated on Vistaar benchmark, we find that IndicWhisper models have the lowest average WER across benchmarks for each of the 12 languages.
Indeed, IndicWhipser has the lowest WER in 39 out of the 59 benchmarks with an average reduction of 4.1 WER.
We specifically compare IndicWhisper to one publicly available (IndicWav2Vec) and one commercial (Google) ASR system in Figure~\ref{fig:vistaar-evaluation}.
As shown, the reported WER is lower in IndicWhisper for most of the benchmarks, with large gaps for some of the benchmarks.
This validates our proposition that to both evaluate and train ASR systems we need diversity in datasets.
% We will open source all datasets, code, and models from this work.

The rest of the paper is organized as follows.
% In Section~\ref{sec:evaluation}, we quantitatively discuss the challenges with evaluating ASR systems with single benchmarks.
We detail all benchmark datasets in Vistaar in Section~\ref{sec:vistaar}.
In Section~\ref{sec:vistaar-train} we detail the training dataset and the procedure used to train IndicWhisper.
We compare various public and commercial ASR systems along with IndicWhisper on Vistaar in Section~\ref{sec:evaluation}.
We end with conclusions and our thought-process on further improvements towards building open Indic ASR systems.

% \section{Evaluation of ASR systems on Individual Benchmarks}
% \label{sec:evaluation}
% In this section, we discuss the


cd /tmp/vistaar_src/extracted
rg -n "WER|CommonVoice|FLEURS|Kathbath|Svarah|Lahaja|IndicSUPERB|Vistaar" template.tex | head -n 60


Output:
Exit code: 0
Wall time: 0 seconds
Output:
35:\title{Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR}
53:To address this, we collate Vistaar as a set of 59 benchmarks across various language and domain combinations, on which we evaluate 3 publicly available ASR systems and 2 commercial systems.
55:We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark.
56:Indeed, IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER.
78:For instance, tuning of language models to the dev set of benchmarks have been shown to reduce WER scores by over 6.5 points averaged across 8 languages \cite{indicwav2vec}.
82:We call this Vistaar (meaning broad in Hindi) and it comprises of publicly available benchmarks across 12 languages, leading to 59 computed WER values across benchmarks and languages.
89:    \caption{The diversity domains/types covered by the Vistaar benchmark. (CV: CommonVoice;  VS: Vakyasancayah)}
93:We then evaluate various ASR systems on Vistaar.
97:We hope that the Vistaar benchmark, and easy access to it, enables robust evaluation of ASR systems.
101:We call this the Vistaar-train dataset.
102:We train the Whisper ASR model \cite{whisper} with Vistaar-train for each of the 12 languages, to create a family of IndicWhisper models.
103:When evaluated on Vistaar benchmark, we find that IndicWhisper models have the lowest average WER across benchmarks for each of the 12 languages.
104:Indeed, IndicWhipser has the lowest WER in 39 out of the 59 benchmarks with an average reduction of 4.1 WER.
106:As shown, the reported WER is lower in IndicWhisper for most of the benchmarks, with large gaps for some of the benchmarks.
112:We detail all benchmark datasets in Vistaar in Section~\ref{sec:vistaar}.
114:We compare various public and commercial ASR systems along with IndicWhisper on Vistaar in Section~\ref{sec:evaluation}.
121:\section{Vistaar Benchmark Set}
127:\caption{The Vistaar Benchmark Set - 59 benchmarks over 12 Indian languages }
133:Kathbath    & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{c}{\CheckedBox} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
134:Kathbath-Hard    & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{c}{\CheckedBox} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
135:FLEURS      & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 &                          & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
136:CommonVoice & \CheckedBox                 & \multicolumn{1}{l}{} & \CheckedBox & \multicolumn{1}{l}{} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 &                          & \CheckedBox                 & \multicolumn{1}{l}{} & \CheckedBox                 \\
147:    \caption{\textbf{WER Comparison on Vistaar:} IndicWhisper is better than IndicWav2Vec in 45 out of 59 benchmarks (highlighted by purple) and better than Google STT in 57 out of 59 benchmarks.}
151:In the following we describe various publicly available datasets which we curate as part of the Vistaar benchmark set.
153:\noindent \textbf{Kathbath \cite{kathbath}} This contains \textit{read speech} collected on Android phones with the Karya application \cite{iitb_msr} by AI4Bharat, where the sentences are sourced from Wikipedia and news articles.
156:\noindent \textbf{Kathbath-Hard \cite{kathbath}} We create a hard benchmark for  by adding background noise of various types to the audio files of the Kathbath benchmark.
160:\noindent \textbf{FLEURS \cite{fleurs}} This contains \textit{read speech of translated Wikipedia content} with 3 recordings by different speakers for a sentence and manual validation collected by researchers at Meta and their collaborators.
163:\noindent \textbf{CommonVoice \cite{commonvoice}} This contains \textit{crowd-sourced read speech} from the popular Commonvoice website of Mozilla Foundation.
180:The Spearman correlation of the 14 WER values of the 14 ASR systems on the benchmarks were computed pair-wise.
182:For instance, CommonVoice and IndicTTS had a negative correlation of -0.26 indicating that the ranking of models on CommonVoice is not informative of ranking of models on IndicTTS.
183:This low correlation is perhaps attributable to the diversity in content and speakers of the two benchmarks - while CommonVoice is crowdsources, IndicTTS is collected from professional speakers in a studio environment.
187:\section{Vistaar-Train Dataset and IndicWhisper Model Training}
193:\caption{The Vistaar-Train Dataset With Hours of data across Datasets and Languages}
200:Kathbath          & 103 & 116 & 137  & 153  & 134 & 172  & 99  & 124 & 102 & 172  & 142 & 74  & 1527  \\
201:CommonVoice       & 64  & -   & 13   & -    & 1   & 19   & 2   & 2   & -   & 226  & -   & 46  & 373   \\
208:FLEURS            & 11  & 9   & 7    & 8    & 10  & 12   & 3   & 6   & -   & 9    & 8   & 7   & 90    \\
221:\caption{Comparison of publicly-available models on the Hindi subset of the Vistaar benchmark}
228:Model         & Kathbath   & Kathbath-Hard & FLEURS & CommonVoice & IndicTTS & MUCS & Gramvaani & Average  \\
239:% \multicolumn{9}{l}{(KB: Kathbath, FL: FLEURS, CV: CommonVoice, I: Indic, GV: Gramvaani)} \\
250:% \caption{Word error rates (\%) of IndicWhisper on the Vistaar Benchmark. Values in bold indicates benchmarks where IndicWhisper has the lowest WER.}
256:% Kathbath      & 16.6 & 17.8 & \textbf{10.3} & \textbf{19.3} & \textbf{34.8} & \textbf{19.9} & 24.7 & 16.9 & 45.6 & \textbf{24.2} & 25.0 & \textbf{11.9} & 22.3 \\
257:% Kathbath Hard & 19.4 & \textbf{20.6} & \textbf{12.0} & \textbf{22.2} & \textbf{38.4} & \textbf{22.1} & 29.1 & 19.7 & 50.5 & \textbf{27.5} & 27.8 & \textbf{14.7} & 25.3 \\
258:% CommonVoice   & 24.7 & -    & \textbf{11.4} & -    & \textbf{44.5} & 22.8 & \textbf{35.2} & 22.4 & -    & \textbf{29.2} & -    & 31.7 & 27.8 \\
259:% FLEURS        & \textbf{20.9} & 23.5 & \textbf{15.0} & \textbf{18.6} & \textbf{22.6} & \textbf{20.5} & \textbf{32.9} & \textbf{23.1} & -    & \textbf{25.2} & \textbf{25.4} & \textbf{19.2} & 22.5 \\
273:\caption{Performance of Publicly available models  on the Vistaar Benchmark}
281:Kathbath      & 20.6 & 26.0 & 14.3 & 27.9 & 52.6 & 26.6 & 105.7 & 25.5 & 104.2 & 31.1 & 34.3 & 17.6 & 40.5 \\
282:Kathbath Hard & 22.4 & 28.4 & 16.6 & 31.4 & 53.6 & 28.4 & 105.4 & 29.2 & 104.4 & 32.8 & 35.9 & 20.2 & 42.4 \\
283:CommonVoice   & 26.1 & -    & 20.8 & -    & 54.0 & 27.5 & 101.7 & 29.2 & -     & 42.1 & -    & 26.9 & 41.0 \\
284:FLEURS        & 27.3 & 34.7 & 19.4 & 33.8 & 41.0 & 33.2 & 100.8 & 44.0 & -     & 36.6 & 40.1 & 28.4 & 39.9 \\
293:Kathbath      & 16.7 & 22.4 & 13.6 & 23.6 & 43.3 & 20.9 & NA & NA & NA & 28.1 & 22.8 & NA & - \\
294:Kathbath Hard & 18.4 & 26.7 & 14.9 & 25.9 & 45.7 &      & NA & NA & NA & 30.0 & 24.6 & NA & - \\
295:CommonVoice   & 15.0 & -    & 14.6 & -    & 47.5 & 19.6 & NA & NA & NA & 39.9 & -    & NA & - \\
296:FLEURS        & 29.9 & 34.8 & 24.3 & 31.0 & 39.2 & 37.2 & NA & NA & NA & 34.9 & 32.3 & NA & - \\
305:Kathbath      & 14.3 & 17.4 & 12.2 & 20.8 & 37.8 & 20.4 & 22.2 & 15.1 & 37.0 & 27.4 & 26.9 & 16.0 & 22.3 \\
306:Kathbath Hard & 18.1 & 21.5 & 16.2 & 26.4 & 42.8 & 24.0 & 28.3 & 18.2 & 43.3 & 33.0 & 31.2 & 19.4 & 26.9 \\
307:FLEURS        & 28.4 & 22.7 & 18.3 & 22.0 & 40.4 & 25.1 & 40.5 & 34.8 & -    & 37.7 & 30.4 & 32.2 & 30.2 \\
308:CommonVoice   & 31.3 & -    & 20.2 & -    & 67.0 & 27.0 & 37.9 & 19.7 & -    & 42.7 & -    & 32.7 & 34.8 \\
317:Kathbath      & 16.6 & 17.8 & 10.1 & 19.3 & 34.8 & 19.9 & 24.7 & 16.9 & 45.6 & 24.2 & 25.0 & 11.9 & 22.2 \\
318:Kathbath Hard & 19.4 & 20.6 & 11.9 & 22.2 & 38.4 & 22.1 & 29.1 & 19.7 & 50.5 & 27.5 & 27.8 & 14.7 & 25.3 \\
319:CommonVoice   & 24.7 & -    & 14.2 & -    & 44.5 & 22.8 & 35.2 & 22.4 & -    & 29.2 & -    & 31.7 & 28.1 \\


Summary: auto


**Exploring dataset tables**

I'm planning to check the dataset table starting at around line 121. Then, I'll look at the Hindi comparison table around line 220 and analyze the performance of the models near line 270. It sounds like using `sed -n` with specific line numbers will help me pinpoint the information I need efficiently. Let's open it and see what insights we can gather!


cd /tmp/vistaar_src/extracted
nl -ba template.tex | sed -n '110,210p'

nl -ba template.tex | sed -n '210,270p'

nl -ba template.tex | sed -n '270,340p'


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 233
Output:
   110	The rest of the paper is organized as follows.
   111	% In Section~\ref{sec:evaluation}, we quantitatively discuss the challenges with evaluating ASR systems with single benchmarks.
   112	We detail all benchmark datasets in Vistaar in Section~\ref{sec:vistaar}.
   113	In Section~\ref{sec:vistaar-train} we detail the training dataset and the procedure used to train IndicWhisper.
   114	We compare various public and commercial ASR systems along with IndicWhisper on Vistaar in Section~\ref{sec:evaluation}.
   115	We end with conclusions and our thought-process on further improvements towards building open Indic ASR systems.
   116
   117	% \section{Evaluation of ASR systems on Individual Benchmarks}
   118	% \label{sec:evaluation}
   119	% In this section, we discuss the
   120
   121	\section{Vistaar Benchmark Set}
   122	\label{sec:vistaar}
   123
   124	\begin{table*}[ht!]
   125	\centering
   126	\small
   127	\caption{The Vistaar Benchmark Set - 59 benchmarks over 12 Indian languages }
   128	\label{tab:vistaar-evaluation}
   129	\begin{tabular}{lcccccccccccc}
   130	\toprule
   131	Datasets    & bn                   & gu                   & hi   & kn                   & ml                   & mr                   & or                   & pa                   & \multicolumn{1}{c}{sa}   & ta                   & te                   & ur                   \\
   132	\midrule
   133	Kathbath    & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{c}{\CheckedBox} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
   134	Kathbath-Hard    & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{c}{\CheckedBox} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
   135	FLEURS      & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 &                          & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 \\
   136	CommonVoice & \CheckedBox                 & \multicolumn{1}{l}{} & \CheckedBox & \multicolumn{1}{l}{} & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 &                          & \CheckedBox                 & \multicolumn{1}{l}{} & \CheckedBox                 \\
   137	IndicTTS    & \CheckedBox                 & \CheckedBox                 & \CheckedBox & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{l}{} &                          & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{l}{} \\
   138	MUCS        & \multicolumn{1}{l}{} & \CheckedBox                 & \CheckedBox & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{l}{} &                          & \CheckedBox                 & \CheckedBox                 & \multicolumn{1}{l}{} \\
   139	Gramvaani   & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \CheckedBox & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} &                          & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} & \multicolumn{1}{l}{} \\
   140	\bottomrule
   141	\end{tabular}
   142	\end{table*}
   143
   144	\begin{figure}[t!]
   145	    \centering
   146	    \includegraphics[width=\columnwidth]{plot1.png}
   147	    \caption{\textbf{WER Comparison on Vistaar:} IndicWhisper is better than IndicWav2Vec in 45 out of 59 benchmarks (highlighted by purple) and better than Google STT in 57 out of 59 benchmarks.}
   148	    \label{fig:vistaar-evaluation}
   149	\end{figure}
   150
   151	In the following we describe various publicly available datasets which we curate as part of the Vistaar benchmark set.
   152
   153	\noindent \textbf{Kathbath \cite{kathbath}} This contains \textit{read speech} collected on Android phones with the Karya application \cite{iitb_msr} by AI4Bharat, where the sentences are sourced from Wikipedia and news articles.
   154	This covers all the 12 languages we consider.
   155
   156	\noindent \textbf{Kathbath-Hard \cite{kathbath}} We create a hard benchmark for  by adding background noise of various types to the audio files of the Kathbath benchmark.
   157	We use ESC dataset\cite{esc}, which consists of 2,000 short clips of background noise from 5 differen…8680 chars truncated…ulticolumn{14}{l}{\textbf{Google STT}} \\
   280	\midrule
   281	Kathbath      & 20.6 & 26.0 & 14.3 & 27.9 & 52.6 & 26.6 & 105.7 & 25.5 & 104.2 & 31.1 & 34.3 & 17.6 & 40.5 \\
   282	Kathbath Hard & 22.4 & 28.4 & 16.6 & 31.4 & 53.6 & 28.4 & 105.4 & 29.2 & 104.4 & 32.8 & 35.9 & 20.2 & 42.4 \\
   283	CommonVoice   & 26.1 & -    & 20.8 & -    & 54.0 & 27.5 & 101.7 & 29.2 & -     & 42.1 & -    & 26.9 & 41.0 \\
   284	FLEURS        & 27.3 & 34.7 & 19.4 & 33.8 & 41.0 & 33.2 & 100.8 & 44.0 & -     & 36.6 & 40.1 & 28.4 & 39.9 \\
   285	IndicTTS      & 37.0 & 33.6 & 18.3 & 32.8 & 38.3 & 21.8 & 104.7 & -    & -     & 31.3 & 55.4 & -    & 41.5 \\
   286	MUCS          & -    & 59.7 & 17.8 & -    & -    & 21.4 & 103.5 & -    & -     & 27.8 & 46.4 & -    & 46.1 \\
   287	Gramvaani     & -    & -    & 59.9 & -    & -    & -    & -     & -    & -     & -    & -    & -    & 59.9 \\
   288	\midrule
   289	Average       & 26.7 & 36.5 & 23.9 & 31.5 & 47.9 & 26.5 & 103.6 & 32.0 & 104.3 & 33.6 & 42.4 & 23.3 & 44.3 \\
   290	\midrule
   291	\multicolumn{14}{l}{\textbf{Azure STT}} \\
   292	\midrule
   293	Kathbath      & 16.7 & 22.4 & 13.6 & 23.6 & 43.3 & 20.9 & NA & NA & NA & 28.1 & 22.8 & NA & - \\
   294	Kathbath Hard & 18.4 & 26.7 & 14.9 & 25.9 & 45.7 &      & NA & NA & NA & 30.0 & 24.6 & NA & - \\
   295	CommonVoice   & 15.0 & -    & 14.6 & -    & 47.5 & 19.6 & NA & NA & NA & 39.9 & -    & NA & - \\
   296	FLEURS        & 29.9 & 34.8 & 24.3 & 31.0 & 39.2 & 37.2 & NA & NA & NA & 34.9 & 32.3 & NA & - \\
   297	IndicTTS      & 21.2 & 50.5 & 15.2 & 25.3 & 33.5 & 31.6 & NA & NA & NA & 31.2 & 61.1 & NA & - \\
   298	MUCS          & -    & 23.7 & 15.1 & -    & -    & 27.5 & NA & NA & NA & 25.1 & 16.1 & NA & - \\
   299	Gramvaani     & -    & -    & 42.3 & -    & -    & -    & NA & NA & NA & -    & -    & NA & - \\
   300	\midrule
   301	Average       & 20.2 & 31.6 & 20.0 & 26.4 & 41.8 & 27.4 & NA & NA & NA & 31.5 & 31.4 & NA & - \\
   302	\midrule
   303	\multicolumn{14}{l}{\textbf{IndicWav2Vec}} \\
   304	\midrule
   305	Kathbath      & 14.3 & 17.4 & 12.2 & 20.8 & 37.8 & 20.4 & 22.2 & 15.1 & 37.0 & 27.4 & 26.9 & 16.0 & 22.3 \\
   306	Kathbath Hard & 18.1 & 21.5 & 16.2 & 26.4 & 42.8 & 24.0 & 28.3 & 18.2 & 43.3 & 33.0 & 31.2 & 19.4 & 26.9 \\
   307	FLEURS        & 28.4 & 22.7 & 18.3 & 22.0 & 40.4 & 25.1 & 40.5 & 34.8 & -    & 37.7 & 30.4 & 32.2 & 30.2 \\
   308	CommonVoice   & 31.3 & -    & 20.2 & -    & 67.0 & 27.0 & 37.9 & 19.7 & -    & 42.7 & -    & 32.7 & 34.8 \\
   309	IndicTTS      & 28.1 & 22.7 & 15.0 & 20.1 & 40.5 & 18.6 & 24.4 & -    & -    & 28.0 & 32.3 & -    & 25.5 \\
   310	MUCS          & -    & 32.6 & 22.9 & -    & -    & 34.7 & 30.1 & -    & -    & 37.7 & 30.4 & -    & 31.4 \\
   311	Gramvaani     & -    & -    & 42.1 & -    & -    & -    & -    & -    & -    & -    & -    & -    & 42.1 \\
   312	\midrule
   313	Average       & 24.0 & 23.4 & 21.0 & 22.3 & 45.7 & 25.0 & 30.6 & 21.9 & 40.2 & 34.4 & 30.2 & 25.1 & 28.7 \\
   314	\midrule
   315	\multicolumn{14}{l}{\textbf{IndicWhisper}} \\
   316	\midrule
   317	Kathbath      & 16.6 & 17.8 & 10.1 & 19.3 & 34.8 & 19.9 & 24.7 & 16.9 & 45.6 & 24.2 & 25.0 & 11.9 & 22.2 \\
   318	Kathbath Hard & 19.4 & 20.6 & 11.9 & 22.2 & 38.4 & 22.1 & 29.1 & 19.7 & 50.5 & 27.5 & 27.8 & 14.7 & 25.3 \\
   319	CommonVoice   & 24.7 & -    & 14.2 & -    & 44.5 & 22.8 & 35.2 & 22.4 & -    & 29.2 & -    & 31.7 & 28.1 \\
   320	FLEURS        & 20.9 & 23.5 & 10.5 & 18.6 & 22.6 & 20.5 & 32.9 & 23.1 & -    & 25.2 & 25.4 & 19.2 & 22.0 \\
   321	IndicTTS      & 18.8 & 19.1 & 8.1  & 13.2 & 21.4 & 11.4 & 15.0 & -    & -    & 17.2 & 33.8 & -    & 17.6 \\
   322	MUCS          & -    & 33.2 & 17.1 & -    & -    & 12.8 & 27.5 & -    & -    & 28.3 & 32.1 & -    & 25.2 \\
   323	Gramvaani     & -    & -    & 24.9 & -    & -    & -    & -    & -    & -    & -    & -    & -    & 24.9 \\
   324	\midrule
   325	Average       & 20.1 & 22.8 & 13.8 & 18.3 & 32.3 & 18.2 & 27.4 & 20.5 & 48.0 & 25.3 & 28.8 & 19.4 & 24.6 \\
   326	\bottomrule
   327	\end{tabular}
   328	\end{table*}
   329
   330	%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
   331
   332	In this section, we detail the Vistaar-Train dataset, and then architecture and training methodology to train the IndicWhisper family of models.
   333
   334	\subsection{Vistaar-Train Set}
   335	As discussed in the previous sections, results in correlating the performance of a set of ASR models on a diverse set of benchmarks.
   336	Based on this, we hypothesize that ASR models must also be trained on a diverse set of training data.
   337	To this end, we first put together Vistaar-Train set which curates 13 publicly released datasets.
   338	In the following, we detail those datasets which are included in Vistaar-Train and not already discussed in the earlier section.
   339
   340	\noindent \textbf{Shrutilipi \cite{shrutilipi}} This contains \textit{read speech} collected by mining audio and text pairs from news bulletins aired on All India Radio.


**Analyzing benchmarks for models**

I think I can derive average WER per language from the table, but maybe it's unnecessary. The user specifically asked for benchmarks and WER, so I should focus on providing that. I'll take a closer look at the "Comparison of publicly available models on the Hindi subset of the Vistaar benchmark" around line 221. It seems important to inspect those specific lines for relevant details to share with the user.


cd /tmp/vistaar_src/extracted
nl -ba template.tex | sed -n '216,260p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   216	\end{table*}
   217
   218	\begin{table*}[]
   219	\centering
   220	\small
   221	\caption{Comparison of publicly-available models on the Hindi subset of the Vistaar benchmark}
   222	\label{tab:vistaar_hindi}
   223	\begingroup
   224	% \setlength{\tabcolsep}{2.5pt} % Default value: 6pt
   225	% \renewcommand{\arraystretch}{0.7} % Default value: 1
   226	\begin{tabular}{lcccccccc}
   227	\toprule
   228	Model         & Kathbath   & Kathbath-Hard & FLEURS & CommonVoice & IndicTTS & MUCS & Gramvaani & Average  \\
   229	\midrule
   230	Google STT        & 14.3 & 16.7    & 19.4   & 20.8 & 18.3     & 17.8 & 59.9      & 23.9 \\
   231	IndicWav2vec  & 12.2 & 16.2    & 18.3   & 20.2 & 15.0     & 22.9 & 42.1      & 21.0 \\
   232	Azure STT        & 13.6 & 15.1    & 24.3   & 14.6 & 15.2     & 15.1 & 42.3      & 20.0 \\
   233	Nvidia-medium & 14.0 & 15.6    & 19.4   & 20.4 & 12.3     & 12.4 & 41.3      & 19.4 \\
   234	Nvidia-large  & 12.7 & 14.2    & 15.7   & 21.2 & 12.2     & \textbf{11.8} & 42.6      & 18.6 \\
   235	IndicWhisper  & \textbf{10.3} & \textbf{12.0}    & \textbf{11.4}   & \textbf{15.0} & \textbf{7.6 }     & 12.0 & \textbf{26.8 }     & \textbf{13.6} \\
   236	% \midrule
   237
   238	\bottomrule
   239	% \multicolumn{9}{l}{(KB: Kathbath, FL: FLEURS, CV: CommonVoice, I: Indic, GV: Gramvaani)} \\
   240	\end{tabular}
   241	\endgroup
   242	\end{table*}
   243
   244
   245
   246
   247	% \begin{table*}[h!]
   248	% \centering
   249	% % \small
   250	% \caption{Word error rates (\%) of IndicWhisper on the Vistaar Benchmark. Values in bold indicates benchmarks where IndicWhisper has the lowest WER.}
   251	% \label{tab:indicwhisper}
   252	% \begin{tabular}{lcccccccccccc|c}
   253	% \toprule
   254	% Datasets      & bn   & gu   & hi   & kn   & ml   & mr   & or   & pa   & sa   & ta   & te   & ur   & avg  \\
   255	% \midrule
   256	% Kathbath      & 16.6 & 17.8 & \textbf{10.3} & \textbf{19.3} & \textbf{34.8} & \textbf{19.9} & 24.7 & 16.9 & 45.6 & \textbf{24.2} & 25.0 & \textbf{11.9} & 22.3 \\
   257	% Kathbath Hard & 19.4 & \textbf{20.6} & \textbf{12.0} & \textbf{22.2} & \textbf{38.4} & \textbf{22.1} & 29.1 & 19.7 & 50.5 & \textbf{27.5} & 27.8 & \textbf{14.7} & 25.3 \\
   258	% CommonVoice   & 24.7 & -    & \textbf{11.4} & -    & \textbf{44.5} & 22.8 & \textbf{35.2} & 22.4 & -    & \textbf{29.2} & -    & 31.7 & 27.8 \\
   259	% FLEURS        & \textbf{20.9} & 23.5 & \textbf{15.0} & \textbf{18.6} & \textbf{22.6} & \textbf{20.5} & \textbf{32.9} & \textbf{23.1} & -    & \textbf{25.2} & \textbf{25.4} & \textbf{19.2} & 22.5 \\
   260	% IndicTTS      & \textbf{18.8} & \textbf{19.1} & \textbf{7.6}  & \textbf{13.2} & \textbf{21.4} & \textbf{11.4} & \textbf{15.0} & -    & -    & \textbf{17.2} & 33.8 & -    & 17.5 \\


**Checking dataset accessibility**

I’m looking into whether these datasets are accessible in our pipeline environment at HuggingFace. Some might be gated, but the user specifically asked for open-source data to perform benchmarks.

I could propose several options: Kathbath from AI4Bharat, which includes speech from multiple languages except Assamese and English, FLEURS from Google that covers Assamese and English, and Common Voice from Mozilla, which possibly includes Assamese.

Other options could be the Svarah dataset for accents and the IndicSUPERB dataset. For code-switching, the MUCS dataset might be helpful, although it may not be on HuggingFace.


**Proposing benchmarking subsets**

I’m thinking about how to pick a manageable subset for benchmarking. For each language, we could choose the FLEURS test split because it’s small and uniform. For languages with more resources, Common Voice's test or validation splits might work too, along with Kathbath for Indian-specific data.

For code-switching, I’d suggest using MUCS for Hindi-English, and if that's not available, we could create a dataset from existing data. However, I found an important issue: CTC scoring with MMS models needs the target_lang to be set, but it seems this hasn’t been addressed yet.


**Reviewing benchmarking steps**

I noticed that the user mentioned "Stable segment_id" and "gemini prompt language hint," but they didn’t bring up target_lang, which I think is worth highlighting as a review point. It’s good that using `to_prompt_language_hint` for gemini returns specific strings like "Hindi (Devanagari script)."

So, for Step 1, I can say the code is mostly robust for benchmarking on open-source data but remember the CTC target_lang requirement. We should check if segment_id changes need new references and ensure the pipeline can handle audio sample rates by normalizing to 16k.

For Step 2, I’ll suggest using the `datasets` library to download datasets, create a segments directory, set up refs.jsonl, and then run the pipeline to evaluate and analyze results.


**Suggesting cross-language analysis**

I think we should propose a cross-language pattern analysis. For each language, we could compute metrics like mean WER, p90, p95, and CER. It’s also important to look at the oracle best WER and the selection gap, along with the distribution of accept vs. human vs. reject decisions and their reasons. This could help fine-tune thresholds for each language.

We might want to analyze the CTC best loss distribution by language and token coverage distribution as well. Measuring LID accuracy would also be key, comparing the pipeline's route_lang to the known dataset language and generating a confusion matrix.

For Step 3, I’ll provide setup instructions, including `pip install -e '.[full]' datasets soundfile`, and guide the user to choose the dataset and language. I’ll include a Python snippet for downloading the dataset, writing audio files, and creating refs.jsonl using `maya_transcribe.ids.segment_identity` for consistent segment_ids.


**Providing dataset configuration mapping**

I’m working on a Python snippet to load the FLEURS dataset. I’ve set up a segments directory and started to write audio files. However, I need to confirm the FLEURS configuration names. It looks like they might use ISO codes, like "hi_in" for Hindi and "te_in" for Telugu.

To get accurate config names, I should query the dataset config list using the Hugging Face API or the `datasets` library. I can get the list for google/fleurs to find out the correct codes for all languages.

For example, I have a mapping list in mind:

- Hindi: `hi_in`
- Telugu: `te_in`
- Tamil: `ta_in`
- Kannada: `kn_in`
- Malayalam: `ml_in`
- Gujarati: `gu_in`
- Punjabi: `pa_in`
- Bengali: `bn_in`
- Odia: `or_in`
- Assamese: `as_in`
- Marathi: `mr_in`
- English: I’m considering using `en_us`, but questioning if there is a specific code for Indian English.

I’ll check the Hugging Face API for more details, as it could save me from mixing up config names.


python3 - <<'PY'
import requests
j=requests.get('https://huggingface.co/api/datasets/google/fleurs',timeout=20).json()
configs=(j.get('cardData') or {}).get('configs') or []
print('configs',len(configs))
# print few that end with _in
names=[c.get('config_name') for c in configs if c.get('config_name','').endswith('_in')]
print('configs ending _in',len(names))
print('sample',sorted(names)[:30])
# print specifically our languages
need=['as_in','bn_in','gu_in','hi_in','kn_in','ml_in','mr_in','or_in','pa_in','ta_in','te_in','en_in','en_us']
for n in need:
    print(n,'present',n in names or any(c.get('config_name')==n for c in configs))
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
configs 0
configs ending _in 0
sample []
as_in present False
bn_in present False
gu_in present False
hi_in present False
kn_in present False
ml_in present False
mr_in present False
or_in present False
pa_in present False
ta_in present False
te_in present False
en_in present False
en_us present False


**Evaluating benchmark accuracy**

The code looks solid for benchmarking, which is encouraging. The user asked me to search the internet for ideas, and I’ll need to incorporate Vistaar benchmarks and WER scores from their paper for baseline comparisons. The Vistaar paper provides WER for various models, and although our pipeline isn't identical to IndicWhisper, we can still evaluate relative improvements by comparing it to base Whisper large v3. However, I need to explain that we can’t produce WER without running the tests ourselves.


**Proposing benchmark evaluations**

I can suggest running benchmarks on open-source data, focusing on language performance. For Vistaar benchmark subsets, I’ll include Kathbath, FLEURS, and CommonVoice, which align well with our language segments. For Assamese, since it's not in Vistaar, we can use FLEURS and CommonVoice. I need to verify if the pipeline's evaluation normalizes based on language codes and that normalization isn't overly aggressive. In my final answer, I'll address code robustness, highlight issues, and provide dataset ideas along with WER numbers from the Vistaar paper.


**Clarifying benchmarking steps**

I need to provide disclaimers that our pipeline will need to run for accurate results. Additionally, I should outline step-by-step instructions for the other agent on how to run benchmarks and analyze patterns, including command lines and a minimal Python snippet. They also asked me to search online for usable benchmarks. I'll mention the Vistaar benchmark suite from AI4Bharat, which includes 59 benchmarks across various languages and datasets, providing baseline WER data for context. I should be cautious while quoting these WER% numbers from the Vistaar paper.


**Interpreting average values**

I’m checking the "Average" row in the table, and it seems like it's indeed an overall average across the datasets and languages. The table structure suggests that there are various language columns followed by an average column. The "Average" row for IndicWhisper shows values such as 24.6, which likely represents the average across all included benchmarks. That aligns with the idea that this average is derived from 59 benchmark-language combinations, indicating reductions in error rates effectively.


I'm sorry, but I can't assist with that.


**Summarizing Vistaar resources**

I have the dataset list and baseline WER from the Vistaar paper, which is great for summarizing. I also want to check the Vistaar GitHub page, as it might have code for downloading datasets or even a list of links. My next step is to open the repository at `https://github.com/AI4Bharat/vistaar` and fetch the necessary information using requests. This will help me gather everything I need for a solid overview!


python3 - <<'PY'
import requests, re
url='https://raw.githubusercontent.com/AI4Bharat/vistaar/main/README.md'
r=requests.get(url,timeout=20)
print('status',r.status_code,'len',len(r.text))
text=r.text
print(text[:600])
# list dataset names quickly
for kw in ['Kathbath','FLEURS','CommonVoice','IndicTTS','MUCS','Gramvaani','Vistaar']:
    if kw.lower() in text.lower():
        print('mentions',kw)
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 404 len 14
404: Not Found


Summary: auto


**Checking GitHub repo accessibility**

I'm thinking the GitHub repo path might not exist or the branch could be different. It's possible that `AI4Bharat/vistaar` exists but isn’t accessible or there's a case sensitivity issue. I’ll check this by using the GitHub API. I could also try to open the page directly. Maybe I should fetch `https://github.com/AI4Bharat/vistaar`, but the branch name might differ. I might also use `curl -I` to check the headers.


python3 - <<'PY'
import requests
url='https://github.com/AI4Bharat/vistaar'
r=requests.get(url,timeout=20)
print('status',r.status_code)
print(r.url)
print(r.text[:200])
PY


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
status 200
https://github.com/AI4Bharat/vistaar

<!DOCTYPE html>
<html
  lang="en"

  data-color-mode="auto" data-light-theme="light" data-dark-theme="dark"
  data-a11y-animated-images="system" data-a11y-link-underlines="true"

  >


**Checking repository branch**

I see that the repository exists, and I'm considering if the default branch might be `master` instead of `main`. I plan to check this by parsing the HTML for `default_branch` since it might be dynamic. Another option is to use the GitHub API. The API endpoint is accessible without a token, so I can call it directly to find out more about the repository details. Let's get started!


python3 - <<'PY'
import requests
j=requests.get('https://api.github.com/repos/AI4Bharat/vistaar',timeout=20).json()
print('default_branch', j.get('default_branch'))
print('description', j.get('description'))
PY


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
default_branch master
description Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR


python3 - <<'PY'
import requests
url='https://raw.githubusercontent.com/AI4Bharat/vistaar/master/README.md'
r=requests.get(url,timeout=20)
print('status',r.status_code,'len',len(r.text))
print(r.text[:800])
PY


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
status 200 len 12719
# Vistaar: Diverse Benchmarks and Training sets for Indian Language ASR

 Vistaar is a set of 59 benchmarks and training datasets across various language and domain combinations such as news, education, literature, tourism etc. The training datasets are avaialable for 12 Indian languages amounting to over 10,700 hours of labelled audio data. We also train IndicWhisper models by fine-tuning the Whisper models on the Vistaar train dataset and observe that it has the lowest WER on 39 out of 59 Vistaar benchmarks.

## Benchmarks
Vistaar consists of benchmarks from several public datasets - Kathbath, FLEURS, CommonVoice, IndicTTS, MUCS, GramVaani across 12 languages. We evaluate IndicWhisper on these benchmarks along with 3 publicly available ASR systems and 2 commercially available systems. B


**Parsing README for datasets**

I’m planning to parse the README to gather a list of benchmarks and links to datasets. I’ll extract relevant sections that provide details about the datasets, including any HF dataset IDs. It seems like a good idea to search within the README for terms like "Kathbath" to locate specific information. I’ll also look for instructions on how to download and compute WER. Alright, time to get started on this!


python3 - <<'PY'
import requests, re
text=requests.get('https://raw.githubusercontent.com/AI4Bharat/vistaar/master/README.md',timeout=20).text
# Print headings
for line in text.splitlines():
    if line.startswith('##'):
        print(line)

# show part about benchmarks table maybe
idx=text.lower().find('## benchmarks')
print('\n--- Benchmarks section snippet ---')
print('\n'.join(text.splitlines()[0:80]))
PY