Output:
Exit code: 0
Wall time: 174.6 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 23:29:57.483250: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
`torch_dtype` is deprecated! Use `dtype` instead!
`generation_config` default values have been modified to match model-specific defaults: {'suppress_tokens': [], 'begin_suppress_tokens': [220, 50257]}. If this is not desired, please set these values explicitly.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> to see related `.generate()` flags.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> to see related `.generate()` flags.
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:02:41
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_scriptfilter2/results.jsonl
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_scriptfilter2/summary.json
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_scriptfilter2/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 20,
  "wer_mean": 0.9744574175824177,
  "cer_mean": 0.7177615882678585,
  "wer_p50": 1.0,
  "wer_p90": 1.0,
  "wer_p95": 1.0,
  "cer_p50": 0.8051948051948052,
  "cer_p90": 0.9253731343283582,
  "cer_p95": 0.9264705882352942,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 1.0,
  "wer_by_language_mean": {
    "asm": 0.9744574175824177
  },
  "wer_by_decision_mean": {
    "human": 0.973113071139387,
    "reject": 1.0
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.9888888888888889,
    "10-30s": 0.913919413919414,
    "3-10s": 0.9657738095238096
  },
  "cer_by_language_mean": {
    "asm": 0.7177615882678585
  },
  "cer_by_decision_mean": {
    "human": 0.70683466479099,
    "reject": 0.9253731343283582
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.8241385527709416,
    "10-30s": 0.4850677339901478,
    "3-10s": 0.5825722773542622
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.7959979979171156,
    "selection_gap_wer_mean": 0.17845941966530204,
    "selection_gap_wer_p90": 0.46153846153846156
  }
}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Considering sampling weights**

I’m brainstorming how to set weights for the faster-whisper sampling. One idea is to downweight it by treating outputs from the same backend as less significant. For instance, I could use weights like faster-whisper deterministic at 1, sampling at 0.5, and transformer router at 1.5.

Another thought is to adjust the rover to choose the centroid by considering model diversity but maybe simplifying it is better. If we have a transformer router output, I can treat faster-whisper sampling as auxiliary and not include it in the rover’s calculations. That way, I'd have two clear hypotheses: one from faster-whisper and the other from the transformer, which I’ll use to choose the centroid based on minimal distance.


**Analyzing centroid selection**

I’m thinking about how to choose the centroid by computing the average distance. The function will pick the minimum based on the average distance, but if there’s a tie, it defaults to the first in the insertion order. In the rover_combine process, the tokenized dictionary comes from hypotheses, and since Python preserves insertion order, I can rely on that.

If I drop sampling and include only the transformer, the order will be deterministic first, followed by the transformer. Consequently, while selecting the centroid, the deterministic output could dominate, which might not be the best compared to the transformer.


**Choosing transformer as priority**

I'm considering making the transformer the primary choice for our skeleton. One way is to reorder the hypotheses or adjust the weights to favor the transformer. However, a simpler solution could be passing the hypotheses dictionary with the transformer first, ensuring that in case of a tie, the centroid selector chooses the transformer. This approach would help produce a final output that aligns more closely with the transformer. I could implement this by reordering the rover_hyps when the transformer router output exists.


**Evaluating transformer models**

I'm considering using transformers as the base model, which could enhance performance. While the oracle suggests transformers excel for Assamese, it might not apply to all languages. The plan could involve leveraging the transformers_router with specific language routes for better primary output. Users likely expect fine-tuned models to provide the best results. I'll set weights with the transformer router at 2 and others accordingly, but I'm realizing that weights in current configurations mainly focus on voting, not centroid selection. Adjusting that might be more complex.


**Adjusting centroid selection**

I'm exploring an alternative approach by providing a weights dictionary and adjusting the centroid selection to use a weighted average distance. That could be beneficial, but altering the algorithm too much might be a concern. A simpler option could be to configure the `rover_hyps` dictionary with the router key first, allowing ties to be resolved in the router's favor. However, distances for key comparisons might be symmetric, so I need to be conscious of how average distances impact centroid choices and consider ways to lower the influence of other sampling methods.


**Streamlining the model selection**

I'm considering dropping the sampling hypothesis when the transformer router is in play. By doing this, I can narrow it down to two keys, which could lead to tiebreaking in the centroid selection, favoring the router if it's prioritized. I think implementing this approach is a solid way forward. I might set `include_sampling` to false by default during bench runs, though I have to keep an eye on how it affects overall ROVER performance. Adjusting dynamic conditions could help manage this effectively.


**Defining router behavior**

I'm thinking that the router will serve as the skeleton and will vote at each token position with equal weights. If there are ties in votes, the skeleton's token will be chosen. Therefore, the output will essentially reflect the router, with gaps filled by det matches. So, if a transformer router output is available, I should prioritize it in the rover combine process by reordering the hypothesis dictionary to place the router key first, while potentially incorporating other transformer models for consensus. I’ll need to make sure to identify and store the router key during the routing call.


**Implementing ASR results handling**

I’m considering dropping the sampling output when building ASR results, especially if a transformer router is present. I could add a threshold called `rover_drop_sampling_when_router`, set to true by default for bench runs. This makes sense since sampling offers little diversity when we already have different models. I’ll implement checks to remove sampling keys from the hypothesis list if the router key is detected, and reorder them to place the router first. I’ll need to ensure this handles ties effectively and adjust the combination logic accordingly.


**Implementing ASR result filtering**

I’m planning to refine how we handle ASR results by storing keys for sampling and determining if there are other hypotheses besides faster-whisper. When building `rover_hyps`, I want to remove sampling keys if they're not included after script filtering. I’ll check for diverse backends; if there’s more than one, I’ll set `include_sampling` to false to avoid redundancy.

For configurations, if a transformer router key is present, I’ll reorder the hypotheses to prioritize the router. This helps ensure accurate output tailored for different languages. However, we can consider optional preferences for better WER overall. We might also need a scoring system for selection or even modify centroid selection to give weights to different hypotheses.


**Exploring router preference in ASR**

I'm thinking we could simplify our pipeline by looking at conditions where the router exists and thresholds for `rover_skeleton` are set to "router." We can use `rover_combine` and possibly duplicate the router in the dictionary, which makes it more likely to win the centroid selection due to minimal distance to itself.

So, by duplicating the router, we not only minimize average distances but also affect voting outcomes in favor of the router. This method is a bit hacky since it doesn’t require changing the rover module. Alternatively, we might set the final output directly to the router's normalized output if `prefer_router` is true, while ensuring we're still able to compute results through rover for robustness.


**Testing language models**

I'm considering testing one more language, maybe Hindi. I think using base large-v3 could be useful, but the fine-tuned ARTPARK model might perform even better. We'll start by running a small sample of Hindi segments and measuring WER. Right now, we need a strategy: when we have a transformer router output for a language, I’ll prioritize that for selection. I’ll bias the rover outputs toward the router to ensure it dominates, hopefully streamlining the process!


**Considering hypothesis management**

I realize that using the centroid router will give it two votes compared to one for the det model. However, I might also want to retain other hypotheses, perhaps from extra transformer models. To simplify things, I could set `rover_hyps` to include the `router_key` and `primary_asr_key` from `norm_hyps`. But I need to be careful with duplicates, as they might complicate evaluation since candidate keys are involved. Yet, since these duplicates are temporary for the rover combination, I'm thinking it’s okay to keep a record of that!


**Evaluating model organization**

I’m considering whether to include duplicates at all. Instead, I can build the `rover_hyps` as an ordered list, prioritizing `router_key` first, then `det`. The centroid can help determine which router to choose. If there are only two hypotheses, the distance is the same, so the first one is selected. For more than two, duplicates would be beneficial to help with differences in distances. Right now, I’m focusing on faster-whisper, with det and router used after dropping sampling. So, ordering seems sufficient for now.


**Implementing router logic**

Okay, I’m looking to implement a check. If `router_key` is in `rover_hyps` and the threshold for preferring the router is enabled, I’ll create an ordered dictionary, placing the `router_key` first. Since `rover_hyps` is a normal dictionary, insertion order is crucial.

I’m setting up logic to ensure that when there are two hypotheses, the router wins due to the way distances are computed. However, if there are more than two hypotheses, the centroid might not pick the router as the first choice, but we might still accept it.


**Patching pipeline logic**

I’m considering adding duplicate routers for weighting, but first, I’ll focus on ordering. I also need to drop sampling for cases with other backends, so I’ll implement that in the rover combination step.

I have to modify the pipeline where `rover_hyps` are created by removing sampling and setting router preferences. I need to establish the `router_key` variable by setting it to `None` at the start and updating it when a route is determined. I’ll also track sampling keys, using a detection method based on specific suffixes.


**Utilizing ensemble backends**

I see that we’ve already computed `ensemble_backends` after the ASR stage, and it’s defined before normalization. So, I can use this variable in the rover step, making sure it exists when needed.

I’ll confirm it’s defined correctly around line 243 in the pipeline file. If `include_sampling` is not set, I can default it to check if the length of `ensemble_backends` is less than 2. If sampling isn't included, I can filter `rover_hyps` to remove those keys ending with "|samp_t0.2". Maybe I can also consider appending any removed samples into a `filtered_out` list.


**Recording dropped sampling**

I think it would be useful to record that sampling was dropped, perhaps in `rover_filtered_out`. I’d add a list called `dropped_sampling` to track this.

If sampling is excluded, I’ll loop through `rover_hyps` and check for keys ending with "|samp_t0.2", removing them from `rover_hyps` and adding them to `dropped_sampling`. But it's important to note that these aren't mismatches, just filtered samples. Additionally, I’ll ensure that any modifications to `rover_hyps` account for the `router_key` and, if necessary, reorder. Lastly, if `primary_asr_key` is absent from `rover_hyps`, I'll update it accordingly.


**Implementing dynamic defaults**

I’m thinking about whether to set the threshold for the bench run to True by default. It seems like a good idea to check if `router_key` exists and then set `prefer_router` based on whether `thresholds.get("rover_prefer_router")` is set. If it isn't, I can assign it based on the presence of `router_key`.

I'll clarify the `include_sampling` option similarly and implement these checks to handle defaults dynamically. It feels straightforward!


**Considering sampling weights**

I’m brainstorming how to set weights for the faster-whisper sampling. One idea is to downweight it by treating outputs from the same backend as less significant. For instance, I could use weights like faster-whisper deterministic at 1, sampling at 0.5, and transformer router at 1.5.

Another thought is to adjust the rover to choose the centroid by considering model diversity but maybe simplifying it is better. If we have a transformer router output, I can treat faster-whisper sampling as auxiliary and not include it in the rover’s calculations. That way, I'd have two clear hypotheses: one from faster-whisper and the other from the transformer, which I’ll use to choose the centroid based on minimal distance.

**Analyzing centroid selection**

I’m thinking about how to choose the centroid by computing the average distance. The function will pick the minimum based on the average distance, but if there’s a tie, it defaults to the first in the insertion order. In the rover_combine process, the tokenized dictionary comes from hypotheses, and since Python preserves insertion order, I can rely on that.

If I drop sampling and include only the transformer, the order will be deterministic first, followed by the transformer. Consequently, while selecting the centroid, the deterministic output could dominate, which might not be the best compared to the transformer.

**Choosing transformer as priority**

I'm considering making the transformer the primary choice for our skeleton. One way is to reorder the hypotheses or adjust the weights to favor the transformer. However, a simpler solution could be passing the hypotheses dictionary with the transformer first, ensuring that in case of a tie, the centroid selector chooses the transformer. This approach would help produce a final output that aligns more closely with the transformer. I could implement this by reordering the rover_hyps when the transformer router output exists.

**Evaluating transformer models**

I'm considering using transformers as the base model, which could enhance performance. While the oracle suggests transformers excel for Assamese, it might not apply to all languages. The plan could involve leveraging the transformers_router with specific language routes for better primary output. Users likely expect fine-tuned models to provide the best results. I'll set weights with the transformer router at 2 and others accordingly, but I'm realizing that weights in current configurations mainly focus on voting, not centroid selection. Adjusting that might be more complex.

**Adjusting centroid selection**

I'm exploring an alternative approach by providing a weights dictionary and adjusting the centroid selection to use a weighted average distance. That could be beneficial, but altering the algorithm too much might be a concern. A simpler option could be to configure the `rover_hyps` dictionary with the router key first, allowing ties to be resolved in the router's favor. However, distances for key comparisons might be symmetric, so I need to be conscious of how average distances impact centroid choices and consider ways to lower the influence of other sampling methods.

**Streamlining the model selection**

I'm considering dropping the sampling hypothesis when the transformer router is in play. By doing this, I can narrow it down to two keys, which could lead to tiebreaking in the centroid selection, favoring the router if it's prioritized. I think implementing this approach is a solid way forward. I might set `include_sampling` to false by default during bench runs, though I have to keep an eye on how it affects overall ROVER performance. Adjusting dynamic conditions could help manage this effectively.

**Defining router behavior**

I'm thinking that the router will serve as the skeleton and will vote at each token position with equal weights. If there are ties in votes, the skeleton's token will be chosen. Therefore, the output will essentially reflect the router, with gaps filled by det matches. So, if a transformer router output is available, I should prioritize it in the rover combine process by reordering the hypothesis dictionary to place the router key first, while potentially incorporating other transformer models for consensus. I’ll need to make sure to identify and store the router key during the routing call.

**Implementing ASR results handling**

I’m considering dropping the sampling output when building ASR results, especially if a transformer router is present. I could add a threshold called `rover_drop_sampling_when_router`, set to true by default for bench runs. This makes sense since sampling offers little diversity when we already have different models. I’ll implement checks to remove sampling keys from the hypothesis list if the router key is detected, and reorder them to place the router first. I’ll need to ensure this handles ties effectively and adjust the combination logic accordingly.

**Implementing ASR result filtering**

I’m planning to refine how we handle ASR results by storing keys for sampling and determining if there are other hypotheses besides faster-whisper. When building `rover_hyps`, I want to remove sampling keys if they're not included after script filtering. I’ll check for diverse backends; if there’s more than one, I’ll set `include_sampling` to false to avoid redundancy.

For configurations, if a transformer router key is present, I’ll reorder the hypotheses to prioritize the router. This helps ensure accurate output tailored for different languages. However, we can consider optional preferences for better WER overall. We might also need a scoring system for selection or even modify centroid selection to give weights to different hypotheses.

**Exploring router preference in ASR**

I'm thinking we could simplify our pipeline by looking at conditions where the router exists and thresholds for `rover_skeleton` are set to "router." We can use `rover_combine` and possibly duplicate the router in the dictionary, which makes it more likely to win the centroid selection due to minimal distance to itself.

So, by duplicating the router, we not only minimize average distances but also affect voting outcomes in favor of the router. This method is a bit hacky since it doesn’t require changing the rover module. Alternatively, we might set the final output directly to the router's normalized output if `prefer_router` is true, while ensuring we're still able to compute results through rover for robustness.

**Testing language models**

I'm considering testing one more language, maybe Hindi. I think using base large-v3 could be useful, but the fine-tuned ARTPARK model might perform even better. We'll start by running a small sample of Hindi segments and measuring WER. Right now, we need a strategy: when we have a transformer router output for a language, I’ll prioritize that for selection. I’ll bias the rover outputs toward the router to ensure it dominates, hopefully streamlining the process!

**Considering hypothesis management**

I realize that using the centroid router will give it two votes compared to one for the det model. However, I might also want to retain other hypotheses, perhaps from extra transformer models. To simplify things, I could set `rover_hyps` to include the `router_key` and `primary_asr_key` from `norm_hyps`. But I need to be careful with duplicates, as they might complicate evaluation since candidate keys are involved. Yet, since these duplicates are temporary for the rover combination, I'm thinking it’s okay to keep a record of that!

**Evaluating model organization**

I’m considering whether to include duplicates at all. Instead, I can build the `rover_hyps` as an ordered list, prioritizing `router_key` first, then `det`. The centroid can help determine which router to choose. If there are only two hypotheses, the distance is the same, so the first one is selected. For more than two, duplicates would be beneficial to help with differences in distances. Right now, I’m focusing on faster-whisper, with det and router used after dropping sampling. So, ordering seems sufficient for now.

**Implementing router logic**

Okay, I’m looking to implement a check. If `router_key` is in `rover_hyps` and the threshold for preferring the router is enabled, I’ll create an ordered dictionary, placing the `router_key` first. Since `rover_hyps` is a normal dictionary, insertion order is crucial.

I’m setting up logic to ensure that when there are two hypotheses, the router wins due to the way distances are computed. However, if there are more than two hypotheses, the centroid might not pick the router as the first choice, but we might still accept it.

**Patching pipeline logic**

I’m considering adding duplicate routers for weighting, but first, I’ll focus on ordering. I also need to drop sampling for cases with other backends, so I’ll implement that in the rover combination step.

I have to modify the pipeline where `rover_hyps` are created by removing sampling and setting router preferences. I need to establish the `router_key` variable by setting it to `None` at the start and updating it when a route is determined. I’ll also track sampling keys, using a detection method based on specific suffixes.

**Utilizing ensemble backends**

I see that we’ve already computed `ensemble_backends` after the ASR stage, and it’s defined before normalization. So, I can use this variable in the rover step, making sure it exists when needed.

I’ll confirm it’s defined correctly around line 243 in the pipeline file. If `include_sampling` is not set, I can default it to check if the length of `ensemble_backends` is less than 2. If sampling isn't included, I can filter `rover_hyps` to remove those keys ending with "|samp_t0.2". Maybe I can also consider appending any removed samples into a `filtered_out` list.

**Recording dropped sampling**

I think it would be useful to record that sampling was dropped, perhaps in `rover_filtered_out`. I’d add a list called `dropped_sampling` to track this.

If sampling is excluded, I’ll loop through `rover_hyps` and check for keys ending with "|samp_t0.2", removing them from `rover_hyps` and adding them to `dropped_sampling`. But it's important to note that these aren't mismatches, just filtered samples. Additionally, I’ll ensure that any modifications to `rover_hyps` account for the `router_key` and, if necessary, reorder. Lastly, if `primary_asr_key` is absent from `rover_hyps`, I'll update it accordingly.

**Implementing dynamic defaults**

I’m thinking about whether to set the threshold for the bench run to True by default. It seems like a good idea to check if `router_key` exists and then set `prefer_router` based on whether `thresholds.get("rover_prefer_router")` is set. If it isn't, I can assign it based on the presence of `router_key`.

I'll clarify the `include_sampling` option similarly and implement these checks to handle defaults dynamically. It feels straightforward!


Input:
*** Begin Patch
*** Update File: maya_transcribe/pipeline.py
@@
                 asr_results: dict[str, ASRResult] = {}
                 asr_errors: dict[str, str] = {}
                 primary_asr_key = None
                 det = None
+                router_key: str | None = None

                 # Whisper: deterministic + a small sampling diversity pass.
                 try:
                     det = whisper_backend.transcribe(
                         wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
@@
                 if transformers_router is not None:
                     try:
                         routed = transformers_router.transcribe(
                             wav_path,
                             language=route_lang,
                             num_beams=5,
                             max_new_tokens=256,
                         )
                         if routed is not None:
                             asr_results[routed.model] = routed
+                            router_key = routed.model
                     except Exception as e:
                         asr_errors[f"transformers_router:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Patching rover settings**

I'm working on patching the rover's combination block to prioritize the router and drop sampling. I’ll start by locating the rover_hyps setting and then focus on the script filtering. I need to carefully open the relevant code snippets before making the patch. This requires a bit of attention to detail, as it’s important to ensure everything functions smoothly after the changes!


rg -n "rover_hyps" -n maya_transcribe/pipeline.py | head -n 40


Output:
Exit code: 0
Wall time: 0 seconds
Output:
292:                rover_hyps = dict(norm_hyps)
309:                            rover_hyps = kept
313:                if primary_asr_key is not None and primary_asr_key not in rover_hyps:
314:                    primary_asr_key = sorted(rover_hyps.keys())[0] if rover_hyps else None
316:                rover = rover_combine(rover_hyps)
325:                    candidates = {"rover": rover.text, **rover_hyps}
395:                        "inputs": sorted(rover_hyps.keys()),


sed -n '280,330p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                    ensemble_warnings.append("low_model_diversity")

                # 4) Normalize hypotheses
                t_norm0 = time.perf_counter()
                norm_hyps: dict[str, str] = {}
                for k, res in asr_results.items():
                    lang_hint = res.language or route_lang
                    norm_hyps[k] = normalize_text(res.text, lang=lang_hint)
                t_norm1 = time.perf_counter()

                # 5) ROVER
                t_rover0 = time.perf_counter()
                rover_hyps = dict(norm_hyps)
                rover_filtered_out: list[str] = []
                if bool(thresholds.get("filter_hypotheses_by_script", False)) and route_lang:
                    from maya_transcribe.language import dominant_script, expected_script

                    expected = expected_script(route_lang)
                    if expected:
                        kept: dict[str, str] = {}
                        for k, res in asr_results.items():
                            script = dominant_script(res.text)
                            if script is None or script == expected:
                                kept[k] = norm_hyps.get(k, "")
                            else:
                                rover_filtered_out.append(k)

                        kept = {k: v for k, v in kept.items() if v.strip()}
                        if kept:
                            rover_hyps = kept
                        else:
                            rover_filtered_out = []

                if primary_asr_key is not None and primary_asr_key not in rover_hyps:
                    primary_asr_key = sorted(rover_hyps.keys())[0] if rover_hyps else None

                rover = rover_combine(rover_hyps)
                t_rover1 = time.perf_counter()

                # 5.5) Audio-grounded fit scoring (optional): choose best candidate by CTC loss.
                t_ctc0 = time.perf_counter()
                ctc_info = None
                final_text = rover.text
                final_source = "rover"
                if ctc_scorer is not None and rank_by_ctc is not None:
                    candidates = {"rover": rover.text, **rover_hyps}
                    ctc_info = rank_by_ctc(ctc_scorer, wav_path, candidates=candidates, lang=route_lang)
                    best_key = ctc_info.get("best_key") if isinstance(ctc_info, dict) else None
                    if best_key and best_key in candidates:
                        final_text = candidates[best_key]
                        final_source = "rover" if best_key == "rover" else f"asr:{best_key}"


**Modifying filtering logic**

I need to adjust the logic after script filtering and before the primary ASR key adjustment. First, I’ll determine if I should include sampling. If it's not included, I'll drop keys that end with "|samp_t0.2" from rover_hyps and add them to filtered_out. I’ll check for prefer_router from the thresholds; if it's missing, the default will be set to router_key not being None. If prefer_router exists and router_key is in rover_hyps, I’ll reorder those hyps to prioritize router_key while maintaining at least one hypothesis.


**Implementing filtering patch**

I’m working on a patch after the script filtering block. I’ll start by checking the include_sampling option from thresholds. If it’s missing, I’ll set include_sampling based on the number of ensemble_backends. If I don’t include sampling, I plan to drop keys ending with "|samp_t0.2" from rover_hyps and add those to rover_filtered_out. If dropping makes rover_hyps empty, I’ll revert to the original. Then, I’ll check for prefer_router and reorder if needed. I might also consider prioritizing extra transformer models instead of the router. Alright, let’s patch this!


Input:
*** Begin Patch
*** Update File: maya_transcribe/pipeline.py
@@
                 if bool(thresholds.get("filter_hypotheses_by_script", False)) and route_lang:
                     from maya_transcribe.language import dominant_script, expected_script

                     expected = expected_script(route_lang)
                     if expected:
@@
                         if kept:
                             rover_hyps = kept
                         else:
                             rover_filtered_out = []

+                include_sampling_opt = thresholds.get("rover_include_sampling")
+                if include_sampling_opt is None:
+                    include_sampling = len(ensemble_backends) < 2
+                else:
+                    include_sampling = bool(include_sampling_opt)
+
+                if not include_sampling:
+                    dropped = [k for k in rover_hyps.keys() if k.endswith("|samp_t0.2")]
+                    for k in dropped:
+                        rover_hyps.pop(k, None)
+                    rover_filtered_out.extend(dropped)
+
+                prefer_router_opt = thresholds.get("rover_prefer_router")
+                prefer_router = bool(prefer_router_opt) if prefer_router_opt is not None else (router_key is not None)
+                if prefer_router and router_key and router_key in rover_hyps:
+                    rover_hyps = {
+                        router_key: rover_hyps[router_key],
+                        **{k: v for k, v in rover_hyps.items() if k != router_key},
+                    }
+
                 if primary_asr_key is not None and primary_asr_key not in rover_hyps:
                     primary_asr_key = sorted(rover_hyps.keys())[0] if rover_hyps else None

                 rover = rover_combine(rover_hyps)
                 t_rover1 = time.perf_counter()
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Clarifying sampling processes**

I used `rover_hyps.keys()` in a list comprehension, which is fine but might create a view — though converting it to a list is okay. Now, after dropping sampling, I'm worried `rover_hyps` could end up empty. If we only had sampling and script filtering removed, that's a concern, but probably unlikely.

Now, I should update the benchmark run default thresholds to set `rover_prefer_router` since our pipeline prefers the router if `router_key` isn't None. Default sampling inclusion depends on the ensemble's length, which is sufficient here.


**Analyzing Assamese language processing**

For Assamese, `rover_hyps` will include both the faster-whisper det and transformer router. The `prefer_router` default is true since `router_key` is not None and is present in `rover_hyps`, which means the router takes priority. When there are two hypotheses, the centroid tie will favor the router's output.

This might yield a WER of 0.796 or potentially better. We need to consider other languages too. If the router performs better, it could reduce WER, which is our hypothesis. If not, we can configure thresholds to revert back to the centroid method. I should also revert the faster-whisper mapping from 'as' to 'bn'.


**Reviewing router preferences and mapping**

With `prefer_router`, the final output will rely on the transformer, so the mapping may not matter much right now. However, it could serve as an alternative candidate if the user sets `prefer_router` to false for certain languages. The mapping might improve accuracy for Bengali script, but I'm not completely sure. If we map 'as' to 'bn', it might provide a candidate in the expected script, which could be useful for CTC scoring or selection strategies.

Now, I plan to rerun the limit of 20 samples with the per-language map to see the final WER and run the benchmark in a new directory.


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/per_lang_vaani_limit20_routerpref --whisper-model large-v3 --transformers-whisper-model-map data/bench/model_maps/whisper_vaani_map.json --limit 20


Output:
Exit code: 0
Wall time: 174.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 23:36:35.516195: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
`torch_dtype` is deprecated! Use `dtype` instead!
`generation_config` default values have been modified to match model-specific defaults: {'suppress_tokens': [], 'begin_suppress_tokens': [220, 50257]}. If this is not desired, please set these values explicitly.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> to see related `.generate()` flags.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> to see related `.generate()` flags.
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:02:41
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_routerpref/results.jsonl
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_routerpref/summary.json
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_routerpref/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 20,
  "wer_mean": 0.8340232080011492,
  "cer_mean": 0.6475969721102806,
  "wer_p50": 0.9166666666666666,
  "wer_p90": 1.0,
  "wer_p95": 1.0,
  "cer_p50": 0.7692307692307693,
  "cer_p90": 0.8703703703703703,
  "cer_p95": 0.9,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 1.0,
  "wer_by_language_mean": {
    "asm": 0.8340232080011492
  },
  "wer_by_decision_mean": {
    "human": 0.8290469858658712,
    "reject": 0.9285714285714286
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.9593948412698413,
    "10-30s": 0.5073260073260073,
    "3-10s": 0.692179008355479
  },
  "cer_by_language_mean": {
    "asm": 0.6475969721102806
  },
  "cer_by_decision_mean": {
    "human": 0.6337627200532413,
    "reject": 0.9104477611940298
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.8003659361544649,
    "10-30s": 0.30939039408867,
    "3-10s": 0.45479457002911555
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.7959979979171156,
    "selection_gap_wer_mean": 0.03802521008403362,
    "selection_gap_wer_p90": 0.11764705882352944
  }
}


**Evaluating candidate selection**

I’m thinking about the current run where the best oracle value is 0.796. I wonder if that aligns with the router or rover candidates, but if the rover performs worse than the router, then the router should be selected. It makes sense to add a step to choose the final candidate without CTC, using a condition that checks if `router_key` exists and defaults to True for `final_prefer_router`. The user may want to see the effects as well. I’m also cautious because, for some languages, the router might not be as good as the base large-v3 model.


**Evaluating selection strategies**

I need to decide on a default setting for the "final_prefer_router." Maybe we can set it to False for general runs and True for benchmarks. The user wants minimal WER, which implies thorough testing. It might be better to choose per-language fine-tunes for better accuracy, but there's no guarantee. Setting a threshold for "final_prefer" and adjusting to prefer the router by default during benchmarks could help achieve that minimal WER while keeping it flexible for the user.


**Implementing selection logic**

I’m thinking about using `ctc` scoring for selection eventually, but I can start this implementation now. In the pipeline, right after rover combination and before `ctc` scoring, I can set `final_text` and `final_source` based on certain thresholds. The idea is to prioritize the `router_key` if it meets the preference threshold. Script filtering could be important because we might have dropped the router option due to script mismatches. So, we should ensure that the `router_key` is still included in the rover hypotheses after filtering.


**Deciding on scoring logic**

I need to consider how to implement the selection logic for `final_prefer_router`. If the `router_key` is present in `rover_hyps` and meets the preference threshold, I'll set `final_text` to the normalized value for that key. However, `rover_hyps` could have had keys removed, but the router_key should stay intact. I could allow CTC scoring to override the selection, treating the router as a starting point. For benchmarks, I'll set `final_prefer_router` to True and include it in the thresholds for `cli.py`, while keeping it False for general runs.


**Implementing language preferences**

I’m considering how to optimize the pipeline based on language performance. If the router output is better, WER improves, otherwise it might degrade. I could add a threshold for `final_prefer_transformers` models, but let’s keep it simple for now. In the CTC scoring section, I’ll set `final_text` and `final_source` right before CTC scoring. I’ll incorporate the router preference so that if the `router_key` is valid, it updates the values. I'll ensure the `router_key` is in scope during this block and check against the normalized hypotheses appropriately. Time to implement these changes!


Input:
*** Begin Patch
*** Update File: maya_transcribe/pipeline.py
@@
                 ctc_info = None
                 final_text = rover.text
                 final_source = "rover"
+                if router_key and router_key in rover_hyps and bool(thresholds.get("final_prefer_router", False)):
+                    final_text = rover_hyps[router_key]
+                    final_source = f"asr:{router_key}"
                 if ctc_scorer is not None and rank_by_ctc is not None:
                     candidates = {"rover": rover.text, **rover_hyps}
                     ctc_info = rank_by_ctc(ctc_scorer, wav_path, candidates=candidates, lang=route_lang)
                     best_key = ctc_info.get("best_key") if isinstance(ctc_info, dict) else None
                     if best_key and best_key in candidates:
                         final_text = candidates[best_key]
                         final_source = "rover" if best_key == "rover" else f"asr:{best_key}"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating thresholds for bench_run**

I’m looking at updating the bench_run thresholds, specifically to set the final_prefer_router to True by default. Currently, in the cli.py, the thresholds dictionary includes route_lang_from_path, prefer_path, and filter_hypotheses_by_script. Adding final_prefer_router set to True could streamline things. It feels like a good move—let's patch this!


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
     thresholds: dict[str, object] = {
         "route_lang_from_path": True,
         "route_lang_prefer_path": True,
         "filter_hypotheses_by_script": True,
+        "final_prefer_router": True,
     }
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/per_lang_vaani_limit20_finalpref --whisper-model large-v3 --transformers-whisper-model-map data/bench/model_maps/whisper_vaani_map.json --limit 20


Output:
Exit code: 0
Wall time: 175.2 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 23:40:40.481479: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
`torch_dtype` is deprecated! Use `dtype` instead!
`generation_config` default values have been modified to match model-specific defaults: {'suppress_tokens': [], 'begin_suppress_tokens': [220, 50257]}. If this is not desired, please set these values explicitly.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> to see related `.generate()` flags.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> to see related `.generate()` flags.
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:02:42
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_finalpref/results.jsonl
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_finalpref/summary.json
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit20_finalpref/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 20,
  "wer_mean": 0.7959979979171156,
  "cer_mean": 0.604361854712759,
  "wer_p50": 0.9166666666666666,
  "wer_p90": 1.0,
  "wer_p95": 1.0,
  "cer_p50": 0.7532467532467533,
  "cer_p90": 0.8703703703703703,
  "cer_p95": 0.9,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 1.0,
  "wer_by_language_mean": {
    "asm": 0.7959979979171156
  },
  "wer_by_decision_mean": {
    "human": 0.7890204489353095,
    "reject": 0.9285714285714286
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.9534424603174604,
    "10-30s": 0.4597069597069597,
    "3-10s": 0.5932060858531446
  },
  "cer_by_language_mean": {
    "asm": 0.604361854712759
  },
  "cer_by_decision_mean": {
    "human": 0.5882520701611134,
    "reject": 0.9104477611940298
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.7925376533261822,
    "10-30s": 0.25581896551724137,
    "3-10s": 0.3441912205510858
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.7959979979171156,
    "selection_gap_wer_mean": 0.0,
    "selection_gap_wer_p90": 0.0
  }
}


Summary: auto


**Evaluating benchmark options**

I’m considering how to run a benchmark with certain constraints. The CLI doesn't support offsets, so I think about creating a custom directory that contains just specific files. We may have to run a full benchmark with around 2400 segments, which could take several hours, but the user wants to run a few hundred per language. Running 2400 segments allows for 200 per language. We might also adjust the builder to create a larger dataset later.


**Considering automation and dataset requirements**

I think we can implement automation to sample smaller datasets for quick testing. The user has requested to push a new dataset to Hugging Face, which should be version three with more samples and kept private. They also mentioned wanting to run benchmarks with several hundred samples for each language and analyze why certain languages perform better. Our current Word Error Rate for Assamese is 0.796, which is quite high and will require adjustments.


I'm noticing that the current value is lower than the previous one of 0.981, which is a good sign. Now, I'm thinking about running the limit at 250 again to confirm this finding. It's all about making sure I can reliably check the results and see if they hold up. Let's see how this next run turns out!


**Planning output configuration**

I'm planning to run new code for 36 minutes and output to a new directory named `outputs/bench/maya_hf_v2/per_lang_vaani_limit250_finalpref`. It seems like a good idea to set a timeout of about 9000000 ms, which is roughly 2.5 hours, just to be safe. I’ll proceed with that plan!


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/per_lang_vaani_limit250_finalpref --whisper-model large-v3 --transformers-whisper-model-map data/bench/model_maps/whisper_vaani_map.json --limit 250


Output:
Exit code: 0
Wall time: 2073.1 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-13 23:44:10.292505: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
`torch_dtype` is deprecated! Use `dtype` instead!
`generation_config` default values have been modified to match model-specific defaults: {'suppress_tokens': [], 'begin_suppress_tokens': [220, 50257]}. If this is not desired, please set these values explicitly.
Using custom `forced_decoder_ids` from the (generation) config. This is deprecated in favor of the `task` and `language` flags/config options.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensLogitsProcessor'> to see related `.generate()` flags.
A custom logits processor of type <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> has been passed to `.generate()`, but it was also created in `.generate()`, given its parameterization. The custom <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> will take precedence. Please check the docstring of <class 'transformers.generation.logits_process.SuppressTokensAtBeginLogitsProcessor'> to see related `.generate()` flags.
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:34:20
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit250_finalpref/results.jsonl
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit250_finalpref/summary.json
Wrote: outputs/bench/maya_hf_v2/per_lang_vaani_limit250_finalpref/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 250,
  "wer_mean": 0.7745981207682449,
  "cer_mean": 0.5616479331978755,
  "wer_p50": 0.8076923076923077,
  "wer_p90": 1.0,
  "wer_p95": 1.0,
  "cer_p50": 0.6120689655172413,
  "cer_p90": 0.9084507042253521,
  "cer_p95": 0.9473684210526315,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 0.992,
  "wer_by_language_mean": {
    "asm": 0.8027787032413584,
    "ben": 0.6618757908757908
  },
  "wer_by_decision_mean": {
    "human": 0.7459095745411738,
    "reject": 1.1443616054727168
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.975804095452924,
    "10-30s": 0.5871611598486358,
    "3-10s": 0.6273143763578749
  },
  "cer_by_language_mean": {
    "asm": 0.6151024603598705,
    "ben": 0.3478298245498956
  },
  "cer_by_decision_mean": {
    "human": 0.5243560048501482,
    "reject": 1.0422994541241395
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.8453463458236063,
    "10-30s": 0.2482702061049856,
    "3-10s": 0.3666910323897429
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.7527456902709927,
    "selection_gap_wer_mean": 0.021852430497252228,
    "selection_gap_wer_p90": 0.0
  }
}


**Planning for benchmarking tests**

The user wants to focus on achieving minimal Word Error Rate (WER) and is interested in fine-tuning for each language. They’re looking to create a benchmark dataset using 1000-2000 samples per language. Even though they didn't specifically ask for a plan, this seems like a multi-step, complex task. I think I should create a plan and also consider tracking progress with an update tool. I'll have to ensure previous code changes are accounted for and that new functions avoid import loops while possibly adding a script for automated benchmarking testing.


**Evaluating dataset creation requests**

The user previously requested "no new files," but now they're asking to create a new benchmark dataset under the HF ID, which implies some writing is needed. The tasks I need to tackle include running benchmarks with a few hundred to a thousand samples per language and analyzing languages with high Word Error Rate (WER) to make necessary fixes. The goal is to create a private benchmark dataset while ensuring sufficient sample sizes, checking the lengths of existing datasets for validity.


**Checking dataset lengths**

I think it's a good idea to quickly check the lengths of the 'valid' splits for Kathbath languages using Python. I’ll run a script to print these lengths, which should give me a clearer understanding of what datasets are available. This will help ensure that I have enough samples for each language in the benchmark dataset when I start creating it. It's all about making sure I have the right information first!


python3 - <<'PY'
from datasets import load_dataset
langs=['hindi','tamil','telugu','kannada','malayalam','bengali','gujarati','punjabi','marathi','odia']
for lang in langs:
    ds=load_dataset('ai4bharat/Kathbath',lang,split='valid')
    print(lang,'valid',len(ds))
PY


Output:
Exit code: 0
Wall time: 14.9 seconds
Output:
hindi valid 3151
tamil valid 2770
telugu valid 2379
kannada valid 2062
malayalam valid 1767
bengali valid 2236
gujarati valid 2905
punjabi valid 3256
marathi valid 2378
odia valid 2397


**Evaluating dataset limits**

I’m considering dataset accessibility. Malayalam has 1767 entries, which is below the 2000 target. To boost that, I might look at supplementing with train/test splits from other sources, since the user suggested pulling from various sources if a single dataset doesn’t suffice. For Assamese and English, I look at using FLEURS, which might give us more than 2000 total entries combined from train, validation, and test sets. Let's confirm those figures!


python3 - <<'PY'
from datasets import load_dataset
for cfg in ['as_in','en_us']:
    total=0
    for split in ['train','validation','test']:
        ds=load_dataset('google/fleurs',cfg,split=split,trust_remote_code=True)
        print(cfg,split,len(ds))
        total+=len(ds)
    print(cfg,'total',total)
PY


Output:
Exit code: 0
Wall time: 4.6 seconds
Output:
as_in train 2812
as_in validation 418
as_in test 984
as_in total 4214
en_us train 2602
en_us validation 394
en_us test 647
en_us total 3643


**Evaluating optimization strategies**

I'm realizing that if `require_lid_backend` is set to false and we fail to initialize the model, we'll end up with `lid_backend` as None, which is fine as long as we only call `lid_backend.predict` when it's not None. It might be better to add a parameter to disable LID prediction altogether. Additionally, there's a significant overhead with audio preprocessing using ffmpeg, and we could skip normalization for the existing 16k mono datasets to speed things up massively!


**Optimizing benchmarking pipeline**

I’m thinking we could really optimize our pipeline for benchmarking. If the input audio is already in the right format, we might skip the ffmpeg call and use the source file as `wav_path`. Adding an option like `preprocess_mode` or a threshold like `skip_normalize_if_wav_16k_mono` could help, but it might complicate things. Since the files are mono16 and already available, we could bypass normalization entirely, which is vital for performance with 12k segments. An option to skip preprocessing would streamline everything!


**Improving pipeline preprocessing**

Preprocessing is crucial for the main pipeline, so I’m considering adding a parameter like `use_src_wav: bool = False` to `run_pipeline`. If it’s true, we could set `wav_path` to the source. However, that might create unstable outputs. We could copy the file to `out_dir/audio`, but that could duplicate a large dataset, which we might not need for benchmarking. Implementing an option like `write_audio: bool = True` could help. For benchmarking, a "fast bench mode" using source audio directly without ffmpeg and avoiding audio duplicates would really speed things up and save space!


**Streamlining pipeline for benchmarking**

I see that making changes for user benchmarks could be valuable, especially if they want to run large tests. We might consider running the pipeline in a distributed or parallel fashion, but given time constraints, I think focusing on performance improvements is better. Looking at the preprocessing steps: it usually runs `ffprobe`, normalizes audio, and computes quality checks. If the source is already in WAV format, we could skip normalization and potentially ffprobe, so let’s check how `compute_qc` functions in the audio module.


sed -n '1,260p' maya_transcribe/audio.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
import subprocess
from dataclasses import dataclass
from pathlib import Path

import numpy as np

@dataclass(frozen=True)
class AudioInfo:
    path: str
    duration_sec: float | None
    sample_rate: int | None
    channels: int | None

@dataclass(frozen=True)
class AudioQCMetrics:
    duration_sec: float
    rms_dbfs: float
    peak_dbfs: float
    clipping_fraction: float

def ffprobe(path: Path) -> AudioInfo:
    cmd = [
        "ffprobe",
        "-v",
        "error",
        "-show_entries",
        "format=duration",
        "-show_entries",
        "stream=sample_rate,channels",
        "-of",
        "json",
        str(path),
    ]
    proc = subprocess.run(cmd, check=True, capture_output=True, text=True)
    data = json.loads(proc.stdout or "{}")

    duration_sec = None
    try:
        duration_sec = float(data.get("format", {}).get("duration"))
    except (TypeError, ValueError):
        duration_sec = None

    sample_rate = None
    channels = None
    streams = data.get("streams") or []
    if streams:
        s0 = streams[0] or {}
        try:
            sample_rate = int(s0.get("sample_rate")) if s0.get("sample_rate") else None
        except (TypeError, ValueError):
            sample_rate = None
        try:
            channels = int(s0.get("channels")) if s0.get("channels") else None
        except (TypeError, ValueError):
            channels = None

    return AudioInfo(path=str(path), duration_sec=duration_sec, sample_rate=sample_rate, channels=channels)

def decode_pcm16(path: Path, sample_rate: int = 16000, mono: bool = True) -> tuple[np.ndarray, int]:
    """
    Decode audio with ffmpeg into signed 16-bit PCM (optionally mono) at `sample_rate`.
    Returns (float32 waveform in [-1, 1], sample_rate).
    """
    cmd = ["ffmpeg", "-v", "error", "-i", str(path), "-f", "s16le", "-ar", str(sample_rate)]
    if mono:
        cmd += ["-ac", "1"]
    cmd += ["-"]

    proc = subprocess.run(cmd, check=True, capture_output=True)
    pcm = np.frombuffer(proc.stdout, dtype=np.int16)
    if pcm.size == 0:
        return np.zeros((0,), dtype=np.float32), sample_rate
    wav = pcm.astype(np.float32) / 32768.0
    return wav, sample_rate

def compute_qc(path: Path, sample_rate: int = 16000) -> AudioQCMetrics:
    wav, sr = decode_pcm16(path, sample_rate=sample_rate, mono=True)
    duration_sec = float(wav.shape[0]) / float(sr) if sr else 0.0
    if wav.size == 0:
        return AudioQCMetrics(duration_sec=0.0, rms_dbfs=float("-inf"), peak_dbfs=float("-inf"), clipping_fraction=0.0)

    peak = float(np.max(np.abs(wav)))
    rms = float(np.sqrt(np.mean(np.square(wav))))

    peak_dbfs = 20.0 * float(np.log10(max(peak, 1e-12)))
    rms_dbfs = 20.0 * float(np.log10(max(rms, 1e-12)))

    clipping_threshold = 0.999
    clipping_fraction = float(np.mean(np.abs(wav) >= clipping_threshold))

    return AudioQCMetrics(
        duration_sec=duration_sec,
        rms_dbfs=rms_dbfs,
        peak_dbfs=peak_dbfs,
        clipping_fraction=clipping_fraction,
    )

def normalize_audio(
    in_path: Path,
    out_path: Path,
    sample_rate: int = 16000,
    mono: bool = True,
    trim_silence: bool = True,
    silence_threshold_db: float = -40.0,
    min_silence_sec: float = 0.15,
) -> Path:
    """
    Convert audio to 16kHz mono WAV and optionally trim leading/trailing silence.
    """
    out_path.parent.mkdir(parents=True, exist_ok=True)

    cmd = ["ffmpeg", "-v", "error", "-y", "-i", str(in_path)]

    if trim_silence:
        # Trim leading/trailing silence; tuned for already-clean segments.
        # https://ffmpeg.org/ffmpeg-filters.html#silenceremove
        thr = f"{silence_threshold_db}dB"
        sil = (
            f"silenceremove=start_periods=1:start_duration={min_silence_sec}:start_threshold={thr}:"
            f"stop_periods=1:stop_duration={min_silence_sec}:stop_threshold={thr}"
        )
        cmd += ["-af", sil]

    cmd += ["-ar", str(sample_rate)]
    if mono:
        cmd += ["-ac", "1"]
    cmd += ["-c:a", "pcm_s16le", str(out_path)]

    subprocess.run(cmd, check=True)
    return out_path


Summary: auto


**Evaluating audio processing options**

I'm looking into how compute_qc utilizes ffmpeg for decoding, which is heavy, but it might be manageable. The main overhead seems to come from normalize_audio, which still uses ffmpeg conversion even for 16k .wavs. To benchmark effectively, I could skip normalize_audio and compute qc directly, or even consider skipping qc entirely, but that might not align with the user's quality gating needs. Maybe implementing a `skip_preprocess` option could help reduce overhead, avoiding unnecessary disk writes.


**Considering audio decoding options**

I might consider calling decode_pcm16 for ASR to streamline processes and avoid duplication. However, the faster-whisper and transformer currently decode separately. Since the user needs to run large benchmarks, maybe I'll provide a CLI that directly uses the dataset's wav files without copying.

I could add a run_pipeline parameter, allowing it to skip preprocessing if needed. This means setting wav_path to src and skipping normalize_audio. While we wouldn't store audio in out_dir, the record can have the same src_path and wav_path, simplifying the process.


**Weighing normalization choices**

It's clear that for non-bench runs, the user might prefer normalization, so I think I’ll keep that as the default. However, for bench runs, I might set preprocess_mode to "none" to speed things up, although that could slightly affect evaluation results by not trimming silence, potentially degrading some transcripts.

But for benchmarking, maybe we should evaluate as in production, including preprocessing. We could keep normalization and use the input file if it’s already 16k mono and trimming isn’t needed. Still, if our silence threshold is -40dB and min_silence is 0.15, it might cut important phonemes.


**Considering WER adjustments**

The high WER of 0.6-0.8 might not be solely due to trimming, so for bench evaluations, I’m thinking we could disable trim_silence to prevent any audio modifications. However, the pipeline uses trim_silence True as the default. To optimize for minimal WER, perhaps we should set trim_silence to False for the benchmarks. It’d be interesting to test this quickly on a subset with that option, but I'd need to focus on making the benchmarking automated first. I could allow for smaller sample runs for efficiency.


**Optimizing sample benchmarks**

I’m thinking about running a smaller sample size for each language, like 50, which would take around 81 minutes. However, the user requested a few hundred samples each. I could start with 50 as a test. If we run 200 each, optimizing the pipeline by skipping normalization and LID could significantly reduce the runtime. It might drop the total time to about 1-2 hours. I need to estimate times for each step and see how that might work out.


**Streamlining the pipeline**

I’m considering skipping the faster-whisper sampling altogether since it’s heavy and not currently needed. This would save time in the pipeline. Normally, we run the sampling pass and store results, but it's a waste now. I could add a threshold to disable sampling when using certain backends. I’ll check if `do_sampling` should be based on thresholds. For benchmark runs, I could set it to False when extra transformer models are in use. It seems we can default to False if other backends are present. Let's implement this!


Input:
*** Begin Patch
*** Update File: maya_transcribe/pipeline.py
@@
     gemini_backend = None
     if gemini_model:
         try:
             from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template

@@
         except Exception as e:
             raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e

+    default_whisper_run_sampling = not bool(transformers_backends or transformers_router or gemini_backend is not None)
+
     results_path = out_dir / "results.jsonl"
     summary_path = out_dir / "summary.json"
@@
                 try:
                     det = whisper_backend.transcribe(
                         wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
                     )
                     primary_asr_key = f"{det.model}|det"
                     asr_results[primary_asr_key] = det
                 except Exception as e:
                     asr_errors[f"{whisper_backend.name}|det"] = f"{type(e).__name__}:{e}"

-                try:
-                    samp = whisper_backend.transcribe(wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2)
-                    asr_results[f"{samp.model}|samp_t0.2"] = samp
-                except Exception as e:
-                    asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"
+                run_sampling = bool(thresholds.get("whisper_run_sampling", default_whisper_run_sampling))
+                if run_sampling:
+                    try:
+                        samp = whisper_backend.transcribe(
+                            wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2
+                        )
+                        asr_results[f"{samp.model}|samp_t0.2"] = samp
+                    except Exception as e:
+                        asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Planning analysis and tuning**

I'm considering how to provide a thorough analysis and what adjustments or "knobs" I might need for tuning. It's important that I acknowledge we've completed step one already, but I want to include it in my overall plan for clarity. Ensuring everything is organized and reflects sequential progress will help guide my next steps efficiently. I’m eager to make sure I’m clear and methodical in my approach!


**Estimating processing time**

I’m estimating that we can handle 2400 segments with improved speed, now that sampling is removed. Previously, processing 250 segments took 34 minutes, and this time I believe I can reduce segment time by about 20%. So, it looks like processing 2400 segments might take around 4-5 hours, which is a long run. We could also consider running 200 segments per language sequentially to process initial datasets more quickly.


**Exploring dataset creation**

I’m looking at a total of 2400 segments and considering running each language separately, which might require interpretation after each run. Even though we could compute this overall, managing different sessions could be tricky.

Building a dataset with 2000 files per language means dealing with 24000 audio files, which is quite a bit in terms of disk space and time, but it may be doable. Instead of writing all the audio, I could create symlinks or references to the underlying HF dataset while keeping in mind that the pipeline expects local files.


**Considering dataset modifications**

I'm thinking about modifying the pipeline to support streaming datasets, but that might get complex. As an alternative, we could keep using the current v2 dataset with 2400 segments and maybe extend it to v3 with 500 segments per language, totaling 6000, which sounds more manageable in terms of disk space and time. The user has requested 1000-2000 files, so building a dataset with 1000 per language would mean around 12000 audio files. Each file averages about 0.16MB, totaling roughly 1.9GB; for 2000 files, it’s about 3.8GB, which is manageable!


**Compromising on dataset creation**

I’m considering a compromise: build the v3 dataset with 1000 segments per language, push the manifest to HF, and run the pipeline on a smaller sample of about 200 segments per language to compute WER. This setup will also allow for easier full runs later. The user has requested benchmarks with a few hundred samples each and emphasized the need for constant evaluation.

I could set up infrastructure to evaluate the v2 dataset with around 200 or 300 segments per language. I also want to improve the bench builder by including additional sources like Shrutilipi and Svarah, which could enhance domain diversity and robustness.


**Exploring dataset updates**

I'm thinking about updating the default sources to include more options for each language as a fallback in case there aren't enough samples. It seems we can add more HFSource entries from datasets like Svarah, Shrutilipi, and Common Voice. However, I need to check the dataset structure, including configs, splits, and the specific audio/text keys. This might take some time to inspect thoroughly to ensure everything aligns correctly.


**Considering dataset structure**

I’m thinking about using Common Voice for both Assamese and English. Given the minimal requirement, we can keep the builder as is, allowing the user to extend it if needed. The user wants us to implement something dynamic: if there aren’t enough samples from the primary source, we can explore other datasets. Currently, our default_sources for Assamese includes train, validation, and test. I’m considering updating this to consider test and valid before train, keeping evaluation fair.


**Checking dataset splits**

Right now, the builder only uses the "valid" split, but I think we can extend that. I want to check if the Kathbath dataset includes split names like "train," "valid," and "test." It seems to use "valid" instead of "validation," which is a little confusing. I should look at the dataset configuration to see if it has "train" and "valid" and possibly no test. I'll quickly run a query to load the dataset and check for splits.


python3 - <<'PY'
from datasets import get_dataset_split_names
print(get_dataset_split_names('ai4bharat/Kathbath','hindi'))
PY


Output:
Exit code: 0
Wall time: 3 seconds
Output:
['valid', 'train']


**Considering benchmarking samples**

I'm thinking about how to handle training and validation data. To reach the target for Malayalam, we need 2000 samples, and with 1767 valid ones, I can include train data as a fallback, even if it's not ideal. If the user wants minimal word error rate, it might be better to rely solely on validation for benchmarking. They requested 1000-2000 samples, and since 1767 fits that range, we can proceed without using train data.


**Deciding on dataset configuration**

I'm considering how to handle the dataset for a larger size. It seems like I can keep the builder as it is. For running `bench build`, I’ll set `samples_per_lang=1000` and use the valid and FLEURS splits without needing to make any changes. This approach feels straightforward, and it's good to have clarity on the next steps to make sure everything runs smoothly. So, let's move forward with that plan!


Input:
*** Begin Patch
*** Update File: maya_transcribe/pipeline.py
@@
                 final_text = rover.text
                 final_source = "rover"
                 if router_key and router_key in rover_hyps and bool(thresholds.get("final_prefer_router", False)):
                     final_text = rover_hyps[router_key]
                     final_source = f"asr:{router_key}"
+                elif bool(thresholds.get("final_prefer_transformers", False)):
+                    tf_keys = [k for k in rover_hyps.keys() if k.startswith("transformers:")]
+                    if tf_keys:
+                        preferred = tf_keys[0]
+                        final_text = rover_hyps[preferred]
+                        final_source = f"asr:{preferred}"
                 if ctc_scorer is not None and rank_by_ctc is not None:
                     candidates = {"rover": rover.text, **rover_hyps}
                     ctc_info = rank_by_ctc(ctc_scorer, wav_path, candidates=candidates, lang=route_lang)
                     best_key = ctc_info.get("best_key") if isinstance(ctc_info, dict) else None
                     if best_key and best_key in candidates:
                         final_text = candidates[best_key]
                         final_source = "rover" if best_key == "rover" else f"asr:{best_key}"
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/pipeline.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Updating thresholds**

I’m looking to update the bench_run thresholds to include the final_prefer_transformers set to True by default. It seems like I need to make a change in cli.py to incorporate this. I have to be careful with how I implement this, so I’ll ensure it’s done correctly to avoid any issues down the line. Let’s make sure everything works smoothly after the patch! This improvement could really enhance performance.


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
     thresholds: dict[str, object] = {
         "route_lang_from_path": True,
         "route_lang_prefer_path": True,
         "filter_hypotheses_by_script": True,
         "final_prefer_router": True,
+        "final_prefer_transformers": True,
     }
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/indic_whisper_limit250 --whisper-model large-v3 --transformers-whisper-model Vignesh-M/Indic-whisper --limit 250


Output:
Wall time: 715.8 seconds
aborted by user


# Context from my IDE setup:

## Open tabs:
- results.jsonl: outputs/bench/maya_hf_v2/indic_whisper/results.jsonl
- summary.json: outputs/bench/maya_hf_v2/indic_whisper/summary.json
- eval.json: outputs/bench/maya_hf_v2/indic_whisper/eval.json
- results.jsonl: outputs/bench/maya_hf_v2/per_lang_vaani/results.jsonl
- summary.json: outputs/test_run5/summary.json

## My request for Codex:
but doesnt indicwhisper and whisperv3 large and other ASR models mention they have 0.08 ish WER scores right ? why is our ensemble pipelines giving 80 and 99 on same benchmarks.

Those “0.08 WER” claims are 0.08 = 8% word error rate on a specific setup (dataset + language + normalization + decoding settings). Your runs are showing ~0.86 = 86% word error rate, which is “mostly wrong”, and it’s happening for a few concrete reasons:

You didn’t run “AI4Bharat IndicWhisper” in indic_whisper/. You ran transformers:Vignesh-M/Indic-whisper (unofficial), and in this benchmark it’s worse than faster-whisper:large-v3 (WER ≈ 0.948 vs 0.854 on the same evaluated pairs).
Benchmark corruption from silence trimming: for FLEURS English, 146/200 clips became 0.0s audio after silenceremove at -40dB (the audio is extremely quiet). Those segments inevitably get empty transcripts → WER near 1.0. This alone can drag averages toward ~0.8–1.0 even if the model is decent.
Wrong language-code handling for Odia in this repo version: ISO3_TO_WHISPER_LANG maps ory -> "or", but faster-whisper does not accept "or", so you get many hard errors; your indic_whisper run has 190 errors and 403 rejects in summary.json.
You’re averaging over everything (including rejects/human). In eval.json, wer_by_decision_mean shows accept ≈ 0.595 but reject ≈ 1.075. If your goal is “auto-accepted quality”, you should report WER on decision=accept separately.
WER is brittle for some scripts/spacing (ASR outputs may omit spaces; references include them). CER is usually more stable across Indic scripts.
So no, it’s not that “ensembling makes WER 80%”; it’s mostly pipeline + benchmark configuration issues (trim + lang codes) + evaluating hard/empty clips, and also the added model isn’t the one whose 0.08 numbers you’re thinking of.

Why tests take so long:

Each segment runs ffmpeg preprocess + MMS-LID + two faster-whisper decodes (det + samp) + optional transformers model; transformers Whisper is much slower than faster-whisper and can OOM, which your per_lang_vaani run shows (error=1708/2400).
then why tf is my other agent working so fking jard with all those stupid models. can you instruct it to stick with standard models and all these chapri models dont work well or out of the box ? and when tf did we deviate from original plan. our original plan was master.md and i dont remember we have language specific models and each model for each language.

You’re right to be skeptical, and you should tell the other agent to stop chasing random checkpoints.

What to tell your other agent (copy/paste)

“Stop adding/benchmarking unofficial whisper fine-tunes (random HF users). They’re not improving WER here and they increase failure modes (script mismatch, decoding quirks, OOM).”
“Return to the plan in master.md: baseline must be faster-whisper large-v3 + MMS-LID + quality gating + (optional) audio-grounded CTC-fit scoring + (optional) Gemini on hard segments only.”
“Only use ‘standard’ models from top orgs if we add diversity: OpenAI Whisper (faster-whisper large-v3) and Meta MMS/Seamless (for LID/CTC fit or alternate ASR). No per-language zoo unless we have strong evidence and reproducible wins.”
“Fix benchmark validity first: rerun benches with --trim-silence false (or much lower threshold) because FLEURS audio is very quiet and current trimming zeros out 146/200 English clips, inflating WER.”
“Report metrics correctly: WER overall + WER for decision=accept only; log error-rate and reasons. Don’t compare runs with huge error counts.”
When/why we deviated

The deviation happened while trying to lower WER quickly by adding an extra hypothesis model, then it snowballed into a per-language router experiment (per_lang_vaani*). That’s not required by master.md and it’s currently causing lots of OOM/errors and noisy conclusions.
If you want, I can point to the exact evidence you can cite: eval.json shows the added transformers:Vignesh-M/Indic-whisper is worse than faster-whisper:large-v3|det, and summary.json shows massive errors (OOM) making those runs invalid for decisions.

Stop chasing all these fucking dumbass models. stick with standard ones, report scores and we shall debug why they are so low once you're done. only standard and trusted models only, not random checkpoints from random sources.


**Evaluating WER calculations**

The WER claims of 0.08 refer to the same dataset, like FLEURS, but the evaluation process might have used custom normalization that's too aggressive. For languages like Hindi, word boundaries are defined by spaces, but Bengali or Assamese also have spaces. The observed WER of 0.66 is significantly higher than expected. It’s possible that model performance is lacking or the evaluation might have mismatched languages. Additionally, punctuation removal could be causing issues with transliteration in outputs.


**Addressing WER discrepancies**

I think we can adjust the standard model by setting the faster-whisper language to 'bn' for Assamese, which should help ensure the correct script is output. However, the WER is still high. One major issue is that our evaluation removes punctuation and converts digits, while WER relies on spaces. Indic scripts generally do use spaces, but some datasets may include punctuation that gets removed. The reported 0.08 WER might apply to in-domain data, but our benchmarks reflect more challenging segments. We need to clarify that our evaluations differ.


**Planning benchmark adjustments**

The user wants me to instruct another agent, but since I'm the assistant, I can only provide a message for the user to send. I’ll suggest that they stop chasing certain models and stick with the standard ones, reporting scores for debugging later. I need to implement a few changes: ensure the benchmark run only uses faster-whisper large-v3, disable any dynamic model map, and produce results from the analysis tool with the correct settings. The environment approval is never, and I'm debating whether to run a heavy benchmark or just a subset.


**Deciding on benchmarking strategy**

The user previously requested a few hundred segments per language, but now wants to focus on sticking to the standard process, reporting scores afterward. I’m considering running a baseline of 2400 segments, which might take hours. However, I might run 200 segments per language since the dataset is already organized that way. The faster-whisper large-v3 model could work across all segments, but decoding and processing time could add up. Disabling optional sampling and the LID step could help speed things up, allowing us to optimize the benchmark run.


**Considering benchmark configurations**

I could add a flag to the benchmark run like `--skip-lid` or change the `require_lid_backend` parameter. If it's set to false and initialization fails, we can skip it. But there might be alternatives for the LID backend. While the user prefers standard settings, we want to minimize WER, so it makes sense to compute it by decisions and implement more detailed reporting options. I also need to address the Assamese script mapping and consider the Odia language handling too. Using CER instead of WER might better reflect accuracy across scripts with unique spacing. I'll explain the context to the user regarding WER expectations.


python3 - <<'PY'
import json, collections
from pathlib import Path

def analyze(run_dir):
    p=Path(run_dir)/'results.jsonl'
    if not p.exists():
        print('missing',p); return
    dur0=collections.Counter(); total=collections.Counter(); bylang=collections.Counter();
    for line in p.open(encoding='utf-8'):
        r=json.loads(line)
        if r.get('error'):
            continue
        sid=r.get('segment_id','')
        rel=r.get('relpath','')
        lang = (rel.split('/',1)[0] if '/' in rel else None) or (r.get('route',{}) or {}).get('language_hint')
        qc=r.get('qc') or {}
        dur=float(qc.get('duration_sec') or 0.0)
        total[lang]+=1
        if dur <= 0.01:
            dur0[lang]+=1
        if dur<0.8:
            bylang[lang]+=1
    print('run',run_dir)
    print('total',sum(total.values()))
    print('dur<=0.01 top', dur0.most_common(20))
    print('dur<0.8 top', bylang.most_common(20))

analyze('outputs/bench/maya_hf_v2/indic_whisper')
analyze('outputs/bench/maya_hf_v2/per_lang_vaani')
PY


python3 - <<'PY'
import json, collections
from pathlib import Path
p=Path('outputs/bench/maya_hf_v2/indic_whisper/summary.json')
print(p.read_text() if p.exists() else 'missing')
PY


Output:
Exit code: 0
Wall time: 0.2 seconds
Output:
run outputs/bench/maya_hf_v2/indic_whisper
total 2210
dur<=0.01 top [('eng', 146), ('mar', 6), ('kan', 5), ('tel', 5), ('guj', 1), ('mal', 1)]
dur<0.8 top [('eng', 155), ('mal', 33), ('tel', 32), ('kan', 29), ('mar', 22), ('hin', 20), ('ben', 19), ('asm', 13), ('guj', 7), ('pan', 7), ('tam', 6)]
run outputs/bench/maya_hf_v2/per_lang_vaani
total 692
dur<=0.01 top [('eng', 146), ('mar', 6), ('kan', 5), ('tel', 5), ('guj', 1), ('mal', 1)]
dur<0.8 top [('eng', 146), ('ben', 18), ('asm', 13), ('mar', 6), ('kan', 5), ('mal', 5), ('tel', 5), ('ory', 3), ('guj', 1), ('tam', 1)]


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "segments_total": 2400,
  "counts": {
    "accept": 23,
    "human": 1784,
    "reject": 403,
    "error": 190
  },
  "avg_total_sec": 4.253917385618375,
  "p50_total_sec": 4.277748890221119,
  "p90_total_sec": 6.6006958852522075
}


**Evaluating language output**

I'm checking the output for Assamese, which should be set to Bengali script. That matches expectations, so all good there! For other languages, the script also needs to match. There’s a chance of getting a romanized output if the language doesn't match correctly. I’ve got to think about how the scripts filter out. If Oriya outputs as Latin script, that could cause empty outputs, so disabling filtering could allow for romanization, although it would increase the WER. We’ll keep adjusting.


**Analyzing WER causes**

I’m looking into the overall 0.8 WER and noting a few root causes: silent trimming leading to empty audio segments, errors with Odia, issues with the per-language models, and evaluation discrepancies with rejects and humans. I think I might patch the bench_run to disable silence trimming, which could help. Running a baseline on 250 segments or even a full 2400 might be a good approach, although it could take hours. I should limit the first run to see if the WER improves before gathering results to report.


**Planning benchmarking with languages**

I need to show the instructions to another agent while also focusing on implementing a fix myself and validating it through a subset. I’m thinking of running `bench run` with a limit of around 400 to check the WER. However, the limitation might cause a biased sample since it could sort by languages like `asm` and `ben` first, which isn't representative. To improve on that, I could implement an option in `bench run` to shuffle files or limit based on languages. The user has suggested running benchmarks with hundreds of samples per language, which should be doable!


**Addressing user feedback on models**

The user wants me to stop using extra models and focus on debugging. I think it would be a good idea to build a new benchmark with 1000 samples per language and run a baseline after fixing the issues. We need to remove these "chapri" models from the model map, especially since we updated the Malayalam model to DrishtiSharma. It seems sensible to make flags optional in the code and ensure defaults don't prefer transformers, as that could lead to degradation. Adjusting the documentation might also help.


**Adjusting bench run settings**

I’m setting the new bench thresholds to conservative levels, activating only route_lang_from_path and avoiding filtering by script or transformers. While script filtering helps prevent mixed scripts, I think keeping the default robust is key. Even though the user wants standard settings, I should consider disabling some preferences while ensuring the setup aligns with the master plan that uses faster-whisper.

For the upcoming benchmark, it's essential to disable trim_silence, and I also need to ensure we handle tokens safely in .env, avoiding exposure in responses. I’ll confirm the necessary changes and explain the WER issues regarding trimmed segments. Finally, I’ll propose enhancements for evaluation based on accepted decisions.


**Implementing evaluation updates**

I’m planning to run an evaluation of the old runs while ignoring error segments since the evaluation already overlooks those by not including final results. I think rerunning with the `--decision accept` option could yield better WER scores. I’ll also patch the `cli.py` eval command to allow multiple decision options.

There's a consideration to create an `eval_accept.json` output for easier readability, which the user now seems okay with, even though there was originally a request for no new files. For consistent benchmarking, I could extend the existing `eval.json` to include accept-only statistics, and I’ll also update the summary to reflect WER for only accepted results. Let's get to coding these changes!


Input:
*** Begin Patch
*** Update File: maya_transcribe/cli.py
@@
 def eval(
     results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
     refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
     out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
     hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
     oracle: bool = typer.Option(True, help="Compute oracle-best and selection-gap stats across all candidates."),
     language_source: str = typer.Option("route", help="Use language from: 'route' (LID) or 'refs' (refs.jsonl)."),
+    decision: list[str] = typer.Option([], help="Filter to verification decision(s): accept/human/reject (repeatable)."),
 ) -> None:
@@
     summary = evaluate(
         results_path=results,
         refs_path=refs,
         out_path=out,
         hypothesis=hypothesis,
         oracle=oracle,
         language_source=language_source,
+        decision_filter=decision or None,
     )
     typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))
@@
 def bench_run(
     bench_dir: Path = typer.Option(Path("data/bench/maya_hf_v0"), exists=True, file_okay=False, help="Benchmark corpus directory."),
     out: Path | None = typer.Option(None, help="Output directory (defaults under outputs/bench/)."),
     whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
     lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
     require_lid_backend: bool = typer.Option(True, help="Fail if MMS-LID backend can't initialize."),
@@
     ctc_model_map: Path | None = typer.Option(None, help="JSON map {lang_code: hf_model_id} for CTC scoring."),
     require_ctc: bool = typer.Option(True, help="When CTC is enabled, require a CTC score to accept."),
     min_ctc_margin: float = typer.Option(0.05, help="Min (second-best - best) CTC loss margin to auto-accept."),
-    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
+    trim_silence: bool = typer.Option(
+        False, help="Trim leading/trailing silence (ffmpeg silenceremove). Off by default for benchmarking."
+    ),
     silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
     min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
     limit: int | None = typer.Option(None, help="Process only the first N segments."),
     thresholds_json: Path | None = typer.Option(None, help="Optional JSON file of threshold overrides (verify + routing)."),
     hypothesis: str = typer.Option("final", help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
@@
-    thresholds: dict[str, object] = {
-        "route_lang_from_path": True,
-        "route_lang_prefer_path": True,
-        "filter_hypotheses_by_script": True,
-        "final_prefer_router": True,
-        "final_prefer_transformers": True,
-    }
+    thresholds: dict[str, object] = {"route_lang_from_path": True, "route_lang_prefer_path": True}
     if thresholds_json is not None:
         loaded = json.loads(thresholds_json.read_text(encoding="utf-8"))
         if not isinstance(loaded, dict):
             raise typer.BadParameter("thresholds_json must be a JSON object")
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/cli.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto