Input:
*** Begin Patch
*** Update File: maya_transcribe/eval.py
@@
 def evaluate(
     results_path: Path,
     refs_path: Path,
     out_path: Path | None = None,
     hypothesis: str = "final",
     oracle: bool = True,
     language_source: str = "route",
+    decision_filter: list[str] | None = None,
 ) -> dict[str, Any]:
     refs = load_refs(refs_path)
     results = read_jsonl(results_path)

+    decision_filter_set = None
+    if decision_filter:
+        decision_filter_set = {str(d).strip().lower() for d in decision_filter if str(d).strip()}
+
     rows: list[EvalRow] = []
     for r in results:
         segment_id = str(r.get("segment_id") or "").strip()
         if not segment_id or segment_id not in refs:
             continue
@@
         decision = None
         ver = r.get("verification") or {}
         if isinstance(ver, dict):
             decision = ver.get("decision")
+        if decision_filter_set is not None:
+            dec_norm = str(decision or "").strip().lower()
+            if dec_norm not in decision_filter_set:
+                continue

         duration_sec = None
         qc = r.get("qc") or {}
         if isinstance(qc, dict):
             try:
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/eval.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


python3 -m py_compile maya_transcribe/cli.py maya_transcribe/eval.py maya_transcribe/pipeline.py maya_transcribe/asr.py maya_transcribe/language.py


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:


Summary: auto


maya-transcribe eval --results outputs/bench/maya_hf_v2/indic_whisper/results.jsonl --refs data/bench/maya_hf_v2/refs.jsonl --language-source refs --hypothesis final --decision accept


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:
{
  "hypothesis": "final",
  "pairs_evaluated": 23,
  "wer_mean": 0.594883999638253,
  "cer_mean": 0.5772124867965942,
  "wer_p50": 0.64,
  "wer_p90": 0.8695652173913043,
  "wer_p95": 0.9285714285714286,
  "cer_p50": 0.6792452830188679,
  "cer_p90": 0.8627450980392157,
  "cer_p95": 0.897196261682243,
  "segment_error_rate": 1.0,
  "pct_wer_gt_5pct": 1.0,
  "pct_wer_gt_10pct": 0.9130434782608695,
  "wer_by_language_mean": {
    "eng": 0.580308424264946,
    "tam": 0.6222132034632035
  },
  "wer_by_decision_mean": {
    "accept": 0.594883999638253
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.7338504594094657,
    "3-10s": 0.3787139511052555
  },
  "cer_by_language_mean": {
    "eng": 0.582538788957638,
    "tam": 0.567225670244637
  },
  "cer_by_decision_mean": {
    "accept": 0.5772124867965942
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.7338703900053122,
    "3-10s": 0.3335224151385883
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.5919854489136153,
    "selection_gap_wer_mean": 0.0028985507246376803,
    "selection_gap_wer_p90": 0.0
  }
}


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/fw_largev3_baseline_ntrim_limit650 --whisper-model large-v3 --limit 650 --trim-silence false


Output:
Exit code: 2
Wall time: 0.2 seconds
Output:
Usage: maya-transcribe bench run [OPTIONS]
Try 'maya-transcribe bench run --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Got unexpected extra argument (false)                                        │
╰──────────────────────────────────────────────────────────────────────────────╯


**Evaluating flag options**

I'm considering the boolean flags `--trim-silence` and `--no-trim-silence`. Typer defines these as options, but it seems like we originally used `typer.Option(False, help=...)` without the `--no-trim-silence` option. This means we can pass `--trim-silence` to set it to True, but just omitting the flag sets it to False by default, which actually works out well. I think we should rerun without specifying `--trim-silence false`.


maya-transcribe bench run --bench-dir data/bench/maya_hf_v2 --out outputs/bench/maya_hf_v2/fw_largev3_baseline_ntrim_limit650 --whisper-model large-v3 --limit 650


Output:
Wall time: 719.3 seconds
aborted by user