Output:
Exit code: 0
Wall time: 0.2 seconds
Total output lines: 7593
Output:
/home/ubuntu/maya-transcribe/Agents.md:61:  - **Max disagreement allowed between systems:**
/home/ubuntu/maya-transcribe/Agents.md:200:- **Secondary:** accept rate, reject rate, human edit rate, model-disagreement rate, alignment pass rate, hallucination incidence (e.g., repeated phrases / generic “thanks for watching” patterns).
/home/ubuntu/maya-transcribe/Agents.md:203:  - **Max disagreement allowed between systems:** require strong agreement among at least 2 diverse recognizers (ROVER consensus score / pairwise CER threshold; tune).
/home/ubuntu/maya-transcribe/Agents.md:233:- C2: **Explicit system-combination (ROVER)** + “disagreement map” as a debug artifact.
/home/ubuntu/maya-transcribe/Agents.md:327:- C1: **Hardline LID gating**: disagreement → quarantine/discard to prevent dataset poisoning.
/home/ubuntu/maya-transcribe/Agents.md:350:- Suggestion: 3-model LID ensemble and quarantine on disagreement
/home/ubuntu/maya-transcribe/Agents.md:370:Defines a multi-stage “ensemble + consensus + verification” architecture with an explicit LLM-as-judge prompt (selection-only), disagreement detection, and a composite quality score that combines agreement, confidence, LM plausibility, and audio quality.
/home/ubuntu/maya-transcribe/Agents.md:379:- S2: Provides practical engineering logic for disagreement routing and composite scoring.
/home/ubuntu/maya-transcribe/Agents.md:391:- [ ] Implement disagreement detection (pairwise CER/WER variance) and route only hard cases to the judge/human.
/home/ubuntu/maya-transcribe/Agents.md:568:- Use an annotation UI that shows: audio + top hypotheses + disagreement highlights (Ref1) and lets the annotator pick/edit.
/home/ubuntu/maya-transcribe/Agents.md:577:- `rover_output` + `disagreement_map`
/home/ubuntu/maya-transcribe/Agents.md:627:│   + disagreement map (audit)   │
/home/ubuntu/maya-transcribe/Agents.md:653:- **Combines, not guesses:** ROVER/confusion-network voting reduces WER when recognizers disagree in different places; consensus plus a disagreement map makes errors visible and auditable (Refs 1/4/5).
/home/ubuntu/maya-transcribe/Agents.md:661:4. **Human loop:** Add a review queue/UI for rejects that shows audio + top hypotheses + disagreement spans; feed corrections back into threshold tuning (and optional fine-tunes later).
/home/ubuntu/maya-transcribe/Agents.md:667:**Phase B (quality hardening):** add N-best storage, better disagreement analytics, forced alignment where feasible, and a constrained LLM tie-breaker only for narrow cases.
/home/ubuntu/maya-transcribe/Agents.md:679:- `CombineResult`: `{method: rover, text, consensus_score, disagreement_spans[]}`
/home/ubuntu/maya-transcribe/Agents.md:694:- Track top reject reasons (low LID confidence, high disagreement, low alignment score, audio QC failure).
/home/ubuntu/maya-transcribe/Agents.md:721:- Collect rejects + hard disagreements as an “error discovery” pool.
/home/ubuntu/maya-transcribe/docs/Reference1.md:65:   * a “disagreement map” (positions where models differed)
/home/ubuntu/maya-transcribe/docs/Reference1.md:102:  * ROVER disagreement rate < 2–3%
/home/ubuntu/maya-transcribe/docs/Reference1.md:133:* the disagreement spans only
/home/ubuntu/maya-transcribe/docs/Reference1.md:192:  -> ROVER combine + disagreement map
/home/ubuntu/maya-transcribe/docs/Reference1.md:207:If you want, I can turn this into: (1) concrete threshold recommendations (per-language), (2) a model-ranking protocol using a 1–2k clip gold set across languages, and (3) the exact data structures for storing hypotheses + align scores so you can debug disagreements quickly.
/home/ubuntu/maya-transcribe/docs/Reference3.md:60:    B -- Disagreement --> D[Discard/Manual]
/home/ubuntu/maya-transcribe/docs/Reference2.md:39:Automatic Uncertainty Flags: Use the ASR models’ confidence scores (if available) or the disagreement between models as a signal. For example, any segment where the ensemble had low agreement or the LLM raised an uncertainty can be queued for manual review.
/home/ubuntu/maya-transcribe/outputs/test_run3/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run3/audio/SPEAKER_00_0000_0.03-12.15.wav", "ffprobe": {"path": "outputs/test_run3/audio/SPEAKER_00_0000_0.03-12.15.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "eng", "prob": 0.999092698097229}, {"code": "spa", "prob": 0.00018186250235885382}, {"code": "haw", "prob": 6.795905937906355e-05}, {"code": "lat", "prob": 6.20875071035698e-05}, {"code": "war", "prob": 3.527590160956606e-05}]}, "route": {"language_hint": "eng"}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 1.432625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "eng", "top1_prob": 0.999092698097229, "margin": 0.9989108355948702, "confident": true}, "no_speech_prob_max": 0.1619873046875, "heuristics_asr_key": "faster-whisper:tiny|det", "avg_logprob_mean": -0.5895182490348816, "compression_ratio_mean": 0.6666666666666666}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true}, "timing": {"preprocess_sec": 0.14881448168307543, "lid_sec": 0.44112175004556775, "asr_sec": 0.8713528169319034, "normalize_sec": 5.707796663045883e-05, "rover_sec": 9.884685277938843e-05, "verify_sec": 6.892019882798195e-05, "total_sec": 1.4615906099788845}}
/home/ubuntu/maya-transcribe/outputs/test_run5/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391afc89b", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run5/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "src_ffprobe": {"path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "duration_sec": 12.11625, "sample_rate": 48000, "channels": 1}, "ffprobe": {"path": "outputs/test_run5/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "eng", "prob": 0.999092698097229}, {"code": "spa", "prob": 0.00018186250235885382}, {"code": "haw", "prob": 6.795905937906355e-05}, {"code": "lat", "prob": 6.20875071035698e-05}, {"code": "war", "prob": 3.527590160956606e-05}]}, "route": {"language_hint": "eng"}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "ensemble": {"unique_models": ["faster-whisper:tiny"], "unique_backends": ["faster-whisper"], "warnings": ["low_model_diversity"]}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "ctc": null, "final": {"text": "intricate web of", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 1.432625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "eng", "top1_prob": 0.999092698097229, "margin": 0.9989108355948702, "confident": true}, "no_speech_prob_max": 0.1619873046875, "heuristics_asr_key": "faster-whisper:tiny|det", "avg_logprob_mean": -0.5895182490348816, "compression_ratio_mean": 0.6666666666666666, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.19274936662986875, "lid_sec": 0.4290622090920806, "asr_sec": 0.784179609734565, "normalize_sec": 5.9102196246385574e-05, "rover_sec": 9.4579067081213e-05, "ctc_sec": 2.398155629634857e-07, "verify_sec": 7.834797725081444e-05, "total_sec": 1.406317532993853}}
/home/ubuntu/maya-transcribe/master.md:68:│   + disagreement map (audit)   │
/home/ubuntu/maya-transcribe/master.md:108:- **ROVER / confusion-network voting** (keep the disagreement map)
/home/ubuntu/maya-transcribe/master.md:139:   - Create a review queue that shows audio + top hypotheses + disagreement spans for fast correction.
/home/ubuntu/maya-transcribe/master.md:151:- ROVER output + disagreement map
/home/ubuntu/maya-transcribe/docs/Reference4.md:64:    6. If high disagreement (variance > threshold) → flag for Stage 4
/home/ubuntu/maya-transcribe/docs/Reference4.md:171:│          STAGE 4: LLM-AS-JUDGE (Disagreement Cases)             │
/home/ubuntu/maya-transcribe/docs/Reference4.md:246:When models disagree, check if disagreements are phonetically similar:
/home/ubuntu/maya-transcribe/docs/Reference4.md:266:6. Add LLM-as-Judge for disagreements
/home/ubuntu/maya-transcribe/outputs/test_run6/results.jsonl:1:{"schema_version": 1, "segment_id": "SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391afc89b", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run6/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "src_ffprobe": {"path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "duration_sec": 12.11625, "sample_rate": 48000, "channels": 1}, "ffprobe": {"path": "outputs/test_run6/audio/SPEAKER_00_0000_0.03-12.15-01b2a08391afc89b.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "eng", "prob": 0.999092698097229}, {"code": "spa", "prob": 0.00018186250235885382}, {"code": "haw", "prob": 6.795905937906355e-05}, {"code": "lat", "prob": 6.20875071035698e-05}, {"code": "war", "prob": 3.527590160956606e-05}]}, "route": {"language_hint": "eng"}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 1, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "asr_summary": {"faster-whisper:tiny|det": {"language": "en", "language_probability": 1, "avg_logprob_mean": -0.5895182490348816, "no_speech_prob_mean": 0.1619873046875, "compression_ratio_mean": 0.6666666666666666}, "faster-whisper:tiny|samp_t0.2": {"language": "en", "language_probability": 1, "avg_logprob_mean": -0.5881309509277344, "no_speech_prob_mean": 0.1619873046875, "compression_ratio_mean": 0.6666666666666666}}, "ensemble": {"unique_models": ["faster-whisper:tiny"], "unique_backends": ["faster-whisper"], "warnings": ["low_model_diversity"]}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "ctc": null, "final": {"text": "intricate web of", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 1.432625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "eng", "top1_prob": 0.999092698097229, "margin": 0.9989108355948702, "confident": true}, "no_speech_prob_max": 0.1619873046875, "heuristics_asr_key": "faster-whisper:tiny|det", "avg_logprob_mean": -0.5895182490348816, "compression_ratio_mean": 0.6666666666666666, "ctc": null}}, "config": {"trim_silence": true, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null}, "timing": {"preprocess_sec": 0.20665591303259134, "lid_sec": 0.44643764197826385, "asr_sec": 1.1707691121846437, "normalize_sec": 0.00011232215911149979, "rover_sec": 0.00018567079678177834, "ctc_sec": 6.807968020439148e-07, "verify_sec": 0.000125126913189888, "total_sec": 1.824479534290731}}
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:11:class RoverDisagreement:
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:22:    disagreement_rate: float
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:23:    disagreements: list[RoverDisagreement]
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:64:        return RoverResult(text="", skeleton_key="", disagreement_rate=0.0, disagreements=[])
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:102:    disagreements: list[RoverDisagreement] = []
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:142:            disagreements.append(
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:143:                RoverDisagreement(
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:175:            disagreements.append(
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:176:                RoverDisagreement(kind="token", index=j, chosen=chosen_tok, options=dict(sorted(votes.items())))
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:191:    disagreement_rate = float(len(disagreements)) / float(max(1, total_positions))
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:196:        disagreement_rate=disagreement_rate,
/home/ubuntu/maya-transcribe/maya_transcribe/rover.py:197:        disagreements=disagreements,
/home/ubuntu/maya-transcribe/outputs/test_run/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run/audio/SPEAKER_00_0000_0.03-12.15.wav", "ffprobe": {"path": "outputs/test_run/audio/SPEAKER_00_0000_0.03-12.15.wav", "duration_sec": 1.432625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 1.432625, "rms_dbfs": -27.180731147313708, "peak_dbfs": -13.132241670622648, "clipping_fraction": 0.0}, "lid": null, "route": {"language_hint": null}, "asr": {"faster-whisper:tiny|det": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 0.9814453125, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5895182490348816, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:tiny|samp_t0.2": {"model": "faster-whisper:tiny", "text": "intricate web of", "language": "en", "language_probability": 0.9814453125, "segments": [{"start": 0.0, "end": 2.0, "text": " intricate web of", "avg_logprob": -0.5881309509277344, "no_speech_prob": 0.1619873046875, "compression_ratio": 0.6666666666666666}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}}, "normalized": {"faster-whisper:tiny|det": "intricate web of", "faster-whisper:tiny|samp_t0.2": "intricate web of"}, "rover": {"text": "intricate web of", "skeleton_key": "faster-whisper:tiny|det", "disagreement_rate": 0.0, "disagreements": []}, "verification": {"decision": "accept", "reasons": ["lid_missing"], "scores": {"duration_sec": 1.432625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": false, "top1": null, "top1_prob": null, "margin": null, "confident": false}, "no_speech_prob_max": 0.1619873046875}}, "timing": {"preprocess_sec": 0.1461551017127931, "lid_sec": 1.4016404747962952e-07, "asr_sec": 0.9981085639446974, "normalize_sec": 5.5425334721803665e-05, "rover_sec": 0.00011684978380799294, "verify_sec": 6.624497473239899e-05, "total_sec": 1.1445750738494098}}
/home/ubuntu/maya-transcribe/maya_transcribe/verify.py:48:    max_disagreement = float(thresholds.get("max_rover_disagreement_rate", 0.03))
/home/ubuntu/maya-transcribe/maya_transcribe/verify.py:87:    disagreement_rate = float(rover.get("disagreement_rate") or 0.0)
/home/ubuntu/maya-transcribe/maya_transcribe/verify.py:88:    if disagreement_rate > max_disagreement:
/home/ubuntu/maya-transcribe/maya_transcribe/verify.py:90:        reasons.append("asr_disagreement")
/home/ubuntu/maya-transcribe/maya_transcribe/verify.py:150:        "rover_disagreement_rate": disagreement_rate,
/home/ubuntu/maya-transcribe/outputs/test_run4/results.jsonl:1:{"segment_id": "SPEAKER_00_0000_0.03-12.15", "base_id": "SPEAKER_00_0000_0.03-12.15", "relpath": "SPEAKER_00_0000_0.03-12.15.flac", "uid": "01b2a08391", "src_path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "wav_path": "outputs/test_run4/audio/SPEAKER_00_0000_0.03-12.15.wav", "src_ffprobe": {"path": "data/--0C5YVdu5s/--0C5YVdu5s/segments/SPEAKER_00_0000_0.03-12.15.flac", "duration_sec": 12.11625, "sample_rate": 48000, "channels": 1}, "ffprobe": {"path": "output…12183634 tokens truncated…1817602040816, "no_speech_prob": 0.010772705078125, "compression_ratio": 1.3448275862068966}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "पर यह तरीका भी 100% दीत रूटी रहित नहीं हो सकता है", "language": "hi", "language_probability": 1, "segments": [{"start": 0.0, "end": 4.58, "text": " पर यह तरीका भी 100% दीत रूटी रहित नहीं हो सकता है", "avg_logprob": -0.06317062037331718, "no_speech_prob": 0.010772705078125, "compression_ratio": 1.3448275862068966}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "mms-ctc:mms-1b-all": {"model": "mms-ctc:mms-1b-all", "text": "यन व कन्डे ै ि लि  कमक् न् स ंर ि ं", "language": "hin", "language_probability": null, "segments": [], "meta": {"device": "cuda", "target_lang": "hin"}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.06301817602040816, "no_speech_prob_mean": 0.010772705078125, "compression_ratio_mean": 1.3448275862068966}, "faster-whisper:large-v3|samp_t0.2": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.06317062037331718, "no_speech_prob_mean": 0.010772705078125, "compression_ratio_mean": 1.3448275862068966}, "mms-ctc:mms-1b-all": {"language": "hin", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "mms-ctc:mms-1b-all"], "unique_backends": ["faster-whisper", "mms-ctc"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "पर यह तरीका भी 100 दीत रूटी रहित नहीं हो सकता है", "faster-whisper:large-v3|samp_t0.2": "पर यह तरीका भी 100 दीत रूटी रहित नहीं हो सकता है", "mms-ctc:mms-1b-all": "यन व कन्डे ै ि लि कमक् न् स ंर ि ं"}, "rover": {"text": "पर यह तरीका भी 100 दीत रूटी रहित नहीं हो सकता है", "skeleton_key": "faster-whisper:large-v3|det", "disagreement_rate": 0.48, "inputs": ["faster-whisper:large-v3|det", "mms-ctc:mms-1b-all"], "filtered_out": ["faster-whisper:large-v3|samp_t0.2"], "disagreements": [{"kind": "token", "index": 0, "chosen": "पर", "options": {"पर": 1.0, "यन": 1.0}}, {"kind": "token", "index": 1, "chosen": "यह", "options": {"यह": 1.0, "व": 1.0}}, {"kind": "token", "index": 2, "chosen": "तरीका", "options": {"कन्डे": 1.0, "तरीका": 1.0}}, {"kind": "token", "index": 3, "chosen": "भी", "options": {"भी": 1.0, "ै": 1.0}}, {"kind": "token", "index": 4, "chosen": "100", "options": {"100": 1.0, "ि": 1.0}}, {"kind": "token", "index": 5, "chosen": "दीत", "options": {"दीत": 1.0, "लि": 1.0}}, {"kind": "token", "index": 6, "chosen": "रूटी", "options": {"कमक्": 1.0, "रूटी": 1.0}}, {"kind": "token", "index": 7, "chosen": "रहित", "options": {"न्": 1.0, "रहित": 1.0}}, {"kind": "token", "index": 8, "chosen": "नहीं", "options": {"नहीं": 1.0, "स": 1.0}}, {"kind": "token", "index": 9, "chosen": "हो", "options": {"ंर": 1.0, "हो": 1.0}}, {"kind": "token", "index": 10, "chosen": "सकता", "options": {"सकता": 1.0, "ि": 1.0}}, {"kind": "token", "index": 11, "chosen": "है", "options": {"ं": 1.0, "है": 1.0}}]}, "ctc": null, "final": {"text": "पर यह तरीका भी 100 दीत रूटी रहित नहीं हो सकता है", "source": "rover"}, "verification": {"decision": "human", "reasons": ["asr_disagreement"], "scores": {"duration_sec": 4.8994375, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.48, "lid": {"available": true, "top1": "hin", "top1_prob": 0.9557303786277771, "margin": 0.9318746086210012, "confident": true}, "no_speech_prob_max": 0.010772705078125, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.06301817602040816, "compression_ratio_mean": 1.3448275862068966, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": "facebook/mms-1b-all", "indicconformer_default_model": null}, "timing": {"preprocess_sec": 0.21529207425191998, "lid_sec": 0.10205829003825784, "asr_sec": 1.9556309208273888, "normalize_sec": 0.00010017864406108856, "rover_sec": 0.000338640995323658, "ctc_sec": 1.0021030902862549e-06, "verify_sec": 7.030181586742401e-05, "total_sec": 2.2736111567355692}}
/home/ubuntu/maya-transcribe/outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/results.jsonl:199:{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__989__844424930781692-229-f.m4a-1669e33fc2fb8c47", "base_id": "Kathbath__hindi__valid__989__844424930781692-229-f.m4a", "relpath": "hin/Kathbath__hindi__valid__989__844424930781692-229-f.m4a.wav", "uid": "1669e33fc2fb8c47", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__989__844424930781692-229-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/audio/Kathbath__hindi__valid__989__844424930781692-229-f.m4a-1669e33fc2fb8c47.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__989__844424930781692-229-f.m4a.wav", "duration_sec": 4.504688, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/audio/Kathbath__hindi__valid__989__844424930781692-229-f.m4a-1669e33fc2fb8c47.wav", "duration_sec": 4.504688, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 4.5046875, "rms_dbfs": -18.611064646741994, "peak_dbfs": -2.9176317672634062, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "hin", "prob": 0.9962480664253235}, {"code": "urd", "prob": 0.00213681161403656}, {"code": "pan", "prob": 0.00040728654130361974}, {"code": "mar", "prob": 0.0002929624170064926}, {"code": "san", "prob": 0.00011687108053592965}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"faster-whisper:large-v3|det": {"model": "faster-whisper:large-v3", "text": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "language": "hi", "language_probability": 1, "segments": [{"start": 0.0, "end": 3.66, "text": " नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "avg_logprob": -0.05053711057241474, "no_speech_prob": 0.0017375946044921875, "compression_ratio": 1.4565217391304348}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है।", "language": "hi", "language_probability": 1, "segments": [{"start": 0.0, "end": 4.0, "text": " नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है।", "avg_logprob": -0.06092331230897328, "no_speech_prob": 0.0017375946044921875, "compression_ratio": 1.4891304347826086}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "mms-ctc:mms-1b-all": {"model": "mms-ctc:mms-1b-all", "text": "सांे सा ीेसि डा ीीनस ेयस िक्ह  ै ीुस न", "language": "hin", "language_probability": null, "segments": [], "meta": {"device": "cuda", "target_lang": "hin"}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.05053711057241474, "no_speech_prob_mean": 0.0017375946044921875, "compression_ratio_mean": 1.4565217391304348}, "faster-whisper:large-v3|samp_t0.2": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.06092331230897328, "no_speech_prob_mean": 0.0017375946044921875, "compression_ratio_mean": 1.4891304347826086}, "mms-ctc:mms-1b-all": {"language": "hin", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "mms-ctc:mms-1b-all"], "unique_backends": ["faster-whisper", "mms-ctc"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "faster-whisper:large-v3|samp_t0.2": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "mms-ctc:mms-1b-all": "सांे सा ीेसि डा ीीनस ेयस िक्ह ै ीुस न"}, "rover": {"text": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "skeleton_key": "faster-whisper:large-v3|det", "disagreement_rate": 0.4, "inputs": ["faster-whisper:large-v3|det", "mms-ctc:mms-1b-all"], "filtered_out": ["faster-whisper:large-v3|samp_t0.2"], "disagreements": [{"kind": "token", "index": 2, "chosen": "डांस", "options": {"डांस": 1.0, "सांे": 1.0}}, {"kind": "token", "index": 3, "chosen": "के", "options": {"के": 1.0, "सा": 1.0}}, {"kind": "token", "index": 4, "chosen": "दोरान", "options": {"दोरान": 1.0, "ीेसि": 1.0}}, {"kind": "token", "index": 5, "chosen": "अपने", "options": {"अपने": 1.0, "डा": 1.0}}, {"kind": "token", "index": 6, "chosen": "स्टाइल", "options": {"स्टाइल": 1.0, "ीीनस": 1.0}}, {"kind": "token", "index": 7, "chosen": "का", "options": {"का": 1.0, "ेयस": 1.0}}, {"kind": "token", "index": 8, "chosen": "भी", "options": {"भी": 1.0, "िक्ह": 1.0}}, {"kind": "token", "index": 9, "chosen": "ध्यान", "options": {"ध्यान": 1.0, "ै": 1.0}}, {"kind": "token", "index": 10, "chosen": "रखा", "options": {"रखा": 1.0, "ीुस": 1.0}}, {"kind": "token", "index": 11, "chosen": "है", "options": {"न": 1.0, "है": 1.0}}]}, "ctc": null, "final": {"text": "नेहा ने डांस के दोरान अपने स्टाइल का भी ध्यान रखा है", "source": "rover"}, "verification": {"decision": "human", "reasons": ["asr_disagreement"], "scores": {"duration_sec": 4.5046875, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.4, "lid": {"available": true, "top1": "hin", "top1_prob": 0.9962480664253235, "margin": 0.9941112548112869, "confident": true}, "no_speech_prob_max": 0.0017375946044921875, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.05053711057241474, "compression_ratio_mean": 1.4565217391304348, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": "facebook/mms-1b-all", "indicconformer_default_model": null}, "timing": {"preprocess_sec": 0.21335698384791613, "lid_sec": 0.10158387292176485, "asr_sec": 1.9972986630164087, "normalize_sec": 9.200302883982658e-05, "rover_sec": 0.0002902490086853504, "ctc_sec": 9.82079654932022e-07, "verify_sec": 6.426079198718071e-05, "total_sec": 2.312777797691524}}
/home/ubuntu/maya-transcribe/outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/results.jsonl:200:{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__996__844424932834336-727-f.m4a-eaec03309e81d9f6", "base_id": "Kathbath__hindi__valid__996__844424932834336-727-f.m4a", "relpath": "hin/Kathbath__hindi__valid__996__844424932834336-727-f.m4a.wav", "uid": "eaec03309e81d9f6", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__996__844424932834336-727-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/audio/Kathbath__hindi__valid__996__844424932834336-727-f.m4a-eaec03309e81d9f6.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__996__844424932834336-727-f.m4a.wav", "duration_sec": 3.87775, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/fw_largev3_mms_no_ctc/audio/Kathbath__hindi__valid__996__844424932834336-727-f.m4a-eaec03309e81d9f6.wav", "duration_sec": 3.87775, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 3.87775, "rms_dbfs": -22.769696186916786, "peak_dbfs": -5.109002797556616, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "pan", "prob": 0.7290913462638855}, {"code": "hin", "prob": 0.188624769449234}, {"code": "guj", "prob": 0.06296700239181519}, {"code": "urd", "prob": 0.005503176245838404}, {"code": "mar", "prob": 0.005224794149398804}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"faster-whisper:large-v3|det": {"model": "faster-whisper:large-v3", "text": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "language": "hi", "language_probability": 1, "segments": [{"start": 0.0, "end": 3.66, "text": " उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "avg_logprob": -0.0760869556579037, "no_speech_prob": 0.029449462890625, "compression_ratio": 1.6346153846153846}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 5, "temperature": 0.0, "vad_filter": false}}, "faster-whisper:large-v3|samp_t0.2": {"model": "faster-whisper:large-v3", "text": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "language": "hi", "language_probability": 1, "segments": [{"start": 0.0, "end": 3.66, "text": " उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "avg_logprob": -0.07607306345649388, "no_speech_prob": 0.029449462890625, "compression_ratio": 1.6346153846153846}], "meta": {"device": "cuda", "compute_type": "float16", "beam_size": 5, "best_of": 10, "temperature": 0.2, "vad_filter": false}}, "mms-ctc:mms-1b-all": {"model": "mms-ctc:mms-1b-all", "text": "मसस ब ड्  िव द् स ेयस स् ड् ैन िनअस डन क", "language": "hin", "language_probability": null, "segments": [], "meta": {"device": "cuda", "target_lang": "hin"}}}, "asr_summary": {"faster-whisper:large-v3|det": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.0760869556579037, "no_speech_prob_mean": 0.029449462890625, "compression_ratio_mean": 1.6346153846153846}, "faster-whisper:large-v3|samp_t0.2": {"language": "hi", "language_probability": 1, "avg_logprob_mean": -0.07607306345649388, "no_speech_prob_mean": 0.029449462890625, "compression_ratio_mean": 1.6346153846153846}, "mms-ctc:mms-1b-all": {"language": "hin", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["faster-whisper:large-v3", "mms-ctc:mms-1b-all"], "unique_backends": ["faster-whisper", "mms-ctc"], "warnings": []}, "normalized": {"faster-whisper:large-v3|det": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "faster-whisper:large-v3|samp_t0.2": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "mms-ctc:mms-1b-all": "मसस ब ड् िव द् स ेयस स् ड् ैन िनअस डन क"}, "rover": {"text": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "skeleton_key": "faster-whisper:large-v3|det", "disagreement_rate": 0.41935483870967744, "inputs": ["faster-whisper:large-v3|det", "mms-ctc:mms-1b-all"], "filtered_out": ["faster-whisper:large-v3|samp_t0.2"], "disagreements": [{"kind": "token", "index": 2, "chosen": "कि", "options": {"कि": 1.0, "मसस": 1.0}}, {"kind": "token", "index": 3, "chosen": "हाँ", "options": {"ब": 1.0, "हाँ": 1.0}}, {"kind": "token", "index": 4, "chosen": "श्री", "options": {"ड्": 1.0, "श्री": 1.0}}, {"kind": "token", "index": 5, "chosen": "देवी", "options": {"देवी": 1.0, "िव": 1.0}}, {"kind": "token", "index": 6, "chosen": "ने", "options": {"द्": 1.0, "ने": 1.0}}, {"kind": "token", "index": 7, "chosen": "अपनी", "options": {"अपनी": 1.0, "स": 1.0}}, {"kind": "token", "index": 8, "chosen": "नाक", "options": {"नाक": 1.0, "ेयस": 1.0}}, {"kind": "token", "index": 9, "chosen": "की", "options": {"की": 1.0, "स्": 1.0}}, {"kind": "token", "index": 10, "chosen": "कई", "options": {"कई": 1.0, "ड्": 1.0}}, {"kind": "token", "index": 11, "chosen": "बार", "options": {"बार": 1.0, "ैन": 1.0}}, {"kind": "token", "index": 12, "chosen": "सर्जरी", "options": {"सर्जरी": 1.0, "िनअस": 1.0}}, {"kind": "token", "index": 13, "chosen": "कराई", "options": {"कराई": 1.0, "डन": 1.0}}, {"kind": "token", "index": 14, "chosen": "थी", "options": {"क": 1.0, "थी": 1.0}}]}, "ctc": null, "final": {"text": "उन्होंने कहा कि हाँ श्री देवी ने अपनी नाक की कई बार सर्जरी कराई थी", "source": "rover"}, "verification": {"decision": "human", "reasons": ["lid_uncertain", "asr_disagreement"], "scores": {"duration_sec": 3.87775, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.41935483870967744, "lid": {"available": true, "top1": "pan", "top1_prob": 0.7290913462638855, "margin": 0.5404665768146515, "confident": false}, "no_speech_prob_max": 0.029449462890625, "heuristics_asr_key": "faster-whisper:large-v3|det", "avg_logprob_mean": -0.0760869556579037, "compression_ratio_mean": 1.6346153846153846, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": "facebook/mms-1b-all", "indicconformer_default_model": null}, "timing": {"preprocess_sec": 0.2086212681606412, "lid_sec": 0.10120604885742068, "asr_sec": 2.2244403529912233, "normalize_sec": 0.00010214420035481453, "rover_sec": 0.00043279770761728287, "ctc_sec": 1.5022233128547668e-06, "verify_sec": 8.771615102887154e-05, "total_sec": 2.5350025398656726}}
/home/ubuntu/maya-transcribe/data/bench/model_maps/thresholds_script_filter_ctc_relaxed.json:8:  "max_rover_disagreement_rate": 0.15


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Exploring threshold file**

I'm opening the threshold file in data/bench/model_maps to understand its contents and relevance.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "filter_hypotheses_by_script": true,
  "require_ctc": false,
  "min_ctc_margin": 0.05,
  "min_ctc_token_coverage": 0.0,
  "min_ctc_token_score_mean": -999.0,
  "fail_on_ctc_span_mismatch": false,
  "max_rover_disagreement_rate": 0.15
}


Summary: auto


**Checking for strict threshold file**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
thresholds_ctc_advisory.json
thresholds_script_filter_ctc_advisory.json
thresholds_script_filter_ctc_relaxed.json
whisper_vaani_map.json


**Checking strict mode thresholds**


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "filter_hypotheses_by_script": true,
  "require_ctc": false,
  "min_ctc_margin": 0.05,
  "min_ctc_token_coverage": 0.0,
  "min_ctc_token_score_mean": -999.0,
  "fail_on_ctc_span_mismatch": false
}


**Aligning thresholds and routing for evaluation**

To replicate previous relaxed-threshold results, I'll create a thresholds JSON including relaxed disagreement rates and enabling route_lang_from_path and route_lang_prefer_path, matching bench behavior within the run pipeline.


**Planning run and eval with gemini integration**

I'll run the pipeline with the gemini model on the bench dataset using a thresholds file including route settings, then evaluate with language_source set to "refs" to match prior results. Next, I'll verify how gemini is triggered in the pipeline, especially how "hard" segments are detected.


rg -n "gemini" /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
74:    gemini_model: str | None = None,
75:    gemini_prompt_path: Path | None = None,
76:    gemini_max_output_tokens: int = 512,
77:    gemini_hard_only: bool = True,
156:    gemini_backend = None
157:    if gemini_model:
159:            from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template
161:            prompt = load_prompt_template(gemini_prompt_path or Path("Prompt.md"))
162:            gemini_backend = GeminiASR(
163:                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
169:    default_whisper_run_sampling = not bool(transformers_backends or transformers_router or gemini_backend is not None)
303:                if gemini_backend is not None:
304:                    run_gemini = True
305:                    if gemini_hard_only:
311:                        run_gemini = bool(
321:                    if run_gemini:
323:                            g = gemini_backend.transcribe(wav_path, language=route_lang)
496:                        "gemini_model": gemini_model,
497:                        "gemini_hard_only": gemini_hard_only,


sed -n '280,360p' /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                # MMS ASR for CTC-based diversity (per master.md plan).
                if mms_backend is not None:
                    try:
                        mms_res = mms_backend.transcribe(wav_path, language=route_lang)
                        asr_results[mms_res.model] = mms_res
                    except Exception as e:
                        asr_errors[f"mms-ctc:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                # IndicConformer for Conformer architecture diversity (per master.md plan).
                if indicconformer_router is not None:
                    try:
                        conformer_res = indicconformer_router.transcribe(wav_path, language=route_lang)
                        if conformer_res is not None:
                            asr_results[conformer_res.model] = conformer_res
                    except Exception as e:
                        asr_errors[f"indicconformer:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if primary_asr_key is None and asr_results:
                    primary_asr_key = sorted(asr_results.keys())[0]
                if not asr_results:
                    raise RuntimeError(f"no_asr_hypotheses:{asr_errors or 'unknown'}")

                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
                if gemini_backend is not None:
                    run_gemini = True
                    if gemini_hard_only:
                        base_for_heur = det or (next(iter(asr_results.values())) if asr_results else None)
                        det_sum = _asr_summary(base_for_heur) if base_for_heur is not None else {}
                        avg_lp = det_sum.get("avg_logprob_mean")
                        comp = det_sum.get("compression_ratio_mean")
                        no_speech = det_sum.get("no_speech_prob_mean")
                        run_gemini = bool(
                            route_lang is None
                            or (no_speech is not None and float(no_speech) > 0.5)
                            or (avg_lp is not None and float(avg_lp) < float(thresholds.get("min_avg_logprob_mean", -1.0)))
                            or (
                                comp is not None
                                and float(comp) > float(thresholds.get("max_compression_ratio_mean", 2.4))
                            )
                        )

                    if run_gemini:
                        try:
                            g = gemini_backend.transcribe(wav_path, language=route_lang)
                            asr_results[g.model] = g
                        except Exception:
                            pass

                t_asr1 = time.perf_counter()

                ensemble_models = sorted({v.model for v in asr_results.values()})
                ensemble_backends = sorted({v.model.split(":", 1)[0] for v in asr_results.values()})
                ensemble_warnings: list[str] = []
                if len(ensemble_models) < 2:
                    ensemble_warnings.append("low_model_diversity")

                # 4) Normalize hypotheses
                t_norm0 = time.perf_counter()
                norm_hyps: dict[str, str] = {}
                for k, res in asr_results.items():
                    lang_hint = res.language or route_lang
                    norm_hyps[k] = normalize_text(res.text, lang=lang_hint)
                t_norm1 = time.perf_counter()

                # 5) ROVER
                t_rover0 = time.perf_counter()
                rover_hyps = dict(norm_hyps)
                rover_filtered_out: list[str] = []
                if bool(thresholds.get("filter_hypotheses_by_script", False)) and route_lang:
                    from maya_transcribe.language import dominant_script, expected_script

                    expected = expected_script(route_lang)
                    if expected:
                        kept: dict[str, str] = {}
                        for k, res in asr_results.items():
                            script = dominant_script(res.text)
                            if script is None or script == expected:
                                kept[k] = norm_hyps.get(k, "")
                            else:
                                rover_filtered_out.append(k)


**Clarifying Gemini evaluation keys**

Discovered Gemini results use model keys like `gemini:gemini-2.0-flash` under `asr`, which may complicate specifying hypotheses for evaluation due to colon usage; next, I'll verify how the eval script parses `asr:` keys to correctly evaluate Gemini-only WER.


sed -n '1,140p' /home/ubuntu/maya-transcribe/maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from jiwer import cer, wer

from maya_transcribe.io import read_jsonl
from maya_transcribe.textnorm import normalize_text

@dataclass(frozen=True)
class EvalRow:
    segment_id: str
    language: str | None
    decision: str | None
    duration_sec: float | None
    duration_bucket: str | None
    ref: str
    hyp: str
    wer: float
    cer: float
    oracle_best_wer: float | None = None
    oracle_best_cer: float | None = None
    oracle_best_key: str | None = None
    selection_gap_wer: float | None = None

def load_refs(refs_path: Path) -> dict[str, dict[str, Any]]:
    """
    Read a JSONL file of references.
    Expected keys (any one): `text`, `transcript`, `reference`.
    Required key: `segment_id` (or `id`).
    """
    refs: dict[str, dict[str, Any]] = {}
    for row in read_jsonl(refs_path):
        segment_id = str(row.get("segment_id") or row.get("id") or "").strip()
        if not segment_id:
            continue
        refs[segment_id] = row
    return refs

def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    for k in keys:
        v = row.get(k)
        if isinstance(v, str) and v.strip():
            return v
    return None

def evaluate(
    results_path: Path,
    refs_path: Path,
    out_path: Path | None = None,
    hypothesis: str = "final",
    oracle: bool = True,
    language_source: str = "route",
    decision_filter: list[str] | None = None,
) -> dict[str, Any]:
    refs = load_refs(refs_path)
    results = read_jsonl(results_path)

    decision_filter_set = None
    if decision_filter:
        decision_filter_set = {str(d).strip().lower() for d in decision_filter if str(d).strip()}

    rows: list[EvalRow] = []
    for r in results:
        segment_id = str(r.get("segment_id") or "").strip()
        if not segment_id or segment_id not in refs:
            continue

        ref_row = refs[segment_id]
        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
        if ref_text is None:
            continue

        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
        if hyp_text is None:
            continue

        route_lang = None
        route = r.get("route") or {}
        if isinstance(route, dict):
            route_lang = route.get("language_hint")

        ref_lang = None
        if language_source == "refs":
            v = ref_row.get("lang") or ref_row.get("language")
            if isinstance(v, str) and v.strip():
                ref_lang = v.strip()
        elif language_source != "route":
            raise ValueError("language_source must be 'route' or 'refs'")

        lang = ref_lang or route_lang

        decision = None
        ver = r.get("verification") or {}
        if isinstance(ver, dict):
            decision = ver.get("decision")
        if decision_filter_set is not None:
            dec_norm = str(decision or "").strip().lower()
            if dec_norm not in decision_filter_set:
                continue

        duration_sec = None
        qc = r.get("qc") or {}
        if isinstance(qc, dict):
            try:
                duration_sec = float(qc.get("duration_sec")) if qc.get("duration_sec") is not None else None
            except (TypeError, ValueError):
                duration_sec = None
        bucket = duration_bucket(duration_sec)

        ref_norm = normalize_text(ref_text, lang=lang)
        hyp_norm = normalize_text(hyp_text, lang=lang)
        w = float(wer(ref_norm, hyp_norm))
        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))

        oracle_best_wer = None
        oracle_best_cer = None
        oracle_best_key = None
        selection_gap_wer = None

        if oracle:
            cand_wer: dict[str, float] = {}
            cand_cer: dict[str, float] = {}
            for key, text in extract_all_candidates(result_row=r).items():
                ref_n = ref_norm
                hyp_n = normalize_text(text, lang=lang)
                cand_wer[key] = float(wer(ref_n, hyp_n))
                cand_cer[key] = float(cer(ref_n.replace(" ", ""), hyp_n.replace(" ", "")))

            if cand_wer:
                oracle_best_key, oracle_best_wer = min(cand_wer.items(), key=lambda kv: kv[1])
                oracle_best_cer = cand_cer.get(oracle_best_key)
                selection_gap_wer = float(w) - float(oracle_best_wer)


**Checking extraction functions for ASR key**


rg -n "extract_hypothesis_text|extract_all_candidates" -n /home/ubuntu/maya-transcribe/maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
81:        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
131:            for key, text in extract_all_candidates(result_row=r).items():
218:def extract_hypothesis_text(result_row: dict[str, Any], hypothesis: str) -> str | None:
249:def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:
256:    final = extract_hypothesis_text(result_row, "final")
260:    rover = extract_hypothesis_text(result_row, "rover")


sed -n '200,280p' /home/ubuntu/maya-transcribe/maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        "cer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang_cer.items())},
        "cer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec_cer.items())},
        "cer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket_cer.items())},
        "oracle": {
            "enabled": bool(oracle),
            "oracle_best_wer_mean": (sum(oracle_best_wers) / len(oracle_best_wers)) if oracle_best_wers else None,
            "selection_gap_wer_mean": (sum(selection_gaps) / len(selection_gaps)) if selection_gaps else None,
            "selection_gap_wer_p90": _percentile(selection_gaps, 0.90),
        },
    }

    if out_path is not None:
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")

    return summary

def extract_hypothesis_text(result_row: dict[str, Any], hypothesis: str) -> str | None:
    """
    hypothesis:
      - "final": uses result_row["final"]["text"] (falls back to rover if missing)
      - "rover" (default): uses result_row["rover"]["text"]
      - "asr:<key>": uses result_row["asr"][<key>]["text"]
    """
    hypothesis = (hypothesis or "rover").strip()
    if hypothesis == "final":
        final = result_row.get("final") or {}
        if isinstance(final, dict) and isinstance(final.get("text"), str):
            return final["text"]
        hypothesis = "rover"
    if hypothesis == "rover":
        rover = result_row.get("rover") or {}
        if isinstance(rover, dict) and isinstance(rover.get("text"), str):
            return rover["text"]
        return None

    if hypothesis.startswith("asr:"):
        key = hypothesis[len("asr:") :].strip()
        asr = result_row.get("asr") or {}
        if isinstance(asr, dict):
            entry = asr.get(key) or {}
            if isinstance(entry, dict) and isinstance(entry.get("text"), str):
                return entry["text"]
        return None

    return None

def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:
    """
    Collect all text candidates available in a pipeline result row.
    Keys are stable and namespaced (final/rover/asr:<key>).
    """
    candidates: dict[str, str] = {}

    final = extract_hypothesis_text(result_row, "final")
    if final is not None:
        candidates["final"] = final

    rover = extract_hypothesis_text(result_row, "rover")
    if rover is not None:
        candidates["rover"] = rover

    asr = result_row.get("asr") or {}
    if isinstance(asr, dict):
        for key, entry in asr.items():
            if not isinstance(key, str):
                continue
            if not isinstance(entry, dict):
                continue
            text = entry.get("text")
            if isinstance(text, str) and text.strip():
                candidates[f"asr:{key}"] = text

    # Also include normalized hypotheses if present (useful for debugging).
    normalized = result_row.get("normalized") or {}
    if isinstance(normalized, dict):
        for key, text in normalized.items():
            if not isinstance(key, str) or not isinstance(text, str):
                continue


**Clarifying ASR model keys and segment handling**

Gemini ASR requires the full `--hypothesis` key with colons, which is supported by quoting; it only runs on hard segments, so evaluation excludes others unless forced to run on all segments, which is costly but planned for 200 segments. I'll also clarify the ambiguous "Gemma3n" input with the user.


**Planning benchmark runs with Gemini and baseline models**

Planning to run benchmarks on the maya_hf_hin_v2 dataset using the fast-whisper large-v3 baseline plus Vaani and Gemini as extra ASRs, matching previous threshold and filtering settings for comparable WER results. I'll verify existing run configs from summary files before executing and hold off on Voxtral integration due to API uncertainty.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "segments_total": 200,
  "counts": {
    "accept": 81,
    "human": 119,
    "reject": 0,
    "error": 0
  },
  "avg_total_sec": 3.6557701411796732,
  "p50_total_sec": 3.5481748580932617,
  "p90_total_sec": 4.488278137985617
}


**Inspecting results file for config**

I'll check the first line of the results.jsonl file to identify config fields missing from the summary.


python - <<'PY'
import json
from pathlib import Path
p=Path('/home/ubuntu/maya-transcribe/outputs/bench/maya_hf_hin_v2/fw_largev3_vaani_relaxed/results.jsonl')
with p.open() as f:
    line=f.readline()
obj=json.loads(line)
print(obj.get('config'))
PY


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


**Noticing Python command ambiguity**


python3 - <<'PY'
import json
from pathlib import Path
p=Path('/home/ubuntu/maya-transcribe/outputs/bench/maya_hf_hin_v2/fw_largev3_vaani_relaxed/results.jsonl')
with p.open() as f:
    line=f.readline()
obj=json.loads(line)
print(obj.get('config'))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{'trim_silence': False, 'silence_threshold_db': -40.0, 'min_silence_sec': 0.15, 'gemini_model': None, 'gemini_hard_only': True, 'ctc_model': 'facebook/mms-1b-all', 'mms_asr_model': None, 'indicconformer_default_model': None}


Summary: auto


**Analyzing model config and usage**

I confirmed the `ctc_model` is set to an MMS CTC-based model without an MMS ASR model, suggesting MMS is used solely for CTC scoring. The previous run likely used a Vaani Whisper model mapping; I'll inspect output files with Python to verify how Vaani was configured and used.


python3 - <<'PY'
import json
from pathlib import Path
p=Path('/home/ubuntu/maya-transcribe/outputs/bench/maya_hf_hin_v2/fw_largev3_vaani_relaxed/results.jsonl')
with p.open() as f:
    obj=json.loads(f.readline())
print(obj['asr'].keys())
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
dict_keys(['faster-whisper:large-v3|det', 'transformers:ARTPARK-IISc/whisper-large-v3-vaani-hindi'])


**Planning threshold config for replication**

I'm deciding to replicate the original setup by creating or merging a threshold JSON that includes route and CTC settings, using `trim_silence` false, and possibly adapting `bench run` or running gemini separately with `maya-transcribe run` to match the model pipeline accurately.


**Finalizing thresholds and run CLI options**

I'm preparing a new thresholds file merging relaxed CTC with routing settings and confirming CLI flags for gemini usage, trim silence false, and deterministic sampling. Next, I'll run `maya-transcribe run` with matching models and evaluate outputs with gemini enabled on all segments for comprehensive WER comparison, ensuring dependencies like `google-genai` are installed.


Output:
Exit code: 0
Wall time: 1.6 seconds
Output:
Name: google-genai
Version: 1.54.0
Summary: GenAI Python SDK
Home-page: https://github.com/googleapis/python-genai
Author:
Author-email: Google LLC <googleapis-packages@google.com>
License-Expression: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: anyio, google-auth, httpx, pydantic, requests, tenacity, typing-extensions, websockets
Required-by:


**Creating thresholds JSON file**

I'm preparing to create a new thresholds configuration file under `data/bench/model_maps` using a heredoc write approach.


cat <<'EOF' > /home/ubuntu/maya-transcribe/data/bench/model_maps/thresholds_script_filter_ctc_relaxed_route.json
{
  "route_lang_from_path": true,
  "route_lang_prefer_path": true,
  "filter_hypotheses_by_script": true,
  "require_ctc": false,
  "min_ctc_margin": 0.05,
  "min_ctc_token_coverage": 0.0,
  "min_ctc_token_score_mean": -999.0,
  "fail_on_ctc_span_mismatch": false,
  "max_rover_disagreement_rate": 0.15
}
EOF


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Running transcription pipeline with Gemini**


maya-transcribe run --segments-dir data/bench/maya_hf_hin_v2/audio --out outputs/bench/maya_hf_hin_v2/fw_largev3_vaani_gemini_relaxed --whisper-model large-v3 --transformers-whisper-model-map data/bench/model_maps/whisper_vaani_map.json --ctc-model facebook/mms-1b-all --no-trim-silence --gemini-model gemini-2.0-flash --no-gemini-hard-only --thresholds-json data/bench/model_maps/thresholds_script_filter_ctc_relaxed_route.json


Output:
Wall time: 475.7 seconds
aborted by user


benchmarks are for Gemma3n and voxtral. not whisper and gemini.


<permissions instructions>Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `workspace-write`: The sandbox permits reading files, and editing files in `cwd` and `writable_roots`. Editing files in other directories requires approval. Network access is restricted. Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `on-request`: Commands will be run in the sandbox by default, and you can specify in your tool call if you want to escalate a command to run without sandboxing. If the completing the task requires escalated permissions, Do not let these settings or the sandbox deter you from attempting to accomplish the user's task.

Here are scenarios where you'll need to request approval:
- You need to run a command that writes to a directory that requires it (e.g. running tests that write to /var)
- You need to run a GUI app (e.g., open/xdg-open/osascript) to open browsers or files.
- You are running sandboxed and need to run a command that requires network access (e.g. installing packages)
- If you run a command that is important to solving the user's query, but it fails because of sandboxing, rerun the command with approval. ALWAYS proceed to use the `sandbox_permissions` and `justification` parameters - do not message the user before requesting approval for the command.
- You are about to take a potentially destructive action such as an `rm` or `git reset` that the user did not explicitly ask for.

When requesting approval to execute a command that will require escalated privileges:
  - Provide the `sandbox_permissions` parameter with the value `"require_escalated"`
  - Include a short, 1 sentence explanation for why you need escalated permissions in the justification parameter The writable roots are `/home/ubuntu/maya-transcribe`, `/tmp`.</permissions instructions>


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <shell>bash</shell>
</environment_context>


**Clarifying missing Voxtral/Gemma3n integrations**

The repo has no built-in support for Voxtral or Gemma3n benchmarks, so I'll explain that we can't run them natively yet. I'll ask the user how they want to proceed—whether to integrate these backends into the pipeline or run external scripts—and request details on models, API keys, or paths for setup.


**Requesting model access and benchmark details**

I’ll explain that Voxtral and Gemma3n aren’t yet integrated for benchmarking in the repo, so I can’t run tests immediately. I’ll ask the user to provide model IDs or API access info for both, confirm whether Gemma3n supports audio transcription, and specify which benchmark dataset and thresholds to use, so I can plan integration and evaluation accordingly.


Got it — I’ll benchmark **Gemma3n** and **Voxtral** only. The repo doesn’t have backends for those yet (it currently supports Whisper/MMS/IndicConformer and Gemini as the only audio‑LLM), so I can’t run the benchmark until we wire them in.

To proceed, I need a few specifics:
- Voxtral access: API vs local weights? If API, which provider/SDK and env var name; if local, the exact HF model ID.
- Gemma3n: confirm it’s **audio‑capable** (not text‑only), and provide the exact model ID or API endpoint.
- Benchmark scope: run on `data/bench/maya_hf_hin_v2` (200 Hindi clips, matches your recent summary) or the full multi‑language set `data/bench/maya_hf_v2`.
- Thresholds to mirror your relaxed run: should I reuse `data/bench/model_maps/thresholds_script_filter_ctc_relaxed_route.json`?

Once you confirm those, I’ll add the backends, run the benchmark, and report WER/CER + accept/human rates side‑by‑side with the recent numbers.