**Reviewing inserted function snippet**


sed -n '40,140p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    avg_logprobs = [s.avg_logprob for s in result.segments if s.avg_logprob is not None]
    no_speech_probs = [s.no_speech_prob for s in result.segments if s.no_speech_prob is not None]
    compression_ratios = [s.compression_ratio for s in result.segments if s.compression_ratio is not None]

    def mean(xs: list[float]) -> float | None:
        return (sum(xs) / len(xs)) if xs else None

    return {
        "language": result.language,
        "language_probability": result.language_probability,
        "avg_logprob_mean": mean([float(x) for x in avg_logprobs]),
        "no_speech_prob_mean": mean([float(x) for x in no_speech_probs]),
        "compression_ratio_mean": mean([float(x) for x in compression_ratios]),
    }

def _infer_lang_from_relpath(relpath: str) -> str | None:
    parts = [p.strip().lower() for p in (relpath or "").split("/") if p.strip()]
    for p in parts[:-1]:
        if p in LANG_CODE_TO_NAME:
            return p
    return None

def _word_count(text: str, lang: str | None) -> int:
    norm = normalize_text(text or "", lang=lang)
    if not norm.strip():
        return 0
    return len(norm.split())

def _guard_llm_output(
    *,
    anchor_text: str,
    candidate_text: str,
    lang: str | None,
    thresholds: dict[str, Any],
) -> dict[str, Any]:
    info: dict[str, Any] = {
        "accept": True,
        "reason": None,
        "expected_script": None,
        "script_ratio": None,
        "length_ratio": None,
        "anchor_wer": None,
        "anchor_word_count": None,
        "candidate_word_count": None,
    }

    candidate_text = str(candidate_text or "")
    anchor_text = str(anchor_text or "")
    if not candidate_text.strip():
        info["accept"] = False
        info["reason"] = "empty"
        return info

    expected = expected_script(lang)
    info["expected_script"] = expected
    min_letters = int(thresholds.get("vllm_guard_min_letters", 4))
    if expected:
        ratio = script_ratio(candidate_text, expected, min_letters=min_letters)
        info["script_ratio"] = ratio
        min_ratio = float(thresholds.get("vllm_guard_script_ratio_min", 0.6))
        if ratio is not None and ratio < min_ratio:
            info["accept"] = False
            info["reason"] = "script_ratio"
            return info

    anchor_norm = normalize_text(anchor_text, lang=lang)
    cand_norm = normalize_text(candidate_text, lang=lang)
    anchor_words = _word_count(anchor_norm, lang)
    cand_words = _word_count(cand_norm, lang)
    info["anchor_word_count"] = anchor_words
    info["candidate_word_count"] = cand_words

    length_ratio = cand_words / float(max(anchor_words, 1))
    info["length_ratio"] = length_ratio
    min_len = float(thresholds.get("vllm_guard_length_ratio_min", 0.7))
    max_len = float(thresholds.get("vllm_guard_length_ratio_max", 1.3))
    if length_ratio < min_len or length_ratio > max_len:
        info["accept"] = False
        info["reason"] = "length_ratio"
        return info

    if anchor_norm.strip():
        if jiwer_wer is not None:
            anchor_wer = float(jiwer_wer(anchor_norm, cand_norm))
        else:
            from difflib import SequenceMatcher

            anchor_wer = 1.0 - SequenceMatcher(None, anchor_norm, cand_norm).ratio()
        info["anchor_wer"] = anchor_wer
        max_wer = float(thresholds.get("vllm_guard_anchor_wer_max", 0.2))
        if anchor_wer > max_wer:
            info["accept"] = False
            info["reason"] = "anchor_wer"
            return info

    return info


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Summary: auto


**Refining text normalization to avoid redundancy**

Plan to update the word count calls to use original texts rather than already normalized ones to prevent double normalization and ensure consistency, making the code cleaner without changing functionality.


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')
text = text.replace('    anchor_norm = normalize_text(anchor_text, lang=lang)\n    cand_norm = normalize_text(candidate_text, lang=lang)\n    anchor_words = _word_count(anchor_norm, lang)\n    cand_words = _word_count(cand_norm, lang)\n',
                   '    anchor_norm = normalize_text(anchor_text, lang=lang)\n    cand_norm = normalize_text(candidate_text, lang=lang)\n    anchor_words = _word_count(anchor_text, lang)\n    cand_words = _word_count(candidate_text, lang)\n')
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Adjusting strict prompt insertion in pipeline**

Planning to insert logic in run_pipeline right after thresholds are set to enable strict prompts conditionally, ensuring these prompts are assigned before the VLLM backend runs and uses them.


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')

needle = '    if (ctc_model or ctc_model_map) and "min_ctc_margin" not in thresholds:\n        thresholds["min_ctc_margin"] = 0.05\n'
if needle not in text:
    raise SystemExit('thresholds block not found')

insert = needle + '\n    use_strict_prompt = bool(thresholds.get("vllm_strict_prompt", False))\n    if use_strict_prompt and not vllm_transcribe_prompt:\n        vllm_transcribe_prompt = STRICT_VLLM_TRANSCRIBE_PROMPT\n\n    strict_proofread = bool(thresholds.get("vllm_strict_proofread_prompt", use_strict_prompt))\n    if strict_proofread and not vllm_proofread_prompt:\n        vllm_proofread_prompt = STRICT_VLLM_PROOFREAD_PROMPT\n'

text = text.replace(needle, insert)
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Planning vllm guard variable initialization**

Deciding to declare `vllm_guard` and `vllm_guard_final` as None before the vllm block alongside existing variables like `det` and `router_key` to prepare for enhanced proofread and guard support. Next, I'll add these declarations early in the segment.


**Designing comprehensive vllm block replacement with guard logic**

Mapped out a detailed plan to replace the existing vllm block with a new one supporting anchor-based proofread, guard checks, and optional raw transcript skipping. I clarified variable initialization, conditional branches for anchor proofread usage, fallback to old logic, and guard processing for multiple candidate outputs. Next, I'll implement this replacement string carefully with correct indentation for precise substitution.


python3 - <<'PY'
from pathlib import Path

path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')

start = text.find('                if vllm_backend is not None:')
end = text.find('                # MMS ASR for CTC-based diversity', start)
if start == -1 or end == -1:
    raise SystemExit('Could not locate vllm block to replace')

new_block = '''                if vllm_backend is not None:
                    anchor_text = det.text if det is not None else None
                    anchor_key = primary_asr_key
                    guard_enabled = bool(thresholds.get("vllm_guard_enable", False))
                    guard_override = bool(thresholds.get("vllm_guard_override_final", False))
                    proofread_use_anchor = bool(thresholds.get("vllm_proofread_use_anchor", False))
                    skip_raw_if_anchor = bool(thresholds.get("vllm_skip_raw_if_anchor", False))

                    if proofread_use_anchor and anchor_text:
                        prompt_tpl = vllm_proofread_prompt or DEFAULT_VLLM_PROOFREAD_PROMPT
                        proof_prompt = prompt_tpl.format(transcript=anchor_text)
                        try:
                            mode = "proofread" if vllm_backend.supports_proofread else "transcribe"
                            proof_res = vllm_backend.transcribe(
                                wav_path, language=route_lang, prompt=proof_prompt, mode=mode
                            )
                            proof_key = f"{proof_res.model}|proofread_anchor"
                            asr_results[proof_key] = replace(proof_res, model=proof_key)
                            if primary_asr_key is None:
                                primary_asr_key = proof_key
                            if guard_enabled:
                                vllm_guard = _guard_llm_output(
                                    anchor_text=anchor_text,
                                    candidate_text=proof_res.text,
                                    lang=route_lang,
                                    thresholds=thresholds,
                                )
                                vllm_guard["candidate_key"] = proof_key
                                vllm_guard["anchor_key"] = anchor_key
                                if guard_override:
                                    if vllm_guard.get("accept"):
                                        vllm_guard_final = {"text": proof_res.text, "source": f"asr:{proof_key}"}
                                    else:
                                        fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                        vllm_guard_final = {"text": anchor_text, "source": fallback_source}
                        except Exception as e:
                            asr_errors[f"{vllm_backend.name}|proofread"] = f"{type(e).__name__}:{e}"

                    if not (proofread_use_anchor and anchor_text and skip_raw_if_anchor):
                        try:
                            vllm_res = vllm_backend.transcribe(
                                wav_path, language=route_lang, prompt=vllm_transcribe_prompt
                            )
                        except Exception as e:
                            asr_errors[vllm_backend.name] = f"{type(e).__name__}:{e}"
                        else:
                            if vllm_proofread and vllm_backend.supports_proofread and not proofread_use_anchor:
                                prompt_tpl = vllm_proofread_prompt or DEFAULT_VLLM_PROOFREAD_PROMPT
                                proof_prompt = prompt_tpl.format(transcript=vllm_res.text)
                                try:
                                    proof_res = vllm_backend.transcribe(
                                        wav_path, language=route_lang, prompt=proof_prompt, mode="proofread"
                                    )
                                    raw_key = f"{vllm_res.model}|raw"
                                    proof_key = f"{vllm_res.model}|proofread"
                                    asr_results[raw_key] = vllm_res
                                    asr_results[proof_key] = replace(proof_res, model=proof_key)
                                    if primary_asr_key is None:
                                        primary_asr_key = proof_key
                                    if guard_enabled and anchor_text:
                                        vllm_guard = _guard_llm_output(
                                            anchor_text=anchor_text,
                                            candidate_text=proof_res.text,
                                            lang=route_lang,
                                            thresholds=thresholds,
                                        )
                                        vllm_guard["candidate_key"] = proof_key
                                        vllm_guard["anchor_key"] = anchor_key
                                        if guard_override:
                                            if vllm_guard.get("accept"):
                                                vllm_guard_final = {"text": proof_res.text, "source": f"asr:{proof_key}"}
                                            else:
                                                fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                                vllm_guard_final = {"text": anchor_text, "source": fallback_source}
                                except Exception as e:
                                    asr_errors[f"{vllm_backend.name}|proofread"] = f"{type(e).__name__}:{e}"
                                    asr_results[vllm_res.model] = vllm_res
                            else:
                                asr_results[vllm_res.model] = vllm_res

                            if guard_enabled and anchor_text and vllm_guard is None:
                                vllm_guard = _guard_llm_output(
                                    anchor_text=anchor_text,
                                    candidate_text=vllm_res.text,
                                    lang=route_lang,
                                    thresholds=thresholds,
                                )
                                vllm_guard["candidate_key"] = vllm_res.model
                                vllm_guard["anchor_key"] = anchor_key
                                if guard_override:
                                    if vllm_guard.get("accept"):
                                        vllm_guard_final = {"text": vllm_res.text, "source": f"asr:{vllm_res.model}"}
                                    else:
                                        fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                        vllm_guard_final = {"text": anchor_text, "source": fallback_source}

'''

text = text[:start] + new_block + text[end:]
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Reviewing guard variable placement**

Planning to add `vllm_guard` variables and final override later, currently checking indentation and correctness around the vllm-related code snippet.


sed -n '300,460p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
                prompt_template=prompt,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e

    vllm_backend: VllmTranscriptionASR | None = None
    if vllm_asr_model:
        try:
            vllm_backend = VllmTranscriptionASR(
                model_id=vllm_asr_model,
                gpu_memory_utilization=vllm_gpu_memory_utilization,
                max_model_len=vllm_max_model_len,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize vLLM ASR backend.") from e

    default_whisper_run_sampling = bool(whisper_backend is not None) and not bool(
        transformers_backends or transformers_router or gemini_backend is not None
    )

    results_path = out_dir / "results.jsonl"
    summary_path = out_dir / "summary.json"

    counts = {"accept": 0, "human": 0, "reject": 0, "error": 0}
    timings_total: list[float] = []

    with results_path.open("w", encoding="utf-8") as f:
        for src in track(audio_files, description="Transcribing"):
            ident = segment_identity(src, root=segments_dir)
            segment_id = ident.segment_id
            t0 = time.perf_counter()

            try:
                # 1) Preprocess
                t_pre0 = time.perf_counter()
                src_info = ffprobe(src)
                wav_path = out_dir / "audio" / f"{segment_id}.wav"
                normalize_audio(
                    src,
                    wav_path,
                    trim_silence=trim_silence,
                    silence_threshold_db=silence_threshold_db,
                    min_silence_sec=min_silence_sec,
                )
                info = ffprobe(wav_path)
                qc = compute_qc(wav_path)
                t_pre1 = time.perf_counter()

                # 2) LID
                t_lid0 = time.perf_counter()
                lid: LIDResult | None = None
                if lid_backend is not None:
                    lid = lid_backend.predict(wav_path, top_k=5)
                t_lid1 = time.perf_counter()

                # Route language hint (only if confident).
                route_lang = None
                route_lang_source = None
                prefer_path = bool(thresholds.get("route_lang_prefer_path", False))
                if prefer_path and bool(thresholds.get("route_lang_from_path", False)):
                    inferred = _infer_lang_from_relpath(ident.relpath)
                    if inferred:
                        route_lang = inferred
                        route_lang_source = "path"

                if route_lang is None and lid and lid.predictions:
                    top1 = lid.predictions[0]
                    top2p = lid.predictions[1].prob if len(lid.predictions) > 1 else 0.0
                    if float(top1.prob) >= float(thresholds.get("lid_min_prob", 0.90)) and (
                        float(top1.prob) - float(top2p)
                    ) >= float(thresholds.get("lid_min_margin", 0.20)):
                        route_lang = top1.code
                        route_lang_source = "lid"

                if route_lang is None and bool(thresholds.get("route_lang_from_path", False)):
                    inferred = _infer_lang_from_relpath(ident.relpath)
                    if inferred:
                        route_lang = inferred
                        route_lang_source = "path"

                # 3) ASR ensemble
                t_asr0 = time.perf_counter()
                asr_results: dict[str, ASRResult] = {}
                asr_errors: dict[str, str] = {}
                primary_asr_key = None
                det = None
                router_key: str | None = None

                # Whisper: deterministic + a small sampling diversity pass.
                if whisper_backend is not None:
                    try:
                        det = whisper_backend.transcribe(
                            wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
                        )
                        primary_asr_key = f"{det.model}|det"
                        asr_results[primary_asr_key] = det
                    except Exception as e:
                        asr_errors[f"{whisper_backend.name}|det"] = f"{type(e).__name__}:{e}"

                run_sampling = bool(thresholds.get("whisper_run_sampling", default_whisper_run_sampling))
                if run_sampling and whisper_backend is not None:
                    try:
                        samp = whisper_backend.transcribe(
                            wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2
                        )
                        asr_results[f"{samp.model}|samp_t0.2"] = samp
                    except Exception as e:
                        asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"

                for tf_backend in transformers_backends:
                    try:
                        res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
                        asr_results[res.model] = res
                    except Exception as e:
                        asr_errors[tf_backend.name] = f"{type(e).__name__}:{e}"

                if transformers_router is not None:
                    try:
                        routed = transformers_router.transcribe(
                            wav_path,
                            language=route_lang,
                            num_beams=5,
                            max_new_tokens=256,
                        )
                        if routed is not None:
                            asr_results[routed.model] = routed
                            router_key = routed.model
                    except Exception as e:
                        asr_errors[f"transformers_router:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if vllm_backend is not None:
                    anchor_text = det.text if det is not None else None
                    anchor_key = primary_asr_key
                    guard_enabled = bool(thresholds.get("vllm_guard_enable", False))
                    guard_override = bool(thresholds.get("vllm_guard_override_final", False))
                    proofread_use_anchor = bool(thresholds.get("vllm_proofread_use_anchor", False))
                    skip_raw_if_anchor = bool(thresholds.get("vllm_skip_raw_if_anchor", False))

                    if proofread_use_anchor and anchor_text:
                        prompt_tpl = vllm_proofread_prompt or DEFAULT_VLLM_PROOFREAD_PROMPT
                        proof_prompt = prompt_tpl.format(transcript=anchor_text)
                        try:
                            mode = "proofread" if vllm_backend.supports_proofread else "transcribe"
                            proof_res = vllm_backend.transcribe(
                                wav_path, language=route_lang, prompt=proof_prompt, mode=mode
                            )
                            proof_key = f"{proof_res.model}|proofread_anchor"
                            asr_results[proof_key] = replace(proof_res, model=proof_key)
                            if primary_asr_key is None:
                                primary_asr_key = proof_key
                            if guard_enabled:
                                vllm_guard = _guard_llm_output(
                                    anchor_text=anchor_text,
                                    candidate_text=proof_res.text,
                                    lang=route_lang,
                                    thresholds=thresholds,
                                )
                                vllm_guard["candidate_key"] = proof_key
                                vllm_guard["anchor_key"] = anchor_key
                                if guard_override:


sed -n '460,640p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                                if guard_override:
                                    if vllm_guard.get("accept"):
                                        vllm_guard_final = {"text": proof_res.text, "source": f"asr:{proof_key}"}
                                    else:
                                        fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                        vllm_guard_final = {"text": anchor_text, "source": fallback_source}
                        except Exception as e:
                            asr_errors[f"{vllm_backend.name}|proofread"] = f"{type(e).__name__}:{e}"

                    if not (proofread_use_anchor and anchor_text and skip_raw_if_anchor):
                        try:
                            vllm_res = vllm_backend.transcribe(
                                wav_path, language=route_lang, prompt=vllm_transcribe_prompt
                            )
                        except Exception as e:
                            asr_errors[vllm_backend.name] = f"{type(e).__name__}:{e}"
                        else:
                            if vllm_proofread and vllm_backend.supports_proofread and not proofread_use_anchor:
                                prompt_tpl = vllm_proofread_prompt or DEFAULT_VLLM_PROOFREAD_PROMPT
                                proof_prompt = prompt_tpl.format(transcript=vllm_res.text)
                                try:
                                    proof_res = vllm_backend.transcribe(
                                        wav_path, language=route_lang, prompt=proof_prompt, mode="proofread"
                                    )
                                    raw_key = f"{vllm_res.model}|raw"
                                    proof_key = f"{vllm_res.model}|proofread"
                                    asr_results[raw_key] = vllm_res
                                    asr_results[proof_key] = replace(proof_res, model=proof_key)
                                    if primary_asr_key is None:
                                        primary_asr_key = proof_key
                                    if guard_enabled and anchor_text:
                                        vllm_guard = _guard_llm_output(
                                            anchor_text=anchor_text,
                                            candidate_text=proof_res.text,
                                            lang=route_lang,
                                            thresholds=thresholds,
                                        )
                                        vllm_guard["candidate_key"] = proof_key
                                        vllm_guard["anchor_key"] = anchor_key
                                        if guard_override:
                                            if vllm_guard.get("accept"):
                                                vllm_guard_final = {"text": proof_res.text, "source": f"asr:{proof_key}"}
                                            else:
                                                fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                                vllm_guard_final = {"text": anchor_text, "source": fallback_source}
                                except Exception as e:
                                    asr_errors[f"{vllm_backend.name}|proofread"] = f"{type(e).__name__}:{e}"
                                    asr_results[vllm_res.model] = vllm_res
                            else:
                                asr_results[vllm_res.model] = vllm_res

                            if guard_enabled and anchor_text and vllm_guard is None:
                                vllm_guard = _guard_llm_output(
                                    anchor_text=anchor_text,
                                    candidate_text=vllm_res.text,
                                    lang=route_lang,
                                    thresholds=thresholds,
                                )
                                vllm_guard["candidate_key"] = vllm_res.model
                                vllm_guard["anchor_key"] = anchor_key
                                if guard_override:
                                    if vllm_guard.get("accept"):
                                        vllm_guard_final = {"text": vllm_res.text, "source": f"asr:{vllm_res.model}"}
                                    else:
                                        fallback_source = f"asr:{anchor_key}" if anchor_key else "anchor"
                                        vllm_guard_final = {"text": anchor_text, "source": fallback_source}

                # MMS ASR for CTC-based diversity (per master.md plan).
                if mms_backend is not None:
                    try:
                        mms_res = mms_backend.transcribe(wav_path, language=route_lang)
                        asr_results[mms_res.model] = mms_res
                    except Exception as e:
                        asr_errors[f"mms-ctc:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                # IndicConformer for Conformer architecture diversity (per master.md plan).
                if indicconformer_router is not None:
                    try:
                        conformer_res = indicconformer_router.transcribe(wav_path, language=route_lang)
                        if conformer_res is not None:
                            asr_results[conformer_res.model] = conformer_res
                    except Exception as e:
                        asr_errors[f"indicconformer:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if hf_indicconformer_backend is not None:
                    try:
                        hf_res = hf_indicconformer_backend.transcribe(wav_path, language=route_lang)
                        asr_results[hf_res.model] = hf_res
                    except Exception as e:
                        asr_errors[f"hf-indicconformer:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if primary_asr_key is None and asr_results:
                    primary_asr_key = sorted(asr_results.keys())[0]
                if not asr_results:
                    raise RuntimeError(f"no_asr_hypotheses:{asr_errors or 'unknown'}")

                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
                if gemini_backend is not None:
                    run_gemini = True
                    if gemini_hard_only:
                        base_for_heur = det or (next(iter(asr_results.values())) if asr_results else None)
                        det_sum = _asr_summary(base_for_heur) if base_for_heur is not None else {}
                        avg_lp = det_sum.get("avg_logprob_mean")
                        comp = det_sum.get("compression_ratio_mean")
                        no_speech = det_sum.get("no_speech_prob_mean")
                        run_gemini = bool(
                            route_lang is None
                            or (no_speech is not None and float(no_speech) > 0.5)
                            or (avg_lp is not None and float(avg_lp) < float(thresholds.get("min_avg_logprob_mean", -1.0)))
                            or (
                                comp is not None
                                and float(comp) > float(thresholds.get("max_compression_ratio_mean", 2.4))
                            )
                        )

                    if run_gemini:
                        try:
                            g = gemini_backend.transcribe(wav_path, language=route_lang)
                            asr_results[g.model] = g
                        except Exception:
                            pass

                t_asr1 = time.perf_counter()

                ensemble_models = sorted({v.model for v in asr_results.values()})
                ensemble_backends = sorted({v.model.split(":", 1)[0] for v in asr_results.values()})
                ensemble_warnings: list[str] = []
                if len(ensemble_models) < 2:
                    ensemble_warnings.append("low_model_diversity")

                # 4) Normalize hypotheses
                t_norm0 = time.perf_counter()
                norm_hyps: dict[str, str] = {}
                for k, res in asr_results.items():
                    lang_hint = res.language or route_lang
                    norm_hyps[k] = normalize_text(res.text, lang=lang_hint)
                t_norm1 = time.perf_counter()

                # 5) ROVER
                t_rover0 = time.perf_counter()
                rover_hyps = dict(norm_hyps)
                rover_filtered_out: list[str] = []
                if bool(thresholds.get("filter_hypotheses_by_script", False)) and route_lang:
                    from maya_transcribe.language import dominant_script, expected_script

                    expected = expected_script(route_lang)
                    if expected:
                        kept: dict[str, str] = {}
                        for k, res in asr_results.items():
                            script = dominant_script(res.text)
                            if script is None or script == expected:
                                kept[k] = norm_hyps.get(k, "")
                            else:
                                rover_filtered_out.append(k)

                        kept = {k: v for k, v in kept.items() if v.strip()}
                        if kept:
                            rover_hyps = kept
                        else:
                            rover_filtered_out = []

                include_sampling_opt = thresholds.get("rover_include_sampling")
                if include_sampling_opt is None:
                    include_sampling = len(ensemble_backends) < 2
                else:
                    include_sampling = bool(include_sampling_opt)

                if not include_sampling:
                    dropped = [k for k in rover_hyps.keys() if k.endswith("|samp_t0.2")]
                    for k in dropped:
                        rover_hyps.pop(k, None)
                    rover_filtered_out.extend(dropped)

                prefer_router_opt = thresholds.get("rover_prefer_router")
                prefer_router = bool(prefer_router_opt) if prefer_router_opt is not None else (router_key is not None)
                if prefer_router and router_key and router_key in rover_hyps:
                    rover_hyps = {
                        router_key: rover_hyps[router_key],
                        **{k: v for k, v in rover_hyps.items() if k != router_key},
                    }


**Adding missing variable declarations**

I'm adding declarations for `vllm_guard` and `vllm_guard_final` near the existing `router_key` variable to prevent NameErrors in the ASR ensemble stage.


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')
old = '                primary_asr_key = None\n                det = None\n                router_key: str | None = None\n'
new = old + '                vllm_guard: dict[str, Any] | None = None\n                vllm_guard_final: dict[str, str] | None = None\n'
if old not in text:
    raise SystemExit('Anchor block not found')
text = text.replace(old, new)
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Adding final selection guard override**

I'll insert the guard override of final_text and final_source after the CTC step but before verification, and annotate vllm_guard data into the record accordingly.


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')

needle = '                t_ctc1 = time.perf_counter()\n\n                # 6) Verify\n'
if needle not in text:
    raise SystemExit('ctc block end not found')

insert = '                t_ctc1 = time.perf_counter()\n\n                if vllm_guard_final is not None:\n                    final_text = vllm_guard_final.get("text", final_text)\n                    final_source = vllm_guard_final.get("source", final_source)\n\n                # 6) Verify\n'
text = text.replace(needle, insert)
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Adding guard field to record**

Planning to add a `guard` field to the record dictionary, likely inserting it just before the "config" field for logical grouping near "verification".


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')

needle = '                    "verification": {\n                        "decision": verification.decision,\n                        "reasons": verification.reasons,\n                        "scores": verification.scores,\n                    },\n'
if needle not in text:
    raise SystemExit('verification block not found')

insert = needle + '                    "guard": vllm_guard,\n'
text = text.replace(needle, insert)
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


Summary: auto


**Cleaning imports and preparing config**

I'm confirming `vllm_guard` is set to None if no backend, considering removing redundant local import of `expected_script`, and deciding to add a blank line between `_infer_lang` and `_word_count` for style. Next, I'll create a new config file with threshold settings for the guard feature.


**Drafting new config with guard thresholds**

I'm preparing a config JSON that enables the guard feature with specific threshold values and flags like disabling sampling and enforcing strict prompts, ready to create the file via Python or shell.


cat <<'JSON' > configs/bench_hin_vllm_guard.json
{
  "route_lang_from_path": true,
  "route_lang_prefer_path": true,
  "whisper_run_sampling": false,
  "vllm_strict_prompt": true,
  "vllm_strict_proofread_prompt": true,
  "vllm_proofread_use_anchor": true,
  "vllm_skip_raw_if_anchor": true,
  "vllm_guard_enable": true,
  "vllm_guard_override_final": true,
  "vllm_guard_script_ratio_min": 0.6,
  "vllm_guard_length_ratio_min": 0.7,
  "vllm_guard_length_ratio_max": 1.3,
  "vllm_guard_anchor_wer_max": 0.2,
  "vllm_guard_min_letters": 4
}
JSON


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Adjusting prompt handling and style**

I'm ensuring the guard override final is properly set and used in the pipeline, confirming language placeholders in prompt rendering align with supported languages, and inserting blank lines after `_infer_lang_from_relpath` to meet style standards.


python3 - <<'PY'
from pathlib import Path
path = Path('maya_transcribe/pipeline.py')
text = path.read_text(encoding='utf-8')
text = text.replace('    return None\n\ndef _word_count', '    return None\n\n\ndef _word_count')
path.write_text(text, encoding='utf-8')
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Reviewing pipeline imports and record block**

I'm verifying the import of jiwer_wer and expected_script at the pipeline's top and checking the record block to confirm correct indentation, the presence of vllm_guard with a trailing comma, and the field order.


python3 - <<'PY'
from pathlib import Path
text = Path('maya_transcribe/pipeline.py').read_text(encoding='utf-8')
# Print record block to confirm
start = text.find('record = {')
print(text[start:start+800])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
record = {
                    "schema_version": 1,
                    "segment_id": segment_id,
                    "base_id": ident.base_id,
                    "relpath": ident.relpath,
                    "uid": ident.uid,
                    "src_path": str(src),
                    "wav_path": str(wav_path),
                    "src_ffprobe": asdict(src_info),
                    "ffprobe": asdict(info),
                    "qc": asdict(qc),
                    "lid": (
                        None
                        if lid is None
                        else {
                            "model": lid.model,
                            "predictions": [{"code": p.code, "prob": p.prob} for p in lid.predictions],
                        }
                    ),


python3 - <<'PY'
from pathlib import Path
text = Path('maya_transcribe/pipeline.py').read_text(encoding='utf-8')
start = text.find('"verification":')
print(text[start:start+200])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"verification": {
                        "decision": verification.decision,
                        "reasons": verification.reasons,
                        "scores": verification.scores,


**Extending code context**


python3 - <<'PY'
from pathlib import Path
text = Path('maya_transcribe/pipeline.py').read_text(encoding='utf-8')
start = text.find('"verification":')
print(text[start:start+280])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
"verification": {
                        "decision": verification.decision,
                        "reasons": verification.reasons,
                        "scores": verification.scores,
                    },
                    "guard": vllm_guard,
                    "config


**Verifying guard selection logic**

The guard override logic safely sets the guard variable only when anchor text exists, aligning with proofread and raw branch conditions, so no risk of None anchor usage; also confirmed the guard variable is initialized even without a vllm backend.


**Checking language module updates**

Planning to confirm the new script-related functions are properly imported and the dominant script function is intact without affecting language mapping utilities.


sed -n '40,180p' maya_transcribe/language.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    "kn": "Kannada",
    "mal": "Malayalam",
    "ml": "Malayalam",
    "ben": "Bengali",
    "bn": "Bengali",
    "asm": "Assamese",
    "as": "Assamese",
    "ori": "Odia",
    "ory": "Odia",
    "or": "Odia",
}

LANG_NAME_TO_SCRIPT: dict[str, str] = {
    "English": "Latin",
    "Hindi": "Devanagari",
    "Marathi": "Devanagari",
    "Gujarati": "Gujarati",
    "Punjabi": "Gurmukhi",
    "Tamil": "Tamil",
    "Telugu": "Telugu",
    "Kannada": "Kannada",
    "Malayalam": "Malayalam",
    "Bengali": "Bengali",
    "Assamese": "Assamese",
    "Odia": "Odia",
}

# Coarse script expectations for filtering/normalization heuristics.
# Assamese uses the Bengali Unicode block.
LANG_CODE_TO_SCRIPT: dict[str, str] = {
    "en": "Latin",
    "eng": "Latin",
    "hi": "Devanagari",
    "hin": "Devanagari",
    "mr": "Devanagari",
    "mar": "Devanagari",
    "gu": "Gujarati",
    "guj": "Gujarati",
    "pa": "Gurmukhi",
    "pan": "Gurmukhi",
    "ta": "Tamil",
    "tam": "Tamil",
    "te": "Telugu",
    "tel": "Telugu",
    "kn": "Kannada",
    "kan": "Kannada",
    "ml": "Malayalam",
    "mal": "Malayalam",
    "bn": "Bengali",
    "ben": "Bengali",
    "as": "Bengali",
    "asm": "Bengali",
    "or": "Oriya",
    "ori": "Oriya",
    "ory": "Oriya",
    "ur": "Arabic",
    "urd": "Arabic",
}

def expected_script(lang_code: str | None) -> str | None:
    if not lang_code:
        return None
    return LANG_CODE_TO_SCRIPT.get(lang_code.strip().lower())

def _char_script(ch: str) -> str | None:
    o = ord(ch)
    if 0x0900 <= o <= 0x097F:
        return "Devanagari"
    if 0x0980 <= o <= 0x09FF:
        return "Bengali"
    if 0x0A00 <= o <= 0x0A7F:
        return "Gurmukhi"
    if 0x0A80 <= o <= 0x0AFF:
        return "Gujarati"
    if 0x0B00 <= o <= 0x0B7F:
        return "Oriya"
    if 0x0B80 <= o <= 0x0BFF:
        return "Tamil"
    if 0x0C00 <= o <= 0x0C7F:
        return "Telugu"
    if 0x0C80 <= o <= 0x0CFF:
        return "Kannada"
    if 0x0D00 <= o <= 0x0D7F:
        return "Malayalam"
    if ("LATIN" in unicodedata.name(ch, "")) or ("A" <= ch <= "z"):
        return "Latin"
    return None

def dominant_script(text: str, *, min_letters: int = 4) -> str | None:
    """
    Heuristic: return the dominant script bucket based on Unicode block ranges.
    Returns None if there's not enough signal (e.g., mostly digits/punctuation).
    """
    if not text:
        return None

    counts: dict[str, int] = {}
    total_letters = 0

    for ch in text:
        if ch.isspace():
            continue
        cat = unicodedata.category(ch)
        if not cat.startswith("L"):
            continue

        script = _char_script(ch)
        if script is None:
            continue
        counts[script] = counts.get(script, 0) + 1
        total_letters += 1

    if total_letters < int(min_letters):
        return None
    return max(counts.items(), key=lambda kv: kv[1])[0] if counts else None

def script_ratio(text: str, expected_script: str | None, *, min_letters: int = 4) -> float | None:
    if not text or not expected_script:
        return None

    total_letters = 0
    expected_letters = 0

    for ch in text:
        if ch.isspace():
            continue
        cat = unicodedata.category(ch)
        if not cat.startswith("L"):
            continue

        script = _char_script(ch)
        if script is None:
            continue
        total_letters += 1
        if script == expected_script:
            expected_letters += 1


**Confirming integration details**

The pipeline correctly imports typing.Any and defines `vllm_guard` properly outside conditional blocks. Script ratio logic aligns with expected_script outputs for languages like Hindi, matching the script check as planned. Next steps include running benchmarks and updating results if improvements show.


**Clarifying benchmark run flags**

The guard depends on whisper anchors, so `--skip-whisper` must not be used despite runtime cost. The thresholds enable proofread anchored to whisper output regardless of the `vllm_proofread` flag, which is acceptable though could be clearer. I'll run Voxtral then Gemma sequentially with consistent thresholds and GPU settings.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_guarded --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --thresholds-json configs/bench_hin_vllm_guard.json


Output:
Exit code: 1
Wall time: 7.3 seconds
Total output lines: 1577
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_u │
│ tils.py:2317 in __getattr__                                                  │
│                                                                              │
│   2314 │   │   │   value = Placeholder                                       │
│   2315 │   │   elif name in self._class_to_module:                           │
│   2316 │   │   │   try:                                                      │
│ ❱ 2317 │   │   │   │   module = self._get_module(self._class_to_module[name] │
│   2318 │   │   │   │   value = getattr(module, name)                         │
│   2319 │   │   │   except (ModuleNotFoundError, RuntimeError) as e:          │
│   2320 │   │   │   │   raise ModuleNotFoundError(                            │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ name = 'Wav2Vec2ForSequenceClassification'                               │ │
│ │ self = <module 'transformers.models.wav2vec2' from                       │ │
│ │        '/home/ubuntu/.local/lib/python3.10/site-packages/transformers/m… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_u │
│ tils.py:2347 in _get_module                                                  │
│                                                                              │
│   2344 │   │   try:                                                          │
│   2345 │   │   │   return importlib.import_module("." + module_name, self.__ │
│   2346 │   │   except Exception as e:                                        │
│ ❱ 2347 │   │   │   raise e                                                   │
│   2348 │                                                                     │
│   2349 │   def __reduce__(self):                                             │
│   2350 │   │   return (self.__class__, (self._name, self.__file__, self._imp │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ module_name = 'modeling_wav2vec2'                                        │ │
│ │        self = <module 'transformers.models.wav2vec2' from                │ │
│ │               '/home/ubuntu/.local/lib/python3.10/site-packages/transfo… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_u │
│ tils.py:2345 in _get_module                                                  │
│                                                                              │
│   2342 │                                                                     │
│   2343 │   def _get_module(self, module_name: str):                          │
│   2344 │   │   try:                                                          │
│ ❱ 2345 │   │   │   return importlib.import_module("." + module_name, self.__ │
│   2346 │   │   except Exception as e:                                        │
│   2347 │   │   │   raise e                                                   │
│   2348                                                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ module_name = 'modeling_wav2vec2'                                        │ │
│ │        self = <module 'transformers.models.wav2vec2' from                │ │
│ │               '/home/ubuntu/.local/lib/python3.10/site-packages/transfo… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /usr/lib/python3.10/importlib/__init__.py:126 in import_module               │
│                                                                              │
│   123 │   │   │   if character != '.':                                       │
│   124 │   │   │   │   break                                                  │
│   125 │   │   │   level += 1                                                 │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│   127                                                                        │
│   128                                                                        │
│   129 _RELOADING = {}                                                        │
│                                                                              │
│ ╭────────────────── locals ──────────────────╮                               │
│ │ character = 'm'                            │                               │
│ │     level = 1                              │                               │
│ │      name = '.modeling_wav2vec2'           │                               │
│ │   package = 'transformers.models.wav2vec2' │                               │
│ ╰────────────────────────────────────────────╯                               │
│ in _gcd_import:1050                                                          │
│ ╭────────────────────────── locals ──────────────────────────╮               │
│ │   level = 1                                                │               │
│ │    name = 'transformers.models.wav2vec2.modeling_wav2vec2' │               │
│ │ package = 'transformers.models.wav2vec2'                   │               │
│ ╰────────────────────────────────────────────────────────────╯               │
│ in _find_and_load:1027                                                       │
│ ╭───────────────────────── locals ──────────────────────────╮                │
│ │ module = <object object at 0x7b7c63b50060>                │                │
│ │   name = 'transformers.models.wav2vec2.modeling_wav2vec2' │                │
│ ╰───────────────────────────────────────────────────────────╯                │
│ in _find_and_load_unlocked:1006                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          name = 'transformers.models.wav2vec2.modeling_wav2vec2'         │ │
│ │        parent = 'transformers.models.wav2vec2'                           │ │
│ │ parent_module = <module 'transformers.models.wav2vec2' from              │ │
│ │                 '/home/ubuntu/.local/lib/python3.10/site-packages/trans… │ │
│ │          path = [                                                        │ │
│ │                 │                                                        │ │
│ │                 '/home/ubuntu/.local/lib/python3.10/site-packages/trans… │ │
│ │                 ]                                                        │ │
│ │          spec = ModuleSpec(name='transformers.models.wav2vec2.modeling_… │ │
│ │                 loader=<_frozen_importlib_external.SourceFileLoader      │ │
│ │                 object at 0x7b7af5641e70>,                               │ │
│ │                 origin='/home/ubuntu/.local/lib/python3.10/site-package… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ in _load_unlocked:688                                                        │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ module = <module 'transformers.models.wav2vec2.modeling_wav2vec2' from   │ │
│ │          '/home/ubuntu/.local/lib/python3.10/site-packages/transformers… │ │
│ │   spec = ModuleSpec(name='transformers.models.wav2vec2.modeling_wav2vec… │ │
│ │          loader=<_frozen_importlib_external.SourceFileLoader object at   │ │
│ │          0x7b7af5641e70>,                                                │ │
│ │          origin='/home/ubuntu/.local/lib/python3.10/site-packages/trans… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ in exec_module:883                                                           │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │   code = <code object <module> at 0x7b7af54aa290, file                   │ │
│ │          "/home/ubuntu/.local/lib/python3.10/site-packages/transformers… │ │
│ │          line 1>                                                         │ │
│ │ module = <module 'transformers.models.wav2vec2.modeling_wav2vec2' from   │ │
│ │          '/home/ubuntu/.local/lib/python3.10/site-packages/transformers… │ │
│ │   self = <_frozen_importlib_external.SourceFileLoader object at          │ │
│ │          0x7b7af5641e70>                                                 │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│ in _call_with_frames_removed:241                                             │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ args = (                                                                 │ │
│ │        │   <code object <module> at 0x7b7af54aa290, file                 │ │
│ │        "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/m… │ │
│ │        line 1>,                                                          │ │
│ │        │   {                                                             │ │
│ │        │   │   '__name__':                                               │ │
│ │        'transformers.models.wav2vec2.modeling_wav2vec2',                 │ │
│ │        │   │   '__doc__': 'PyTorch Wav2Vec2 model.',                     │ │
│ │        │   │   '__package__': 'transformers.models.wav2vec2',            │ │
│ │        │   │   '__loader__':                                             │ │
│ │        <_frozen_importlib_external.SourceFileLoader object at            │ │
│ │        0x7b7af5641e70>,                                                  │ │
│ │        │   │   '__spec__':                                               │ │
│ │        ModuleSpec(name='transformers.models.wav2vec2.modeling_wav2vec2', │ │
│ │        loader=<_frozen_importlib_external.SourceFileLoader object at     │ │
│ │        0x7b7af5641e70>,                                                  │ │
│ │        origin='/home/ubuntu/.local/lib/python3.10/site-packages/transfo… │ │
│ │        │   │   '__file__':                                               │ │
│ │        '/home/ubuntu/.local/lib/python3.10/site-packages/transformers/m… │ │
│ │        │   │   '__cached__':                                             │ │
│ │        '/home/ubuntu/.local/lib/python3.10/site-packages/transformers/m… │ │
│ │        │   │   '__builtins__': {                                         │ │
│ │        │   │   │   '__name__': 'builtins',                               │ │
│ │        │   │   │   '__doc__': 'Built-in functions, exceptions, and other │ │
│ │        objects.\n\nNoteworthy: None is the `nil'+46,                     │ │
│ │        │   │   │   '__package__': '',                                    │ │
│ │        │   │   │   '__loader__': <class                                  │ │
│ │        '_frozen_importlib.BuiltinImporter'>,                             │ │
│ │        │   │   │   '__spec__': ModuleSpec(name='builtins', loader=<class │ │
│ │        '_frozen_importlib.BuiltinImporter'>, origin='built-in'),         │ │
│ │        │   │   │   '__build_class__': <built-in function                 │ │
│ │        __build_class__>,                                                 │ │
│ │        │   │   │   '__import__': <built-in function __import__>,         │ │
│ │        │   │   │   'abs': <built-in function abs>,                       │ │
│ │        │   │   │   'all': <built-in function all>,                       │ │
│ │        │   │   │   'any': <built-in function any>,                       │ │
│ │        │   │   │   ... +147                                              │ │
│ │        │   │   },                                                        │ │
│ │        │   │   'math': <module 'math' (built-in)>,                       │ │
│ │        │   │   'warnings': <module 'warnings' from                       │ │
│ │        '/usr/lib/python3.10/warnings.py'>,                               │ │
│ │        │   │   ... +15                                                   │ │
│ │        │   }                                                             │ │
│ │        )                                                                 │ │
│ │    f = <built-in function exec>                                          │ │
│ │ kwds = {}                                                                │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/wav2vec │
│ 2/modeling_wav2vec2.py:36 in <module>                                        │
│                                                                              │
│     33 │   _prepare_4d_attention_mask_for_sdpa,                              │
│     34 )                                                                     │
│     35 from ...modeling_flash_attention_utils import FlashAttentionKwargs    │
│ ❱   36 from ...modeling_layers import GradientCheckpointingLayer             │
│     37 from ...modeling_outputs import (                                     │
│     38 │   BaseModelOutput,                                                  │
│     39 │   CausalLMOutput,                                                   │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │   ACT2FN = ClassInstantier([('gelu', <class                              │ │
│ │            'transformers.activations.GELUActivation'>), ('gelu_10',      │ │
│ │            (<class 'transformers.activations.ClippedGELUActivation'>,    │ │
│ │            {'min': -10, 'max': 10})), ('gelu_fast', <class               │ │
│ │            'transformers.activations.FastGELUActivation'>), ('gelu_new', │ │
│ │            <class 'transformers.activations.NewGELUActivation'>),        │ │
│ │            ('gelu_python', (<class                                       │ │
│ │            'transformers.activations.GELUActivation'>,                   │ │
│ │            {'use_gelu_python': True})), ('gelu_pytorch_tanh', <class     │ │
│ │            'transformers.activations.GELUTanh'>), ('gelu_python_tanh',   │ │
│ │            (<class 'transformers.activations.GELUTanh'>,                 │ │
│ │            {'use_gelu_tanh_python': True})), ('gelu_accurate', <class    │ │
│ │            'transformers.activations.AccurateGELUActivation'>),          │ │
│ │            ('laplace', <class                                            │ │
│ │            'transformers.activati…27337 tokens truncated…                                                                              │
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/auto/au │
│ to_factory.py:733 in getattribute_from_module                                │
│                                                                              │
│   730 │   │   return None                                                    │
│   731 │   if isinstance(attr, tuple):                                        │
│   732 │   │   return tuple(getattribute_from_module(module, a) for a in attr │
│ ❱ 733 │   if hasattr(module, attr):                                          │
│   734 │   │   return getattr(module, attr)                                   │
│   735 │   # Some of the mappings have entries model_type -> object of anothe │
│   736 │   # object at the top level.                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │   attr = 'Wav2Vec2ForSequenceClassification'                             │ │
│ │ module = <module 'transformers.models.wav2vec2' from                     │ │
│ │          '/home/ubuntu/.local/lib/python3.10/site-packages/transformers… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_u │
│ tils.py:2320 in __getattr__                                                  │
│                                                                              │
│   2317 │   │   │   │   module = self._get_module(self._class_to_module[name] │
│   2318 │   │   │   │   value = getattr(module, name)                         │
│   2319 │   │   │   except (ModuleNotFoundError, RuntimeError) as e:          │
│ ❱ 2320 │   │   │   │   raise ModuleNotFoundError(                            │
│   2321 │   │   │   │   │   f"Could not import module '{name}'. Are this obje │
│   2322 │   │   │   │   ) from e                                              │
│   2323                                                                       │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │ name = 'Wav2Vec2ForSequenceClassification'                               │ │
│ │ self = <module 'transformers.models.wav2vec2' from                       │ │
│ │        '/home/ubuntu/.local/lib/python3.10/site-packages/transformers/m… │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: Could not import module
'Wav2Vec2ForSequenceClassification'. Are this object's requirements defined
correctly?

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/maya-transcribe/maya_transcribe/cli.py:496 in bench_run         │
│                                                                              │
│   493 │   │   thresholds["require_ctc"] = require_ctc                        │
│   494 │   │   thresholds["min_ctc_margin"] = float(min_ctc_margin)           │
│   495 │                                                                      │
│ ❱ 496 │   results_path = run_pipeline(                                       │
│   497 │   │   segments_dir=audio_dir,                                        │
│   498 │   │   out_dir=out,                                                   │
│   499 │   │   lid_model_id=lid_model_id,                                     │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                      audio_dir = PosixPath('data/bench/maya_hf_hin_v2/a… │ │
│ │                      bench_dir = PosixPath('data/bench/maya_hf_hin_v2')  │ │
│ │                        ctc_map = None                                    │ │
│ │                      ctc_model = None                                    │ │
│ │                  ctc_model_map = None                                    │ │
│ │       hf_indicconformer_decode = 'rnnt'                                  │ │
│ │        hf_indicconformer_model = None                                    │ │
│ │                     hypothesis = 'final'                                 │ │
│ │                         ic_map = None                                    │ │
│ │   indicconformer_default_model = None                                    │ │
│ │       indicconformer_model_map = None                                    │ │
│ │                   lid_model_id = 'facebook/mms-lid-256'                  │ │
│ │                          limit = None                                    │ │
│ │                         loaded = {                                       │ │
│ │                                  │   'route_lang_from_path': True,       │ │
│ │                                  │   'route_lang_prefer_path': True,     │ │
│ │                                  │   'whisper_run_sampling': False,      │ │
│ │                                  │   'vllm_strict_prompt': True,         │ │
│ │                                  │   'vllm_strict_proofread_prompt':     │ │
│ │                                  True,                                   │ │
│ │                                  │   'vllm_proofread_use_anchor': True,  │ │
│ │                                  │   'vllm_skip_raw_if_anchor': True,    │ │
│ │                                  │   'vllm_guard_enable': True,          │ │
│ │                                  │   'vllm_guard_override_final': True,  │ │
│ │                                  │   'vllm_guard_script_ratio_min': 0.6, │ │
│ │                                  │   ... +4                              │ │
│ │                                  }                                       │ │
│ │                 min_ctc_margin = 0.05                                    │ │
│ │                min_silence_sec = 0.15                                    │ │
│ │                  mms_asr_model = None                                    │ │
│ │                            out = PosixPath('outputs/bench/maya_hf_hin_v… │ │
│ │                      refs_path = PosixPath('data/bench/maya_hf_hin_v2/r… │ │
│ │                    require_ctc = True                                    │ │
│ │            require_lid_backend = True                                    │ │
│ │           silence_threshold_db = -40.0                                   │ │
│ │                   skip_whisper = False                                   │ │
│ │                         tf_map = None                                    │ │
│ │                     thresholds = {                                       │ │
│ │                                  │   'route_lang_from_path': True,       │ │
│ │                                  │   'route_lang_prefer_path': True,     │ │
│ │                                  │   'whisper_run_sampling': False,      │ │
│ │                                  │   'vllm_strict_prompt': True,         │ │
│ │                                  │   'vllm_strict_proofread_prompt':     │ │
│ │                                  True,                                   │ │
│ │                                  │   'vllm_proofread_use_anchor': True,  │ │
│ │                                  │   'vllm_skip_raw_if_anchor': True,    │ │
│ │                                  │   'vllm_guard_enable': True,          │ │
│ │                                  │   'vllm_guard_override_final': True,  │ │
│ │                                  │   'vllm_guard_script_ratio_min': 0.6, │ │
│ │                                  │   ... +4                              │ │
│ │                                  }                                       │ │
│ │                thresholds_json = PosixPath('configs/bench_hin_vllm_guar… │ │
│ │     transformers_whisper_model = []                                      │ │
│ │ transformers_whisper_model_map = None                                    │ │
│ │                   trim_silence = False                                   │ │
│ │                 vllm_asr_model = 'mistralai/Voxtral-Small-24B-2507'      │ │
│ │    vllm_gpu_memory_utilization = 0.9                                     │ │
│ │             vllm_max_model_len = 4096                                    │ │
│ │                 vllm_proofread = False                                   │ │
│ │          vllm_proofread_prompt = None                                    │ │
│ │         vllm_transcribe_prompt = None                                    │ │
│ │    vllm_transcribe_prompt_file = None                                    │ │
│ │    vllm_transcribe_prompt_text = None                                    │ │
│ │                  whisper_model = 'large-v3'                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py:236 in run_pipeline │
│                                                                              │
│   233 │   │   lid_backend = MMSLID(model_id=lid_model_id)                    │
│   234 │   except Exception as e:                                             │
│   235 │   │   if require_lid_backend:                                        │
│ ❱ 236 │   │   │   raise RuntimeError(                                        │
│   237 │   │   │   │   "Failed to initialize MMS-LID. Install torch/transform │
│   238 │   │   │   │   "or run with require_lid_backend=False."               │
│   239 │   │   │   ) from e                                                   │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                       audio_files = [                                    │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │   ... +190                         │ │
│ │                                     ]                                    │ │
│ │                         ctc_model = None                                 │ │
│ │                     ctc_model_map = None                                 │ │
│ │ extra_transformers_whisper_models = []                                   │ │
│ │                  gemini_hard_only = True                                 │ │
│ │          gemini_max_output_tokens = 512                                  │ │
│ │                      gemini_model = None                                 │ │
│ │                gemini_prompt_path = None                                 │ │
│ │          hf_indicconformer_decode = 'rnnt'                               │ │
│ │           hf_indicconformer_model = None                                 │ │
│ │      indicconformer_default_model = None                                 │ │
│ │          indicconformer_model_map = None                                 │ │
│ │                      lid_model_id = 'facebook/mms-lid-256'               │ │
│ │                             limit = None                                 │ │
│ │                   min_silence_sec = 0.15                                 │ │
│ │                     mms_asr_model = None                                 │ │
│ │                           out_dir = PosixPath('outputs/bench/maya_hf_hi… │ │
│ │               require_lid_backend = True                                 │ │
│ │                      segments_dir = PosixPath('data/bench/maya_hf_hin_v… │ │
│ │              silence_threshold_db = -40.0                                │ │
│ │                      skip_whisper = False                                │ │
│ │                  strict_proofread = True                                 │ │
│ │                        thresholds = {                                    │ │
│ │                                     │   'route_lang_from_path': True,    │ │
│ │                                     │   'route_lang_prefer_path': True,  │ │
│ │                                     │   'whisper_run_sampling': False,   │ │
│ │                                     │   'vllm_strict_prompt': True,      │ │
│ │                                     │   'vllm_strict_proofread_prompt':  │ │
│ │                                     True,                                │ │
│ │                                     │   'vllm_proofread_use_anchor':     │ │
│ │                                     True,                                │ │
│ │                                     │   'vllm_skip_raw_if_anchor': True, │ │
│ │                                     │   'vllm_guard_enable': True,       │ │
│ │                                     │   'vllm_guard_override_final':     │ │
│ │                                     True,                                │ │
│ │                                     │   'vllm_guard_script_ratio_min':   │ │
│ │                                     0.6,                                 │ │
│ │                                     │   ... +4                           │ │
│ │                                     }                                    │ │
│ │  transformers_whisper_default_model None                                 │ │
│ │                                   =                                      │ │
│ │    transformers_whisper_model_map = None                                 │ │
│ │                      trim_silence = False                                │ │
│ │                 use_strict_prompt = True                                 │ │
│ │                    vllm_asr_model = 'mistralai/Voxtral-Small-24B-2507'   │ │
│ │       vllm_gpu_memory_utilization = 0.9                                  │ │
│ │                vllm_max_model_len = 4096                                 │ │
│ │                    vllm_proofread = False                                │ │
│ │             vllm_proofread_prompt = 'Correct transcription errors using  │ │
│ │                                     the audio. Keep the same             │ │
│ │                                     language/script.\nRul'+229           │ │
│ │            vllm_transcribe_prompt = 'You are an ASR system. Transcribe   │ │
│ │                                     the audio verbatim in <language>     │ │
│ │                                     script.\nRules'+191                  │ │
│ │                     whisper_model = 'large-v3'                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to initialize MMS-LID. Install torch/transformers deps (pip
install -e '.[torch]') or run with require_lid_backend=False.


Summary: auto


**Bypassing LID backend due to missing dependency**

The user’s error stems from a missing transformer model needed for language ID; setting `--require-lid-backend False` will skip LID initialization and avoid this failure, allowing the pipeline to run using path-based language routing instead.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_guarded --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --thresholds-json configs/bench_hin_vllm_guard.json --require-lid-backend False


Output:
Exit code: 2
Wall time: 0.3 seconds
Output:
Usage: python -m maya_transcribe.cli bench run [OPTIONS]
Try 'python -m maya_transcribe.cli bench run --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Got unexpected extra argument (False)                                        │
╰──────────────────────────────────────────────────────────────────────────────╯


**Verifying boolean flag usage**

I checked Typer's handling of boolean flags and decided to use the `--no-require-lid-backend` option to disable the feature as intended.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_guarded --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --thresholds-json configs/bench_hin_vllm_guard.json --no-require-lid-backend


Output: execution error: Sandbox(Signal(6))


**Investigating vllm crash cause**

The command likely aborted with signal 6 due to a memory or torch issue during vllm import or initialization; I'll check logs and rerun with lowered GPU memory utilization and increased logging to diagnose.


**Confirming bypass of LID import**

I'll rerun with the `--no-require-lid-backend` flag to ensure LID import is bypassed and capture the output to verify any relevant messages.


Output: execution error: Sandbox(Signal(6))


**Diagnosing crash cause**

Suspecting the crash is from vllm initialization or memory issues causing signal 6 abort; next steps involve isolating whether importing or instantiating vllm triggers the failure by running minimal test commands and trying the CLI with `--skip-whisper` to confirm root cause.


**Testing minimal vllm instantiation**

Planning to run a minimal script importing and instantiating VllmTranscriptionASR with debug and fault handler enabled to capture logs, aiming to pinpoint if vllm or torch initialization triggers the abort despite prior working setups.


PYTHONFAULTHANDLER=1 python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_guarded --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --thresholds-json configs/bench_hin_vllm_guard.json --no-require-lid-backend


Output: execution error: Sandbox(Signal(6))


**Diagnosing silent vLLM crash**

The vLLM process aborts early with no output, possibly due to memory or initialization issues despite moderate settings; testing minimal vLLM runs and verifying safe imports will clarify whether installation or resource limits cause the failure.


**Testing vLLM import stability**

Since vLLM import may trigger aborts affecting the pipeline, I'll test importing vLLM with a simple script to confirm stability and explore setting CUDA allocation limits if needed.


python3 - <<'PY'
import vllm
print('vllm', vllm.__version__)
PY


Output:
Exit code: 0
Wall time: 2.6 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
vllm 0.13.0


**Testing model instantiation for GPU issues**

I'm running a minimal script to instantiate the VllmTranscriptionASR with a small model to check if the GPU memory allocation causes the crash. The goal is to confirm whether the failure happens at model load before deeper pipeline steps.


python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init start')
backend = VllmTranscriptionASR(
    model_id='mistralai/Voxtral-Small-24B-2507',
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)
print('init done', backend.name)
PY


Output:
Exit code: 134
Wall time: 3.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 10: 2925153 Aborted                 (core dumped) python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init start')
backend = VllmTranscriptionASR(
    model_id='mistralai/Voxtral-Small-24B-2507',
    gpu_memory_utilization=0.9,
    max_model_len=4096,
)
print('init done', backend.name)
PY


**Diagnosing model init memory error**

vLLM initialization fails with a CPU out-of-memory error, likely due to large RAM allocation rather than GPU memory; checking system RAM is needed to confirm.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
               total        used        free      shared  buff/cache   available
Mem:           118Gi       5.9Gi        41Gi       156Mi        71Gi       110Gi
Swap:          4.0Gi        26Mi       4.0Gi


**Analyzing std::bad_alloc cause**

The std::bad_alloc likely stems from CPU memory allocation failure during model loading despite ample system memory, possibly triggered by vLLM's handling of large model weights or KV cache sizing related to GPU parameters.


**Suspecting GPU memory shortage**

The error may stem from insufficient GPU memory caused by other processes, leading to CPU allocation failure during model loading; checking GPU usage with nvidia-smi is the next step.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
81920 MiB, 1 MiB, 81156 MiB


Summary: auto


**Assessing memory cause of bad_alloc error**

The error likely arises when loading model weights into CPU memory before GPU allocation, possibly due to dtype or model size. Next, testing loading a smaller vLLM model will help confirm if memory or initialization is the issue.


python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init gemma')
backend = VllmTranscriptionASR(model_id='google/gemma-3n-E2B-it', gpu_memory_utilization=0.9, max_model_len=4096)
print('done', backend.name)
PY


Output:
Exit code: 134
Wall time: 3.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 6: 2925546 Aborted                 (core dumped) python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init gemma')
backend = VllmTranscriptionASR(model_id='google/gemma-3n-E2B-it', gpu_memory_utilization=0.9, max_model_len=4096)
print('done', backend.name)
PY


**Diagnosing memory allocation failure**

Suspecting the initialization crash from std::bad_alloc relates to vLLM's internal memory allocation, possibly due to large max_model_len or GPU memory handling; next step is to test with reduced memory settings and a smaller model to isolate the cause.


python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init gemma small')
backend = VllmTranscriptionASR(model_id='google/gemma-3n-E2B-it', gpu_memory_utilization=0.5, max_model_len=1024)
print('done', backend.name)
PY


Output:
Exit code: 134
Wall time: 3.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 6: 2925720 Aborted                 (core dumped) python3 - <<'PY'
from maya_transcribe.asr import VllmTranscriptionASR
print('init gemma small')
backend = VllmTranscriptionASR(model_id='google/gemma-3n-E2B-it', gpu_memory_utilization=0.5, max_model_len=1024)
print('done', backend.name)
PY


**Diagnosing vLLM initialization failure**

The failure may be linked to missing model weights after clearing the HF cache, causing vLLM to abort silently during allocation or quantization. Inspecting environment variables and core dumps is planned since logs don’t appear before the abort.


**Testing vLLM with tiny model**

I'm setting up vLLM to load a minimal model to verify if the initialization issue is general or model-specific.


python3 - <<'PY'
from vllm import LLM
print('init small')
llm = LLM(model='hf-internal-testing/tiny-random-gpt2', gpu_memory_utilization=0.3, max_model_len=128)
print('done')
PY


Output:
Exit code: 134
Wall time: 3.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 6: 2926395 Aborted                 (core dumped) python3 - <<'PY'
from vllm import LLM
print('init small')
llm = LLM(model='hf-internal-testing/tiny-random-gpt2', gpu_memory_utilization=0.3, max_model_len=128)
print('done')
PY


**Diagnosing vLLM memory allocation failure**

The error likely stems from vLLM's C++ extension failing to allocate memory during initialization, possibly due to environment changes or incompatible builds after cache cleaning, but key suspects like shared memory limits or env variables have been ruled out so far. Running in the separate `.venv_omni` environment with vLLM intact may be the next pragmatic step.


**Investigating vLLM device configuration**

The memory error might relate to CUDA graph allocations, so testing vLLM on CPU by passing a device parameter could help confirm this; I'll check vLLM's LLM initialization code to see if a CPU device option exists to enable that test.


sed -n '1,160p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

import itertools
from collections.abc import Callable, Sequence
from typing import TYPE_CHECKING, Any, cast

import cloudpickle
import torch.nn as nn
from pydantic import ValidationError
from tqdm.auto import tqdm
from typing_extensions import TypeVar

from vllm.beam_search import (
    BeamSearchInstance,
    BeamSearchOutput,
    BeamSearchSequence,
    create_sort_beams_key_function,
)
from vllm.config import (
    CompilationConfig,
    PoolerConfig,
    ProfilerConfig,
    StructuredOutputsConfig,
    is_init_field,
)
from vllm.config.compilation import CompilationMode
from vllm.config.model import (
    ConvertOption,
    HfOverrides,
    ModelDType,
    RunnerOption,
    TokenizerMode,
)
from vllm.engine.arg_utils import EngineArgs
from vllm.entrypoints.chat_utils import (
    ChatCompletionMessageParam,
    ChatTemplateContentFormatOption,
    apply_hf_chat_template,
    apply_mistral_chat_template,
    parse_chat_messages,
    resolve_chat_template_content_format,
)
from vllm.entrypoints.score_utils import (
    ScoreContentPartParam,
    ScoreMultiModalParam,
    _cosine_similarity,
    _validate_score_input_lens,
    compress_token_type_ids,
    get_score_prompt,
)
from vllm.entrypoints.utils import _validate_truncation_size, log_non_default_args
from vllm.inputs import (
    DataPrompt,
    PromptType,
    SingletonPrompt,
    TextPrompt,
    TokensPrompt,
)
from vllm.inputs.parse import get_prompt_components
from vllm.logger import init_logger
from vllm.lora.request import LoRARequest
from vllm.model_executor.layers.quantization import QuantizationMethods
from vllm.outputs import (
    ClassificationRequestOutput,
    EmbeddingRequestOutput,
    PoolingRequestOutput,
    RequestOutput,
    ScoringRequestOutput,
)
from vllm.platforms import current_platform
from vllm.pooling_params import PoolingParams
from vllm.sampling_params import BeamSearchParams, RequestOutputKind, SamplingParams
from vllm.tasks import PoolingTask
from vllm.tokenizers import TokenizerLike
from vllm.tokenizers.mistral import MistralTokenizer
from vllm.usage.usage_lib import UsageContext
from vllm.utils.collection_utils import as_iter, is_list_of
from vllm.utils.counter import Counter
from vllm.v1.engine import EngineCoreRequest
from vllm.v1.engine.llm_engine import LLMEngine
from vllm.v1.sample.logits_processor import LogitsProcessor

if TYPE_CHECKING:
    from vllm.v1.metrics.reader import Metric

logger = init_logger(__name__)

_R = TypeVar("_R", default=Any)

class LLM:
    """An LLM for generating texts from given prompts and sampling parameters.

    This class includes a tokenizer, a language model (possibly distributed
    across multiple GPUs), and GPU memory space allocated for intermediate
    states (aka KV cache). Given a batch of prompts and sampling parameters,
    this class generates texts from the model, using an intelligent batching
    mechanism and efficient memory management.

    Args:
        model: The name or path of a HuggingFace Transformers model.
        tokenizer: The name or path of a HuggingFace Transformers tokenizer.
        tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer
            if available, and "slow" will always use the slow tokenizer.
        skip_tokenizer_init: If true, skip initialization of tokenizer and
            detokenizer. Expect valid prompt_token_ids and None for prompt
            from the input.
        trust_remote_code: Trust remote code (e.g., from HuggingFace) when
            downloading the model and tokenizer.
        allowed_local_media_path: Allowing API requests to read local images
            or videos from directories specified by the server file system.
            This is a security risk. Should only be enabled in trusted
            environments.
        allowed_media_domains: If set, only media URLs that belong to this
            domain can be used for multi-modal inputs.
        tensor_parallel_size: The number of GPUs to use for distributed
            execution with tensor parallelism.
        dtype: The data type for the model weights and activations. Currently,
            we support `float32`, `float16`, and `bfloat16`. If `auto`, we use
            the `dtype` attribute of the Transformers model's config. However,
            if the `dtype` in the config is `float32`, we will use `float16` instead.
        quantization: The method used to quantize the model weights. Currently,
            we support "awq", "gptq", and "fp8" (experimental).
            If None, we first check the `quantization_config` attribute in the
            model config file. If that is None, we assume the model weights are
            not quantized and use `dtype` to determine the data type of
            the weights.
        revision: The specific model version to use. It can be a branch name,
            a tag name, or a commit id.
        tokenizer_revision: The specific tokenizer version to use. It can be a
            branch name, a tag name, or a commit id.
        seed: The seed to initialize the random number generator for sampling.
        gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to
            reserve for the model weights, activations, and KV cache. Higher
            values will increase the KV cache size and thus improve the model's
            throughput. However, if the value is too high, it may cause out-of-
            memory (OOM) errors.
        kv_cache_memory_bytes: Size of KV Cache per GPU in bytes. By default,
            this is set to None and vllm can automatically infer the kv cache
            size based on gpu_memory_utilization. However, users may want to
            manually specify the kv cache memory size. kv_cache_memory_bytes
            allows more fine-grain control of how much memory gets used when
            compared with using gpu_memory_utilization. Note that
            kv_cache_memory_bytes (when not-None) ignores
            gpu_memory_utilization
        swap_space: The size (GiB) of CPU memory per GPU to use as swap space.
            This can be used for temporarily storing the states of the requests
            when their `best_of` sampling parameters are larger than 1. If all
            requests will have `best_of=1`, you can safely set this to 0.
            Noting that `best_of` is only supported in V0. Otherwise, too small
            values may cause out-of-memory (OOM) errors.
        cpu_offload_gb: The size (GiB) of CPU memory to use for offloading
            the model weights. This virtually increases the GPU memory space
            you can use to hold the model weights, at the cost of CPU-GPU data
            transfer for every forward pass.
        enforce_eager: Whether to enforce eager execution. If True, we will
            disable CUDA graph and always execute the model in eager mode.
            If False, we will use CUDA graph and eager execution in hybrid.
        disable_custom_all_reduce: See


**Analyzing memory error causes**

I'm considering that the memory error might come from vLLM's internal allocations, possibly related to eager enforcement, GPU availability, engine defaults, or environment variables triggering large buffer allocations despite small model sizes.


**Identifying KV cache memory over-allocation**

The large GPU memory leads vLLM to allocate an excessively large CPU KV cache, possibly exceeding available CPU RAM despite ample total memory; lowering GPU memory utilization further may help avoid this overallocation.


python3 - <<'PY'
from vllm import LLM
print('init tiny low mem')
llm = LLM(model='hf-internal-testing/tiny-random-gpt2', gpu_memory_utilization=0.05, max_model_len=128)
print('done')
PY


Output:
Exit code: 134
Wall time: 3.4 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 6: 2926668 Aborted                 (core dumped) python3 - <<'PY'
from vllm import LLM
print('init tiny low mem')
llm = LLM(model='hf-internal-testing/tiny-random-gpt2', gpu_memory_utilization=0.05, max_model_len=128)
print('done')
PY


**Testing vLLM on CPU to isolate GPU issues**

Planning to disable GPU by setting CUDA_VISIBLE_DEVICES="" and run vLLM with a tiny model to check if the crash is due to CUDA initialization or GPU-specific features like CUDA graphs or eager enforcement settings.


CUDA_VISIBLE_DEVICES= python3 - <<'PY'
from vllm import LLM
print('init tiny cpu?')
llm = LLM(model='hf-internal-testing/tiny-random-gpt2', max_model_len=128)
print('done')
PY