Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path

import numpy as np

from maya_transcribe.audio import decode_pcm16

@dataclass(frozen=True)
class LanguagePrediction:
    code: str
    prob: float

@dataclass(frozen=True)
class LIDResult:
    model: str
    predictions: list[LanguagePrediction]

class MMSLID:
    def __init__(self, model_id: str = "facebook/mms-lid-256", device: str | None = None) -> None:
        try:
            import torch
            from transformers import AutoFeatureExtractor, AutoModelForAudioClassification
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "MMS-LID requires the optional dependencies: torch + transformers. "
                "Install with: pip install -e '.[torch]'"
            ) from e

        if device is None:
            device = "cuda" if torch.cuda.is_available() else "cpu"

        self.model_id = model_id
        self.device = device
        self._torch = torch
        self._processor = AutoFeatureExtractor.from_pretrained(model_id)
        self._model = AutoModelForAudioClassification.from_pretrained(model_id).to(device)
        self._model.eval()

    @property
    def name(self) -> str:
        return self.model_id

    def predict(self, audio_path: Path, top_k: int = 5) -> LIDResult:
        wav, sr = decode_pcm16(audio_path, sample_rate=16000, mono=True)
        if wav.size == 0:
            return LIDResult(model=self.model_id, predictions=[])

        inputs = self._processor(wav, sampling_rate=sr, return_tensors="pt")
        inputs = {k: v.to(self.device) for k, v in inputs.items()}

        with self._torch.no_grad():
            logits = self._model(**inputs).logits
            probs = self._torch.softmax(logits, dim=-1)[0].detach().cpu().numpy().astype(np.float32)

        top_idx = np.argsort(probs)[::-1][: max(top_k, 1)]
        id2label = getattr(self._model.config, "id2label", {}) or {}

        preds: list[LanguagePrediction] = []
        for idx in top_idx:
            label = id2label.get(int(idx), str(int(idx)))
            preds.append(LanguagePrediction(code=str(label), prob=float(probs[int(idx)])))

        return LIDResult(model=self.model_id, predictions=preds)


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Investigating audio decoding**

I'm looking into LID, specifically the `decode_pcm16` function, which likely uses FFmpeg to decode audio into PCM format. It seems crucial to check the `audio.py` file to confirm this and gain a deeper understanding of how it all works together. I'll inspect the code there to clarify if my assumptions are correct. It’s always good to double-check!


sed -n '1,260p' maya_transcribe/audio.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
import subprocess
from dataclasses import dataclass
from pathlib import Path

import numpy as np

@dataclass(frozen=True)
class AudioInfo:
    path: str
    duration_sec: float | None
    sample_rate: int | None
    channels: int | None

@dataclass(frozen=True)
class AudioQCMetrics:
    duration_sec: float
    rms_dbfs: float
    peak_dbfs: float
    clipping_fraction: float

def ffprobe(path: Path) -> AudioInfo:
    cmd = [
        "ffprobe",
        "-v",
        "error",
        "-show_entries",
        "format=duration",
        "-show_entries",
        "stream=sample_rate,channels",
        "-of",
        "json",
        str(path),
    ]
    proc = subprocess.run(cmd, check=True, capture_output=True, text=True)
    data = json.loads(proc.stdout or "{}")

    duration_sec = None
    try:
        duration_sec = float(data.get("format", {}).get("duration"))
    except (TypeError, ValueError):
        duration_sec = None

    sample_rate = None
    channels = None
    streams = data.get("streams") or []
    if streams:
        s0 = streams[0] or {}
        try:
            sample_rate = int(s0.get("sample_rate")) if s0.get("sample_rate") else None
        except (TypeError, ValueError):
            sample_rate = None
        try:
            channels = int(s0.get("channels")) if s0.get("channels") else None
        except (TypeError, ValueError):
            channels = None

    return AudioInfo(path=str(path), duration_sec=duration_sec, sample_rate=sample_rate, channels=channels)

def decode_pcm16(path: Path, sample_rate: int = 16000, mono: bool = True) -> tuple[np.ndarray, int]:
    """
    Decode audio with ffmpeg into signed 16-bit PCM (optionally mono) at `sample_rate`.
    Returns (float32 waveform in [-1, 1], sample_rate).
    """
    cmd = ["ffmpeg", "-v", "error", "-i", str(path), "-f", "s16le", "-ar", str(sample_rate)]
    if mono:
        cmd += ["-ac", "1"]
    cmd += ["-"]

    proc = subprocess.run(cmd, check=True, capture_output=True)
    pcm = np.frombuffer(proc.stdout, dtype=np.int16)
    if pcm.size == 0:
        return np.zeros((0,), dtype=np.float32), sample_rate
    wav = pcm.astype(np.float32) / 32768.0
    return wav, sample_rate

def compute_qc(path: Path, sample_rate: int = 16000) -> AudioQCMetrics:
    wav, sr = decode_pcm16(path, sample_rate=sample_rate, mono=True)
    duration_sec = float(wav.shape[0]) / float(sr) if sr else 0.0
    if wav.size == 0:
        return AudioQCMetrics(duration_sec=0.0, rms_dbfs=float("-inf"), peak_dbfs=float("-inf"), clipping_fraction=0.0)

    peak = float(np.max(np.abs(wav)))
    rms = float(np.sqrt(np.mean(np.square(wav))))

    peak_dbfs = 20.0 * float(np.log10(max(peak, 1e-12)))
    rms_dbfs = 20.0 * float(np.log10(max(rms, 1e-12)))

    clipping_threshold = 0.999
    clipping_fraction = float(np.mean(np.abs(wav) >= clipping_threshold))

    return AudioQCMetrics(
        duration_sec=duration_sec,
        rms_dbfs=rms_dbfs,
        peak_dbfs=peak_dbfs,
        clipping_fraction=clipping_fraction,
    )

def normalize_audio(
    in_path: Path,
    out_path: Path,
    sample_rate: int = 16000,
    mono: bool = True,
    trim_silence: bool = True,
    silence_threshold_db: float = -40.0,
    min_silence_sec: float = 0.15,
) -> Path:
    """
    Convert audio to 16kHz mono WAV and optionally trim leading/trailing silence.
    """
    out_path.parent.mkdir(parents=True, exist_ok=True)

    cmd = ["ffmpeg", "-v", "error", "-y", "-i", str(in_path)]

    if trim_silence:
        # Trim leading/trailing silence; tuned for already-clean segments.
        # https://ffmpeg.org/ffmpeg-filters.html#silenceremove
        thr = f"{silence_threshold_db}dB"
        sil = (
            f"silenceremove=start_periods=1:start_duration={min_silence_sec}:start_threshold={thr}:"
            f"stop_periods=1:stop_duration={min_silence_sec}:stop_threshold={thr}"
        )
        cmd += ["-af", sil]

    cmd += ["-ar", str(sample_rate)]
    if mono:
        cmd += ["-ac", "1"]
    cmd += ["-c:a", "pcm_s16le", str(out_path)]

    subprocess.run(cmd, check=True)
    return out_path


Summary: auto


**Evaluating audio processing methods**

I'm thinking about how audio processing relies on ffmpeg's silenceremove to trim silence. There's a risk of cutting off low-energy speech if the threshold is set too high, especially for Indian languages. A threshold of around -40dB might work, but we have to be careful with VAD trimming as it could remove important sounds like leading consonants. We need to evaluate this carefully, especially since incorrect trimming can negatively impact word error rates. Additionally, using decode_pcm16 may have memory overhead for longer segments, but it's manageable.


sed -n '1,260p' maya_transcribe/rover.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from typing import Any

from maya_transcribe.align import align_tokens, edit_distance
from maya_transcribe.textnorm import tokenize_words

@dataclass(frozen=True)
class RoverDisagreement:
    kind: str  # "token" | "gap"
    index: int
    chosen: Any
    options: Any

@dataclass(frozen=True)
class RoverResult:
    text: str
    skeleton_key: str
    disagreement_rate: float
    disagreements: list[RoverDisagreement]

def _avg_norm_distance(tokens_a: list[str], tokens_b: list[str]) -> float:
    if not tokens_a and not tokens_b:
        return 0.0
    denom = max(1, (len(tokens_a) + len(tokens_b)) // 2)
    return float(edit_distance(tokens_a, tokens_b)) / float(denom)

def choose_centroid(hyps: dict[str, list[str]]) -> str:
    keys = list(hyps.keys())
    if not keys:
        raise ValueError("No hypotheses provided")
    if len(keys) == 1:
        return keys[0]

    avg: dict[str, float] = {}
    for k in keys:
        dsum = 0.0
        for k2 in keys:
            if k2 == k:
                continue
            dsum += _avg_norm_distance(hyps[k], hyps[k2])
        avg[k] = dsum / float(len(keys) - 1)
    return min(avg.items(), key=lambda kv: kv[1])[0]

def rover_combine(
    hypotheses: dict[str, str],
    weights: dict[str, float] | None = None,
    min_gap_support: float = 0.5,
) -> RoverResult:
    """
    Lightweight ROVER-style combination:
    - pick a centroid hypothesis as skeleton
    - align others to skeleton
    - vote per skeleton token position
    - for gap insertions, keep a centroid insertion sequence if it has >= min_gap_support weight
    """
    if not hypotheses:
        return RoverResult(text="", skeleton_key="", disagreement_rate=0.0, disagreements=[])

    weights = weights or {}
    tokenized: dict[str, list[str]] = {k: tokenize_words(v) for k, v in hypotheses.items()}
    skel_key = choose_centroid(tokenized)
    skel_tokens = tokenized[skel_key]

    keys = list(tokenized.keys())
    total_weight = sum(float(weights.get(k, 1.0)) for k in keys)
    if total_weight <= 0:
        total_weight = float(len(keys))

    aligned_tokens: dict[str, list[str | None]] = {k: [None] * len(skel_tokens) for k in keys}
    gap_inserts: dict[str, list[list[str]]] = {k: [[] for _ in range(len(skel_tokens) + 1)] for k in keys}

    # Skeleton aligns to itself.
    for j, tok in enumerate(skel_tokens):
        aligned_tokens[skel_key][j] = tok

    # Align all others to skeleton.
    for k in keys:
        if k == skel_key:
            continue
        hyp_tokens = tokenized[k]
        alignment = align_tokens(skel_tokens, hyp_tokens).aligned

        skel_pos = 0
        for ref_tok, hyp_tok in alignment:
            if ref_tok is None:
                if hyp_tok is not None:
                    gap_inserts[k][skel_pos].append(hyp_tok)
                continue
            # ref_tok is not None
            if skel_pos >= len(skel_tokens):
                break
            aligned_tokens[k][skel_pos] = hyp_tok
            skel_pos += 1

    disagreements: list[RoverDisagreement] = []

    # Combine gap insertions.
    chosen_gaps: list[list[str]] = [[] for _ in range(len(skel_tokens) + 1)]
    for gap_idx in range(len(skel_tokens) + 1):
        sequences: list[tuple[str, tuple[str, ...], float]] = []
        present_weight = 0.0
        for k in keys:
            w = float(weights.get(k, 1.0))
            seq = tuple(gap_inserts[k][gap_idx])
            if seq:
                present_weight += w
            sequences.append((k, seq, w))

        if present_weight < (min_gap_support * total_weight):
            chosen_gaps[gap_idx] = []
            continue

        # Choose centroid sequence among non-empty candidates (weighted by average distance).
        candidates = [s for s in sequences if s[1]]
        if not candidates:
            chosen_gaps[gap_idx] = []
            continue

        best_seq: tuple[str, ...] | None = None
        best_score: float | None = None
        for _, seq, _w in candidates:
            score = 0.0
            for _, seq2, w2 in candidates:
                score += _avg_norm_distance(list(seq), list(seq2)) * w2
            score /= max(1e-9, present_weight)
            if best_score is None or score < best_score:
                best_score = score
                best_seq = seq

        chosen = list(best_seq or ())
        chosen_gaps[gap_idx] = chosen

        unique = {s[1] for s in candidates}
        if len(unique) > 1:
            disagreements.append(
                RoverDisagreement(
                    kind="gap",
                    index=gap_idx,
                    chosen=chosen,
                    options={k: list(seq) for k, seq, _w in candidates},
                )
            )

    # Combine skeleton positions.
    chosen_tokens: list[str] = []
    for j in range(len(skel_tokens)):
        votes: dict[str, float] = {}
        for k in keys:
            tok = aligned_tokens[k][j]
            if tok is None:
                continue
            tok = tok.strip()
            if not tok:
                continue
            votes[tok] = votes.get(tok, 0.0) + float(weights.get(k, 1.0))

        if not votes:
            continue

        max_vote = max(votes.values())
        top = [t for t, v in votes.items() if v == max_vote]
        if len(top) == 1:
            chosen_tok = top[0]
        else:
            chosen_tok = skel_tokens[j] if skel_tokens[j] in top else sorted(top)[0]

        if len(votes) > 1:
            disagreements.append(
                RoverDisagreement(kind="token", index=j, chosen=chosen_tok, options=dict(sorted(votes.items())))
            )

        chosen_tokens.append(chosen_tok)

    # Assemble final transcript with gap insertions.
    final_tokens: list[str] = []
    for j in range(len(skel_tokens)):
        final_tokens.extend(chosen_gaps[j])
        if j < len(chosen_tokens):
            final_tokens.append(chosen_tokens[j])
    final_tokens.extend(chosen_gaps[len(skel_tokens)])

    text = " ".join(final_tokens).strip()
    total_positions = len(skel_tokens) + (len(skel_tokens) + 1)
    disagreement_rate = float(len(disagreements)) / float(max(1, total_positions))

    return RoverResult(
        text=text,
        skeleton_key=skel_key,
        disagreement_rate=disagreement_rate,
        disagreements=disagreements,
    )


**Evaluating tokenization process**

I’m looking into the ROVER implementation and how it uses word tokenization from a function that likely splits based on whitespace and other factors. For Indic scripts, whitespace is present, but punctuation can also be tricky. It’s good to remember that the earlier normalize_text function may remove punctuation, which could affect how tokenization works. I think I need to inspect the textnorm.py file closely to get a clearer picture of this process.


sed -n '1,260p' maya_transcribe/textnorm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import unicodedata

_ZERO_WIDTH = {
    "\u200b",  # zero width space
    "\u200c",  # zero width non-joiner
    "\u200d",  # zero width joiner
    "\ufeff",  # byte order mark
}

def normalize_text(text: str, lang: str | None = None) -> str:
    """
    Conservative normalization for hypothesis alignment/comparison (not for publishing).
    - Unicode NFKC
    - Drop punctuation
    - Normalize all Unicode digits to ASCII
    - Collapse whitespace
    - Lowercase for English only
    """
    if not text:
        return ""

    text = unicodedata.normalize("NFKC", text)

    out_chars: list[str] = []
    for ch in text:
        if ch in _ZERO_WIDTH:
            continue

        cat = unicodedata.category(ch)
        if cat.startswith("P"):
            continue

        if cat == "Zs" or ch in ("\t", "\n", "\r"):
            out_chars.append(" ")
            continue

        if cat == "Nd":
            try:
                out_chars.append(str(unicodedata.digit(ch)))
                continue
            except (TypeError, ValueError):
                pass

        out_chars.append(ch)

    norm = "".join(out_chars)
    norm = " ".join(norm.split())

    if (lang or "").lower() in {"en", "eng"}:
        norm = norm.lower()

    return norm

def tokenize_words(text: str) -> list[str]:
    text = " ".join((text or "").split())
    if not text:
        return []
    return text.split(" ")


**Normalizing characters**

I’m thinking about normalization steps like NFKC being fine, and how dropping punctuation can help with word alignment. Plus, normalizing Unicode digits to ASCII digits works well. Collapsing whitespace seems useful too. I should lower case for English, but only when the predicted language code is 'en'. For code-switched segments, if “hi” is predicted, it won’t lowercase English, which could matter for alignment. Script normalization might be an issue for Indic languages, and tokenization by spaces is essential.


**Considering Indic script details**

I'm evaluating how ZWJ (zero-width joiner) and ZWNJ (zero-width non-joiner) functions in Indic scripts. They might be significant, especially in languages like Malayalam, but it seems manageable overall. Now, verifying the alignment process is pretty straightforward since it doesn't involve forced alignment. However, I notice there's align.py that could be useful for token alignment, perhaps specifically for ROVER. I’ll need to check align.py to see how it might fit into my approach.


sed -n '1,260p' maya_transcribe/align.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass

@dataclass(frozen=True)
class AlignmentResult:
    distance: int
    aligned: list[tuple[str | None, str | None]]

def align_tokens(ref: list[str], hyp: list[str]) -> AlignmentResult:
    """
    Levenshtein alignment between token sequences.
    Returns edit distance and an alignment as (ref_token|None, hyp_token|None) pairs.
    """
    n = len(ref)
    m = len(hyp)
    dp = [[0] * (m + 1) for _ in range(n + 1)]

    for i in range(1, n + 1):
        dp[i][0] = i
    for j in range(1, m + 1):
        dp[0][j] = j

    for i in range(1, n + 1):
        ri = ref[i - 1]
        for j in range(1, m + 1):
            hj = hyp[j - 1]
            sub_cost = 0 if ri == hj else 1
            dp[i][j] = min(
                dp[i - 1][j] + 1,  # deletion
                dp[i][j - 1] + 1,  # insertion
                dp[i - 1][j - 1] + sub_cost,  # substitution/match
            )

    aligned: list[tuple[str | None, str | None]] = []
    i, j = n, m
    while i > 0 or j > 0:
        if i > 0 and j > 0:
            ri = ref[i - 1]
            hj = hyp[j - 1]
            sub_cost = 0 if ri == hj else 1
            if dp[i][j] == dp[i - 1][j - 1] + sub_cost:
                aligned.append((ri, hj))
                i -= 1
                j -= 1
                continue
        if i > 0 and dp[i][j] == dp[i - 1][j] + 1:
            aligned.append((ref[i - 1], None))
            i -= 1
            continue
        if j > 0 and dp[i][j] == dp[i][j - 1] + 1:
            aligned.append((None, hyp[j - 1]))
            j -= 1
            continue

        # Fallback (shouldn't happen): prefer diagonal, then deletion, then insertion.
        if i > 0 and j > 0:
            aligned.append((ref[i - 1], hyp[j - 1]))
            i -= 1
            j -= 1
        elif i > 0:
            aligned.append((ref[i - 1], None))
            i -= 1
        else:
            aligned.append((None, hyp[j - 1]))
            j -= 1

    aligned.reverse()
    return AlignmentResult(distance=dp[n][m], aligned=aligned)

def edit_distance(a: list[str], b: list[str]) -> int:
    return align_tokens(a, b).distance


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Any

from maya_transcribe.language import to_whisper_lang

@dataclass(frozen=True)
class ASRSegment:
    start: float
    end: float
    text: str
    avg_logprob: float | None = None
    no_speech_prob: float | None = None
    compression_ratio: float | None = None

@dataclass(frozen=True)
class ASRResult:
    model: str
    text: str
    language: str | None
    language_probability: float | None
    segments: list[ASRSegment]
    meta: dict[str, Any]

class FasterWhisperASR:
    def __init__(
        self,
        model_size_or_path: str = "large-v3",
        device: str | None = None,
        compute_type: str | None = None,
    ) -> None:
        try:
            from faster_whisper import WhisperModel
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "faster-whisper is not installed. Install with: pip install -e '.[asr]'"
            ) from e

        if device is None:
            try:
                import torch

                device = "cuda" if torch.cuda.is_available() else "cpu"
            except Exception:
                device = "cpu"

        if compute_type is None:
            compute_type = "float16" if device == "cuda" else "float32"

        self.model_size_or_path = model_size_or_path
        self.device = device
        self.compute_type = compute_type
        self._model = WhisperModel(model_size_or_path, device=device, compute_type=compute_type)

    @property
    def name(self) -> str:
        return f"faster-whisper:{self.model_size_or_path}"

    def transcribe(
        self,
        audio_path: Path,
        language: str | None = None,
        beam_size: int = 5,
        best_of: int = 5,
        temperature: float = 0.0,
        vad_filter: bool = False,
    ) -> ASRResult:
        lang = to_whisper_lang(language)

        segments_iter, info = self._model.transcribe(
            str(audio_path),
            language=lang,
            beam_size=beam_size,
            best_of=best_of,
            temperature=temperature,
            vad_filter=vad_filter,
        )

        segments: list[ASRSegment] = []
        texts: list[str] = []
        for s in segments_iter:
            texts.append(s.text)
            segments.append(
                ASRSegment(
                    start=float(s.start),
                    end=float(s.end),
                    text=str(s.text),
                    avg_logprob=getattr(s, "avg_logprob", None),
                    no_speech_prob=getattr(s, "no_speech_prob", None),
                    compression_ratio=getattr(s, "compression_ratio", None),
                )
            )

        return ASRResult(
            model=self.name,
            text="".join(texts).strip(),
            language=getattr(info, "language", None),
            language_probability=getattr(info, "language_probability", None),
            segments=segments,
            meta={
                "device": self.device,
                "compute_type": self.compute_type,
                "beam_size": beam_size,
                "best_of": best_of,
                "temperature": temperature,
                "vad_filter": vad_filter,
            },
        )

class TransformersWhisperASR:
    """
    Whisper-family ASR using HuggingFace Transformers (useful for fine-tuned checkpoints like IndicWhisper).
    """

    def __init__(
        self,
        model_id: str,
        device: str | None = None,
        dtype: str | None = None,
    ) -> None:
        try:
            import torch
            from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "TransformersWhisperASR requires the optional dependencies: torch + transformers. "
                "Install with: pip install -e '.[torch]'"
            ) from e

        if device is None:
            device = "cuda" if torch.cuda.is_available() else "cpu"
        if dtype is None:
            dtype = "float16" if (device == "cuda") else "float32"

        torch_dtype = torch.float16 if dtype == "float16" else torch.float32

        self.model_id = model_id
        self.device = device
        self.dtype = dtype
        self._torch = torch
        self._processor = AutoProcessor.from_pretrained(model_id)
        self._model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype).to(device)
        self._model.eval()

    @property
    def name(self) -> str:
        return f"transformers:{self.model_id}"

    def transcribe(
        self,
        audio_path: Path,
        language: str | None = None,
        max_new_tokens: int = 256,
        num_beams: int = 5,
    ) -> ASRResult:
        from maya_transcribe.audio import decode_pcm16

        wav, sr = decode_pcm16(audio_path, sample_rate=16000, mono=True)
        if wav.size == 0:
            return ASRResult(
                model=self.name,
                text="",
                language=language,
                language_probability=None,
                segments=[],
                meta={"max_new_tokens": max_new_tokens, "num_beams": num_beams},
            )

        inputs = self._processor(wav, sampling_rate=sr, return_tensors="pt")
        input_features = inputs.get("input_features")
        if input_features is None:
            raise RuntimeError("Processor did not return input_features")
        input_features = input_features.to(self.device)

        forced_decoder_ids = None
        whisper_lang = to_whisper_lang(language)
        if whisper_lang:
            get_ids = getattr(self._processor, "get_decoder_prompt_ids", None)
            if callable(get_ids):
                forced_decoder_ids = get_ids(language=whisper_lang, task="transcribe")

        with self._torch.no_grad():
            generated_ids = self._model.generate(
                input_features,
                forced_decoder_ids=forced_decoder_ids,
                max_new_tokens=max_new_tokens,
                num_beams=num_beams,
            )

        text = self._processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
        return ASRResult(
            model=self.name,
            text=text.strip(),
            language=whisper_lang or language,
            language_probability=None,
            segments=[],
            meta={
                "device": self.device,
                "dtype": self.dtype,
                "max_new_tokens": max_new_tokens,
                "num_beams": num_beams,
            },
        )


**Considering model integration**

I'm looking into how TransformersWhisperASR can use fine-tuned checkpoints, like IndicWhisper, which is good. However, it’s not integrated by default, so the user might need to pass extra models through the CLI. I'll also need to check the to_whisper_lang mapping in language.py. It seems there’s a bit of extra work involved for the user.


sed -n '1,260p' maya_transcribe/language.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

ISO3_TO_WHISPER_LANG: dict[str, str] = {
    "eng": "en",
    "hin": "hi",
    "mar": "mr",
    "guj": "gu",
    "pan": "pa",
    "tam": "ta",
    "tel": "te",
    "kan": "kn",
    "mal": "ml",
    "ben": "bn",
    "asm": "as",
    "ori": "or",
    "ory": "or",
}

def to_whisper_lang(lang_code: str | None) -> str | None:
    if not lang_code:
        return None
    lang_code = lang_code.strip().lower()
    if len(lang_code) == 2:
        return lang_code
    return ISO3_TO_WHISPER_LANG.get(lang_code, None)


**Mapping language codes**

I'm mapping ISO3 codes to whisper, but I notice that MMS-LID output labels might not align with ISO3 or two-letter codes. They could be names like 'telugu' or 'te.' I need to verify that. For MMS-LID, id2label might reference language names or ISO formats. In Hugging Face, mms-lid-256 covers 256 languages. When using pipeline, route_lang is set to top1.code, which then passes to faster-whisper that expects two-letter codes. So, I just realized that mapping looks important!


**Addressing mapping challenges**

I’m noticing an issue where if the MMS-LID label is 'hindi' or 'Hindi', the mapping fails because to_whisper_lang returns None. This means that whisper might default to auto-detecting the language, which could degrade performance. So, I really need a strong label mapping for MMS-LID codes. Additionally, when using segment_id = src.stem, there could be duplicate stems leading to file overwrites since different segments may have the same name across directories. It's essential to ensure unique segment IDs by including the relative path to avoid these conflicts!


**Handling audio processing challenges**

It returns a sorted set of audio files and uses a set to eliminate duplicates, which is great! The `normalize_audio` function converts files to WAV and writes them to the specified output directory, overwriting existing files if needed. However, if ffmpeg is missing, it could cause the pipeline to fail. For language identification (LID), if MMSLID fails to load, the pipeline continues, marking the reason as "lid_missing." Interestingly, segments can still be accepted even without LID, which might risk incorrect language transcriptions in a multilingual dataset, and this is something I need to keep an eye on!


**Considering language detection requirements**

ASR models can automatically detect languages, but I think it might be better to require LID instead, or at least treat it as a human interaction. Currently, there are no-speech probabilities in the ASR summary, but I noticed that with TransformersWhisperASR, the segments list is empty and there are no no-speech probabilities available. So, the no-speech check should rely on those probabilities from faster-whisper. It feels important to ensure this functionality is robust!


**Identifying edge cases for ASR**

I'm noticing some edge cases that might affect language detection and overall recognition in ASR. Very short segments (like 1 second) may struggle with accurate LID, but pipeline min_duration defaults to 0.8 seconds, which is promising.

For code-switching, route_lang might misrepresent languages since whisper's auto-detection could choose either Hindi or English. Specifying language can help but might impact recognition negatively. The pipeline uses route_lang only if LID is confident, which is interesting.

Additionally, laughter or breaths in segments could trigger odd results when using silenceremove, and segments with low volume need better verification processes. I also need to consider issues like background noise, repeated phrases, and the model hallucinating during silent periods. Adjusting weights in the model could be another consideration!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from jiwer import wer

from maya_transcribe.io import read_jsonl
from maya_transcribe.textnorm import normalize_text

@dataclass(frozen=True)
class EvalRow:
    segment_id: str
    language: str | None
    decision: str | None
    ref: str
    hyp: str
    wer: float

def load_refs(refs_path: Path) -> dict[str, dict[str, Any]]:
    """
    Read a JSONL file of references.
    Expected keys (any one): `text`, `transcript`, `reference`.
    Required key: `segment_id` (or `id`).
    """
    refs: dict[str, dict[str, Any]] = {}
    for row in read_jsonl(refs_path):
        segment_id = str(row.get("segment_id") or row.get("id") or "").strip()
        if not segment_id:
            continue
        refs[segment_id] = row
    return refs

def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    for k in keys:
        v = row.get(k)
        if isinstance(v, str) and v.strip():
            return v
    return None

def evaluate(results_path: Path, refs_path: Path, out_path: Path | None = None) -> dict[str, Any]:
    refs = load_refs(refs_path)
    results = read_jsonl(results_path)

    rows: list[EvalRow] = []
    for r in results:
        segment_id = str(r.get("segment_id") or "").strip()
        if not segment_id or segment_id not in refs:
            continue

        ref_row = refs[segment_id]
        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
        if ref_text is None:
            continue

        # Use ROVER output as hypothesis by default.
        hyp_text = None
        rover = r.get("rover") or {}
        if isinstance(rover, dict):
            hyp_text = rover.get("text")
        if not isinstance(hyp_text, str):
            continue

        lang = None
        route = r.get("route") or {}
        if isinstance(route, dict):
            lang = route.get("language_hint")

        decision = None
        ver = r.get("verification") or {}
        if isinstance(ver, dict):
            decision = ver.get("decision")

        ref_norm = normalize_text(ref_text, lang=lang)
        hyp_norm = normalize_text(hyp_text, lang=lang)
        w = float(wer(ref_norm, hyp_norm))

        rows.append(
            EvalRow(
                segment_id=segment_id,
                language=lang,
                decision=str(decision) if decision is not None else None,
                ref=ref_text,
                hyp=hyp_text,
                wer=w,
            )
        )

    by_lang: dict[str, list[float]] = {}
    by_dec: dict[str, list[float]] = {}
    for row in rows:
        if row.language:
            by_lang.setdefault(row.language, []).append(row.wer)
        if row.decision:
            by_dec.setdefault(row.decision, []).append(row.wer)

    summary = {
        "pairs_evaluated": len(rows),
        "wer_mean": (sum(r.wer for r in rows) / len(rows)) if rows else None,
        "wer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang.items())},
        "wer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec.items())},
    }

    if out_path is not None:
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")

    return summary


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from pathlib import Path

import typer

from maya_transcribe.audio import compute_qc, ffprobe, normalize_audio
from maya_transcribe.config import Settings
from maya_transcribe.dataset import find_segments_dir
from maya_transcribe.io import write_jsonl
from maya_transcribe.r2 import R2Client
from maya_transcribe.tar import safe_extract_tar

app = typer.Typer(add_completion=False, no_args_is_help=True)

@app.command()
def download(
    video_id: str = typer.Option(..., help="YouTube video id (used to locate the tar in R2)."),
    out: Path = typer.Option(Path("data"), help="Output directory."),
    r2_key: str | None = typer.Option(None, help="Exact R2 object key (skips search)."),
    bucket: str | None = typer.Option(None, help="R2 bucket override (defaults to env R2_BUCKET)."),
    prefix: str | None = typer.Option(None, help="Optional R2 prefix override."),
    use_db: bool = typer.Option(False, help="Fetch `r2_tar_key` from Supabase Postgres (DATABASE_URL)."),
    dry_run: bool = typer.Option(False, help="Only show matched key(s); do not download."),
) -> None:
    """
    Download the cleaned/segmented tar for a video from R2 and extract it.
    """
    settings = Settings.load()
    r2 = R2Client.from_settings(settings)
    if bucket:
        r2 = R2Client(
            endpoint_url=r2.endpoint_url,
            access_key_id=r2.access_key_id,
            secret_access_key=r2.secret_access_key,
            bucket=bucket,
        )

    out.mkdir(parents=True, exist_ok=True)
    r2_prefix = prefix if prefix is not None else settings.r2_prefix

    matches: list[str]
    if use_db and not r2_key:
        from maya_transcribe.db import fetch_video_meta

        meta = fetch_video_meta(video_id)
        if meta and meta.r2_tar_key:
            r2_key = meta.r2_tar_key

    if r2_key:
        matches = [r2_key]
    else:
        best = r2.find_video_tar_key(video_id=video_id, prefix=r2_prefix)
        if not best:
            typer.echo("No matching objects found in R2.")
            raise typer.Exit(code=2)
        matches = [best]

    if dry_run:
        for key in matches:
            typer.echo(key)
        return

    key = matches[0]
    tar_path = r2.download_to(key=key, out_dir=out / video_id)
    extract_dir = out / video_id
    safe_extract_tar(tar_path, extract_dir=extract_dir)
    segments_dir = find_segments_dir(extract_dir)
    typer.echo(f"Downloaded: {tar_path}")
    typer.echo(f"Extracted to: {extract_dir}")
    if segments_dir:
        typer.echo(f"Segments dir: {segments_dir}")

@app.command()
def preprocess(
    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
    out: Path = typer.Option(Path("outputs/preprocessed"), help="Output directory for normalized audio + QC."),
    sample_rate: int = typer.Option(16000, help="Target sample rate."),
    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
    silence_threshold_db: float = typer.Option(-40.0, help="Silence threshold in dB for trimming."),
    min_silence_sec: float = typer.Option(0.15, help="Min silence duration for trimming."),
) -> None:
    """
    Normalize audio to 16kHz mono WAV, optionally trim silence, and write QC metrics.
    """
    out.mkdir(parents=True, exist_ok=True)

    audio_files: list[Path] = []
    for ext in ("*.wav", "*.mp3", "*.m4a", "*.flac", "*.ogg", "*.opus"):
        audio_files.extend(sorted(segments_dir.rglob(ext)))
    audio_files = sorted(set(audio_files))

    if not audio_files:
        typer.echo(f"No audio files found under: {segments_dir}")
        raise typer.Exit(code=2)

    rows = []
    norm_dir = out / "audio"
    for src in audio_files:
        segment_id = src.stem
        dst = norm_dir / f"{segment_id}.wav"
        normalize_audio(
            src,
            dst,
            sample_rate=sample_rate,
            mono=True,
            trim_silence=trim_silence,
            silence_threshold_db=silence_threshold_db,
            min_silence_sec=min_silence_sec,
        )
        info = ffprobe(dst)
        qc = compute_qc(dst, sample_rate=sample_rate)
        rows.append(
            {
                "segment_id": segment_id,
                "src_path": str(src),
                "wav_path": str(dst),
                "ffprobe": {
                    "duration_sec": info.duration_sec,
                    "sample_rate": info.sample_rate,
                    "channels": info.channels,
                },
                "qc": {
                    "duration_sec": qc.duration_sec,
                    "rms_dbfs": qc.rms_dbfs,
                    "peak_dbfs": qc.peak_dbfs,
                    "clipping_fraction": qc.clipping_fraction,
                },
            }
        )

    write_jsonl(out / "qc.jsonl", rows)
    typer.echo(f"Wrote: {out / 'qc.jsonl'}")
    typer.echo(f"Normalized audio: {norm_dir}")

@app.command()
def lid(
    audio: Path = typer.Option(..., exists=True, dir_okay=False, help="Audio file path."),
    model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace model id."),
    top_k: int = typer.Option(5, help="Return top-K predictions."),
) -> None:
    """
    Spoken language identification (MMS-LID).
    """
    from maya_transcribe.lid import MMSLID

    model = MMSLID(model_id=model_id)
    res = model.predict(audio_path=audio, top_k=top_k)
    for p in res.predictions:
        typer.echo(f"{p.code}\t{p.prob:.4f}")

@app.command()
def asr(
    audio: Path = typer.Option(..., exists=True, dir_okay=False, help="Audio file path."),
    language: str | None = typer.Option(None, help="Language hint (ISO-639-1 or ISO-639-3)."),
    model: str = typer.Option("large-v3", help="Whisper model size or path for faster-whisper."),
    beam_size: int = typer.Option(5, help="Beam size."),
    best_of: int = typer.Option(5, help="Best-of (sampling candidates)."),
    temperature: float = typer.Option(0.0, help="Temperature."),
) -> None:
    """
    Transcribe a single file with faster-whisper.
    """
    from maya_transcribe.asr import FasterWhisperASR

    asr_backend = FasterWhisperASR(model_size_or_path=model)
    res = asr_backend.transcribe(
        audio_path=audio, language=language, beam_size=beam_size, best_of=best_of, temperature=temperature
    )
    typer.echo(res.text)

@app.command()
def run(
    segments_dir: Path = typer.Option(..., exists=True, file_okay=False, help="Directory containing audio segments."),
    out: Path = typer.Option(Path("outputs/run"), help="Output directory."),
    whisper_model: str = typer.Option("large-v3", help="Whisper model size/path for faster-whisper."),
    lid_model_id: str = typer.Option("facebook/mms-lid-256", help="HuggingFace MMS-LID model id."),
    transformers_whisper_model: list[str] = typer.Option(
        [], help="Extra whisper-family HF models to run via transformers (repeatable)."
    ),
    limit: int | None = typer.Option(None, help="Process only the first N segments."),
) -> None:
    """
    End-to-end pipeline: preprocess -> LID -> ASR ensemble -> ROVER -> verify.
    """
    from maya_transcribe.pipeline import run_pipeline

    results_path = run_pipeline(
        segments_dir=segments_dir,
        out_dir=out,
        lid_model_id=lid_model_id,
        whisper_model=whisper_model,
        extra_transformers_whisper_models=transformers_whisper_model,
        limit=limit,
    )
    typer.echo(f"Wrote: {results_path}")
    typer.echo(f"Wrote: {out / 'summary.json'}")

@app.command()
def eval(
    results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
    refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
    out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
) -> None:
    """
    Compute WER for `rover.text` against a reference JSONL keyed by `segment_id`.
    """
    from maya_transcribe.eval import evaluate

    summary = evaluate(results_path=results, refs_path=refs, out_path=out)
    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))

if __name__ == "__main__":
    app()


**Evaluating CLI improvements**

I’m noticing that the CLI includes functions like preprocess, lid, asr, run, and eval. However, it seems to lack support for thresholds or additional configuration—run_pipeline can handle thresholds, but the CLI doesn’t currently provide that option. This could be an area for improvement. I also see that preprocess might have a collision risk with segment_id. Let’s check to see if dataset.py can find the segments directory!


sed -n '1,240p' maya_transcribe/dataset.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from pathlib import Path

def find_segments_dir(root: Path) -> Path | None:
    """
    Return the first directory named 'segments' under `root` (including `root`).
    """
    if root.name == "segments" and root.is_dir():
        return root
    for path in root.rglob("segments"):
        if path.is_dir():
            return path
    return None


Summary: auto


sed -n '1,260p' maya_transcribe/config.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Final

from dotenv import load_dotenv

@dataclass(frozen=True)
class Settings:
    r2_endpoint_url: str
    r2_access_key_id: str
    r2_secret_access_key: str
    r2_bucket: str = "1-cleaned-data"
    r2_prefix: str | None = None

    @staticmethod
    def load(env_path: str | Path = ".env") -> "Settings":
        load_dotenv(dotenv_path=env_path, override=False)

        import os

        def required(name: str) -> str:
            value = os.getenv(name)
            if not value:
                raise RuntimeError(f"Missing required env var: {name}")
            return value

        endpoint_url: Final[str] = required("R2_ENDPOINT_URL")
        access_key_id: Final[str] = required("R2_ACCESS_KEY_ID")
        secret_access_key: Final[str] = required("R2_SECRET_ACCESS_KEY")
        bucket: Final[str] = os.getenv("R2_BUCKET") or "1-cleaned-data"
        prefix = os.getenv("R2_PREFIX") or None

        return Settings(
            r2_endpoint_url=endpoint_url,
            r2_access_key_id=access_key_id,
            r2_secret_access_key=secret_access_key,
            r2_bucket=bucket,
            r2_prefix=prefix,
        )


**Reviewing code snippets**

I need to provide some comments on the code snippets! The first snippet should ensure that it loads the key from GEMINI_KEY instead of GEMINI_API_KEY and reads the .env using python-dotenv. It's also crucial to attach audio files properly and avoid instructing the model to guess.

In the second review for the pipeline code, I suggest using a stable unique ID to handle `segment_id = src.stem` collisions and ensuring that language detection (LID) is treated correctly for quality purposes.


**Reviewing LID and ASR strategies**

I think I should suggest making the Language ID (LID) dependency required for the run, or it could degrade gracefully while marking human inputs. I also want to ensure there’s proper mapping from MMS labels to Whisper language codes, and I’ll need to verify this with tests.

For the ASR ensemble, which mostly uses Whisper, it’s important to incorporate diverse models like IndicWhisper and IndicConformer to effectively reduce Word Error Rate (WER). Additionally, calibrating ROVER weights and allowing more hypotheses might help with sampling disagreement. Lastly, normalization should possibly include Indic script normalization and remove diacritics.


**Considering training and verification processes**

I need to keep diacritics for training, especially for alignment purposes. For code-switching, the language route should be None. However, if LID is confident in a code-switched scenario, I may need to adapt how I handle it—maybe if the highest probability is above a certain threshold while the second is English, I can treat it appropriately.

The preprocessing step, like trim_silence, uses silenceremove, which might be too aggressive. I'd like to ensure we can record before and after durations and provide toggle options. Verification steps are crucial, as the current method doesn’t account for audio-grounded verification. I should implement this effectively to minimize WER. Lastly, I want to handle non-speech content detection more accurately.


**Evaluating Whisper's features and implementation**

I’m exploring how Whisper uses `compression_ratio` to detect hallucinations. They compute summaries but don’t seem to utilize them; perhaps a threshold could help reject or flag results for human review.

For TTS training, it may be beneficial to filter out segments that are too fast or have significant background noise. I'm also looking at the `results.jsonl`, which contains a large segments list; while it’s manageable, scaling might require optimizing how I store this data. I noticed that using `sorted(set(audio_files))` can lead to an unstable order with path objects, yet sorting guarantees stability. Finally, I wonder if I need to ensure deterministic seeding for consistency.


**Improving ASR evaluation and processing**

I’m thinking about an enhanced evaluation method by building "confidence metrics" and "calibration" to estimate expected Word Error Rate (WER) on unlabeled data. This could involve using regression with features like disagreement rate and LID confidence.

For TTS training, I may want to filter out segments with disfluencies and consider how to handle one-second segments differently to avoid cutting words.

The user requested a two-part response: first, a code review, and then suggestions for evaluating WER using a superior model. I’ll aim for concise bullet points in both sections, mentioning key file locations and ensuring clarity while addressing their questions on model usage and handling pseudo-references.


**Inspecting Gemini and pipeline outputs**

I want to be cautious about claiming that Gemini is the best option; I think it's wise to label it as inference and focus on "test and measure." I should check if there's any file created by another coding agent related to Gemini, like `Prompt.md`, and it looks like that exists.

Next, I'll want to review the existing pipeline output to understand how it operates, possibly by looking at the summary and results from outputs/test_run2. Examining outputs/test_run2/summary.json will help me see the counts and other important details.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "segments_total": 1,
  "counts": {
    "accept": 1,
    "human": 0,
    "reject": 0,
    "error": 0
  },
  "avg_total_sec": 1.4479315648786724,
  "p50_total_sec": 1.4479315648786724,
  "p90_total_sec": 1.4479315648786724
}


ls -la outputs/test_run2; head -n 3 outputs/test_run2/refs.jsonl


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 24
drwxrwxr-x 3 ubuntu ubuntu 4096 Jan 13 04:36 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Jan 13 04:36 ..
drwxrwxr-x 2 ubuntu ubuntu 4096 Jan 13 04:36 audio
-rw-rw-r-- 1 ubuntu ubuntu   73 Jan 13 04:36 refs.jsonl
-rw-rw-r-- 1 ubuntu ubuntu 2521 Jan 13 04:36 results.jsonl
-rw-rw-r-- 1 ubuntu ubuntu  227 Jan 13 04:36 summary.json
{"segment_id": "SPEAKER_00_0000_0.03-12.15", "text": "intricate web of"}


**Evaluating coding implementation and languages**

The user wants me to check the implementation of their other coding agent and add review comments, so I'll start by reviewing the existing code in the repository to provide robust commentary. They've mentioned language support, stating they can help with Telugu and Hindi, but for others, a stronger model is needed. I could suggest using cross-model consensus and audio verification to reduce word error rates. For robust handling, IndicWhisper can be used, along with generalist models. I wonder if multilingual translation to English might also help.


**Considering TTS training and evaluation**

To train a text-to-speech model, having a script is essential. For reference transcripts, I can use script-specific normalization, and even if I can't read certain languages, I can automatically compute character error rates (CER) and word error rates (WER), provided I have references. For manual reviews, I might suggest using bilingual editors or annotation platforms. It's worth noting that using Gemini 3 Pro is a preview model, which could impact reproducibility. Also, to minimize hallucinations in transcription tasks, I should aim for less reasoning and more straightforward approaches, like using deterministic parameters.


**Clarifying transcription guidelines**

The instructions say not to add punctuation, and while some ASR or LLMs might include it, I can enforce a rule to remove any punctuation post-processing. However, removing it could affect word error rates (WER), so evaluations need to align. They also want "proper native transcription," which requires writing English words phonetically in the native script, through transliteration. Since standards can vary, it might be better to either keep English words in Latin script or use a consistent transliteration approach for known words to reduce WER issues.


**Deciding on phonetic representation for TTS**

The user might want to include phonetic representations in the native script for TTS training, recognizing that English words may be pronounced with a Telugu accent. However, training TTS on code-switched audio can be quite complex, and it may be wise to filter out heavy code-switching. To minimize word error rates (WER), it could be better to keep English words in Latin script for consistency, though this could differ for TTS. For a multilingual dataset, separating by script and including optional romanization could be a practical approach.


**Reviewing edge cases and evaluation methods**

I need to consider several edge cases for YouTube segments. Issues like music intros, names, numerals, and interjections can confuse automatic speech recognition (ASR). Multi-speaker or overlapping speech can complicate things too, so we need a system to detect crosstalk.

For evaluation, I could propose using a gold standard with human transcripts, multi-stage evaluations for language identification and ASR model performance, and metrics for specific error types. Also, I should check if the pipeline can integrate with other models. Lastly, it’s important to ensure proper handling of environment variables to avoid issues during execution.


**Reviewing Gemini Transcription and Repo Pipeline**

I need to create structured feedback focusing on the Gemini transcription snippet and the repository pipeline. Key points include addressing missing audio attachments, ensuring environment variables like `GEMINI_KEY` are correctly loaded, clarifying prompt inconsistencies that encourage guessing, and handling output parsing and error management.

Next, for the repo, I'll identify issues with segment ID collisions, ASR model diversity, and incorporate additional scoring methods.

For evaluation upgrades, I'll suggest building a gold set for languages, enhancing precision measures, and considering verification strategies.

Lastly, for non-Telugu and non-Hindi models, I recommend the AI4Bharat IndicWhisper and similar options, while advising caution with using Gemini in validation.


python3 - <<'PY'
try:
    from google import genai
    from google.genai import types
    print('google-genai installed', genai.__version__ if hasattr(genai,'__version__') else '')
    print('types.Part attrs:', [a for a in dir(types.Part) if 'from_' in a][:20])
except Exception as e:
    print('import failed:', e)
PY


Output:
Exit code: 0
Wall time: 1.5 seconds
Output:
google-genai installed 1.54.0
types.Part attrs: ['_from_response', 'from_bytes', 'from_code_execution_result', 'from_executable_code', 'from_function_call', 'from_function_response', 'from_orm', 'from_text', 'from_uri']


**Inspecting installation details**

I just saw that google-genai version 1.54.0 is installed, which is exciting! It includes functions like Part.from_bytes and Part.from_uri, which can be really useful. I'm thinking it’s important to inspect the signature of Part.from_bytes to understand its exact usage better. I want to ensure I know how to use it properly, since knowing the details will be super helpful in the long run. Let's dive into that!


python3 - <<'PY'
from google.genai import types
import inspect
print(inspect.signature(types.Part.from_bytes))
print(types.Part.from_bytes.__doc__)
PY


Output:
Exit code: 0
Wall time: 1.4 seconds
Output:
(*, data: bytes, mime_type: str, media_resolution: Union[ForwardRef('PartMediaResolutionOrDict'), ForwardRef('PartMediaResolutionLevel'), str, NoneType] = None) -> 'Part'
None


Summary: auto


I’m figuring out how to use `types.Part.from_bytes(data=audio_bytes, mime_type="audio/wav")`. There’s also a possibility of using `from_uri` with `file://`, but that might be tricky with remote files. I think using `from_bytes` is safer. I also need clarity on how to pass `system_instruction` in the configuration—so I’ll check the signature. Introspecting `GenerateContentConfig` should help me with that! Let’s see where this leads me.


python3 - <<'PY'
from google.genai import types
import inspect
print(inspect.signature(types.GenerateContentConfig))
PY


Output:
Exit code: 0
Wall time: 1.4 seconds
Output:
(*, httpOptions: Optional[google.genai.types.HttpOptions] = None, shouldReturnHttpResponse: Optional[bool] = None, systemInstruction: Union[google.genai.types.Content, str, PIL.Image.Image, google.genai.types.File, google.genai.types.Part, list[Union[str, PIL.Image.Image, google.genai.types.File, google.genai.types.Part]], NoneType] = None, temperature: Optional[float] = None, topP: Optional[float] = None, topK: Optional[float] = None, candidateCount: Optional[int] = None, maxOutputTokens: Optional[int] = None, stopSequences: Optional[list[str]] = None, responseLogprobs: Optional[bool] = None, logprobs: Optional[int] = None, presencePenalty: Optional[float] = None, frequencyPenalty: Optional[float] = None, seed: Optional[int] = None, responseMimeType: Optional[str] = None, responseSchema: Union[dict[Any, Any], type, google.genai.types.Schema, types.GenericAlias, types.UnionType, _UnionGenericAlias, NoneType] = None, responseJsonSchema: Optional[Any] = None, routingConfig: Optional[google.genai.types.GenerationConfigRoutingConfig] = None, modelSelectionConfig: Optional[google.genai.types.ModelSelectionConfig] = None, safetySettings: Optional[list[google.genai.types.SafetySetting]] = None, tools: Optional[list[Union[google.genai.types.Tool, Callable[..., Any], mcp.types.Tool, mcp.client.session.ClientSession]]] = None, toolConfig: Optional[google.genai.types.ToolConfig] = None, labels: Optional[dict[str, str]] = None, cachedContent: Optional[str] = None, responseModalities: Optional[list[str]] = None, mediaResolution: Optional[google.genai.types.MediaResolution] = None, speechConfig: Union[str, google.genai.types.SpeechConfig, NoneType] = None, audioTimestamp: Optional[bool] = None, automaticFunctionCalling: Optional[google.genai.types.AutomaticFunctionCallingConfig] = None, thinkingConfig: Optional[google.genai.types.ThinkingConfig] = None, imageConfig: Optional[google.genai.types.ImageConfig] = None) -> None


**Evaluating configuration details**

I'm checking the configuration that uses camelCase for fields like `systemInstruction` and `thinkingConfig`. The code snippet utilizes `types.GenerateContentConfig`, which has a mix of naming conventions. While it might still work with Pydantic, it's best to stick with clarity in naming. Also, content is being passed correctly, but there's the need for an audio part and perhaps to set `responseMimeType` as "text/plain." It's essential to minimize hallucinations by adjusting parameters like temperature and top_p, while considering audio timestamping for evaluation.


python3 - <<'PY'
from google.genai import types
import inspect
print(inspect.signature(types.ThinkingConfig))
print([a for a in dir(types.ThinkingConfig) if 'thinking' in a.lower()][:20])
PY


Output:
Exit code: 0
Wall time: 1.4 seconds
Output:
(*, includeThoughts: Optional[bool] = None, thinkingBudget: Optional[int] = None, thinkingLevel: Optional[google.genai.types.ThinkingLevel] = None) -> None
[]


**Evaluating Gemini output preparation**

I need to ensure that if Gemini outputs in native script, it’s consistent for unification. For languages I don’t speak, I’ll treat them carefully. The user mentioned that the "API key is in .env as GEMINI_KEY,” and I need to include robust instructions as per Prompt.md. It seems like the other agent wrote the code but didn’t load .env. I should craft a final answer accordingly and check for any potential pipeline failure or error handling issues in the script.


**Analyzing the run_pipeline issues**

In the `run_pipeline`, I see that they compute counts and timings while treating decision variables. If the decision is unknown, the counts update, but `verify_segment` returns accept/human/reject. There's a bug in the `counts` dictionary regarding the keys; they update counts with `counts[decision] = counts.get(decision, 0) + 1`, but it seems like they previously set counts as well. If a verify decision is something different, it could lead to new keys being added without summary. Notably, they don't incorporate thresholds or a config file for robust gating, which should be driven by configurations. Plus, there's no multi-language ensemble implementation, although they can pass extra transformer models. Lastly, I’ve noticed an issue in `maya_transcribe/verify.py`, where the `lid_confidence` function returns false confidence despite a computed margin. I need to think of a sophisticated way to evaluate this pipeline.


**Proposing additional metrics**

I can propose several additional metrics to enhance evaluation. First, we could look at the Segment Error Rate (SER), which is the percentage of segments with any error. For training, maybe we want a Word Error Rate (WER) of less than 1%. It could also be weighted by duration.

We should consider evaluating phoneme error for TTS and different error types, like substitution versus deletion, which jiwer can help with. Separately assessing code-switching segments and LID accuracy from references could be useful too.

Additionally, we could build a curve for gating, comparing acceptance thresholds against WER to choose the best one.

I’m thinking about using "adversarial evaluation" through manual reviews of the most problematic segments, focusing on those with high disagreement or low log probability, capturing human transcripts to measure WER and identify failure points. Lastly, implementing "stratified cross-validation" seems like a solid plan. For Telugu and Hindi, I might assist, but we need a top-performing model for other languages.


**Suggesting tech solutions**

I suggest using AI4Bharat IndicWhisper, which covers 12 languages. This might fit the user's needs, particularly for Punjabi and Gujarati, and possibly for Assamese as well.

For additional quality, I can incorporate IndicConformer into the code pipeline via Nemo or transformers, although that isn't set up yet.

Another promising option is Meta's MMS ASR, but it's large and less accurate.

Also, the user mentioned a "superior model like Gemini-3-pro." However, for languages they don't speak, framing it as "superior" isn’t quite right. A strong ensemble plus verification is better, with Gemini as an addition, although human evaluation is still essential for some languages. For those where human review isn't feasible, we can rely on running the pipeline and gating, but we should sample and have humans for quality control. Lastly, I see they've mentioned minimizing WER, which we had previously softened to avoid claiming a global best.


**Crafting a comprehensive response**

I need to emphasize measuring the dataset effectively while including disclaimers. The user asked to finish the code review first and suggest evaluations for WER checks with a superior model.

I’ll structure the response to include a code review for Gemini with around 8 bullet points and a 10-bullet review for the maya-transcribe pipeline. The evaluation plan will have 8 bullets, incorporating Gemini as needed.

Regarding prompt robustness, we could propose key changes like stricter transcription guidelines and transliteration options without lengthy prompts.

Also, I should mention disabling the "ThinkingConfig thinking_level HIGH" to avoid unwanted reasoning outputs. For model context, short audio segments are manageable, but streaming may not be necessary for 1-second clips. Finally, I must remember not to include emojis in the answer.


**Code Review (Gemini snippet + prompt)**
- No audio is actually sent: `contents` only has `INSERT_INPUT_HERE`; you need a `types.Part.from_bytes(data=..., mime_type=...)` audio part or the model will “transcribe” nothing.
- Env var mismatch: `.env` has `GEMINI_KEY`, but code reads `GEMINI_API_KEY`; also load `.env` (e.g., `python-dotenv`) or require the key to be exported.
- Prompt is self-contradictory and *encourages hallucination*: “fix the grammar / predict what comes next” directly violates “No inference”; for WER + training data, you want **“if unsure, output `<unk>`/`<inaudible>` and do not guess.”**
- You say “four outputs” but only define Output 1; many models will invent Outputs 2–4 → extra garbage tokens.
- For code-switching, forcing “primary language: telugu” + “write English phonetically in Telugu” creates **non-standard spellings** and high apparent WER unless your references follow the same convention; pick one convention and keep it consistent across all data + eval.
- Add determinism + safety: set `temperature=0`, cap `maxOutputTokens`, set `responseMimeType="text/plain"`, and validate the output strictly (reject if it contains extra sections, punctuation if forbidden, or unlabelled text).
- Streaming: guard against `chunk.text is None` and capture full text for audit; add retries/backoff for transient API failures.
- Use Gemini as *one more hypothesis*, not truth: always run the same verification gates (agreement + audio-grounded scoring) before accepting.

**Code Review (repo pipeline robustness: `maya_transcribe/`)**
- High-risk ID collision: `segment_id = src.stem` can overwrite outputs if stems repeat across folders; use a stable unique id (e.g., relative path hash) for `wav_path` + keys.
- LID silently optional: if `MMSLID` can’t load, pipeline still “accepts” with `lid_missing`; for quality-first, treat missing LID as **`human` or `reject`** (or fail fast).
- Ensemble isn’t diverse yet: current “ensemble” is Whisper det + Whisper sampled; ROVER won’t reliably reduce WER unless you add genuinely different systems (e.g., IndicWhisper via `TransformersWhisperASR`, plus at least one non-Whisper-family model).
- Verification gate is heuristic-only (`verify.py`): duration/clipping/LID margin/ROVER disagreement/no-speech; it’s missing the critical **audio-grounded transcript fit** check (CTC/forced-alignment scoring) needed to prevent fluent-but-wrong text from being accepted.
- You already collect `avg_logprob` + `compression_ratio` but don’t use them in gating; add thresholds to catch classic Whisper hallucination/repetition failure modes.
- `silenceremove` trimming can clip low-energy initial consonants/breathy speech; keep an option to run ASR on both trimmed/untrimmed (or tune per language), and log pre/post durations.
- `eval.py` only computes WER; for Indic scripts you should also compute **CER** (and report both), and stratify by language + duration buckets + decision (accept/human/reject).

**More sophisticated evaluation (and how to use a “superior model”)**
- Build a *small but high-quality* human gold set per language (even 200–500 clips/lang) + a “challenge set” (short 1–3s, fast speech, code-switch, proper nouns, numbers, mild noise).
- Report: WER + CER by language, plus **tail metrics** (p90/p95 WER, %segments with WER>5%, Segment Error Rate), and acceptance-precision curves (WER on accepted vs accept-rate).
- Add a learned **quality estimator** (trained on the gold set) that predicts expected WER from features you already log: LID confidence, disagreement rate, avg_logprob, compression_ratio, no_speech_prob, QC metrics; use it to drive accept/human/reject thresholds.
- For languages you can’t personally verify: use **IndicWhisper/IndicConformer/MMS(+Whisper v3)** as the robust base, then optionally add Gemini (or another strong API) as an extra hypothesis *only on hard segments*; don’t use Gemini output as “ground truth” (it’s a biased pseudo-reference).
- If you try `gemini-3-pro-preview`, treat it as experimental: benchmark it on the gold set first, and only deploy it where it measurably reduces WER after your verification gate.

If you paste the other agent’s actual Gemini-integration file (or point me to it in the repo), I can give line-by-line review comments too.


# Context from my IDE setup:

## Open tabs:
- .env: .env
- Prompt.md: Prompt.md
- master.md: master.md
- Agents.md: Agents.md
- task.md: task.md

## My request for Codex:
my coding agent says,

Fixes Applied

Prompt made non-hallucinatory + single-output: Prompt.md (line 6) (no “predict/fix grammar”, use <unk> if unsure; English stays Latin; no extra outputs).
Gemini integration added as an optional hypothesis: gemini_asr.py (line 126) + cli.py (line 178) (actually sends audio via types.Part.from_bytes(data=..., mime_type=...), reads GEMINI_KEY, temperature=0, retries/backoff, strict parsing/validation, stores raw output in meta).
Pipeline hardened: pipeline.py (line 44) + verify.py (line 31) (collision-safe segment_id, fail-fast MMS-LID by default, uses Whisper logprob/compression heuristics, optional Gemini, optional CTC-fit scoring that selects final transcript and can gate acceptance).
Eval upgraded: eval.py (line 50) (WER+CER, duration buckets, tail stats; maya-transcribe eval defaults to hypothesis=final).
Try It

maya-transcribe gemini --audio <file.wav> --language eng
wav2vec2-base-960h --gemini-model gemini-2.0-flash
Per-language CTC mapping: maya-transcribe run ... --ctc-model-map ctc_models.json (JSON { "eng": "<hf_id>", "hin": "<hf_id>", ... }).

can you do another round of review.


**Evaluating fail-fast quality measures**

They say the fail-fast MMS-LID approach stops the pipeline if LID isn't available, which seems good for quality. But what if the environment lacks optional dependencies? It's likely okay for quality-first initiatives. I should ensure the CLI has an option like `--allow-missing-lid`. Their use of Whisper logprob/compression heuristics is intriguing because verify.py doesn't utilize them. There's potential for detecting hallucinations with thresholds—like minimum average logprob and maximum compression ratio. Checking for audio-text fit using a separate CTC model is crucial, along with optional CTC-fit scoring for transcripts.


**Exploring alignment and scoring techniques**

I want to explore using a forced alignment algorithm like `ctc_segmentation` or dynamic programming to compute the log probability of transcripts with CTC. There's a known method based on the CTC forward algorithm, but the implementation might be naive. I'll need to inspect that.

Potential issues arise with transcripts in non-Latin scripts because CTC tokenization typically works at the character level. The model's tokenizer might not cover all characters, leading to unknown tokens and possibly invalid scores. Handling out-of-vocabulary characters and ensuring proper normalization for scoring is essential. Also, handling code-switching with mixed languages could complicate things. I'm curious about their evaluation metrics, like WER+CER and the use of jiwer for computing CER. Lastly, I should confirm if the `cli.py` now includes the new command, `maya-transcribe gemini`.


**Checking updates and model recommendations**

It seems our earlier review of `cli.py` didn't show the gemini command, but the user mentioned they added it. I’ll need to check for updates since they might have changed the file.

I want to ensure that the `pyproject.toml` is updated with the google-genai dependency as an optional extra or confirm the user can use pip to install it. I’ll also look for any accidental commits of secrets, like ensuring no `GEMINI_KEY` is in the code.

Regarding the user's question about "superior model for other languages" and gemini-3-pro, I should recommend that for unlimited offline computing, local open models like IndicWhisper and IndicConformer are better, along with verification. APIs can assist but aren’t infallible, so I’ll suggest a thoughtful evaluation approach with methods like cross-model consensus and "oracle" evaluations to compute upper bounds.


**Minimizing WER through metrics**

To minimize WER, I want to compute "oracle WER," which helps estimate potential improvements. I’m considering recommending metrics like "confidence calibration," including expected calibration error (ECE). For further evaluation, we could implement "segment selection" metrics to assess label noise in the accepted set, combining that with sample-based human QA.

I also think "error analysis" is valuable, where we could categorize errors into deletions, insertions, and substitutions at the script level. To get started, I'll check the repository and run `git status` and similar commands.


git status --porcelain=v1 && git diff --stat || true


Output:
Exit code: 0
Wall time: 0 seconds
Output:
fatal: not a git repository (or any of the parent directories): .git


Summary: auto


**Reviewing updates**

I see there's no git repository available, which means I can't do a diff to compare changes. The best course of action now is to review the files directly by reading them. I should check the updated list to identify any recent changes. This is a bit of a hassle, but let's see what I can find by going through the files one by one!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 144
drwxrwxr-x 3 ubuntu ubuntu  4096 Jan 13 05:36 .
drwxrwxr-x 7 ubuntu ubuntu  4096 Jan 13 04:32 ..
-rw-rw-r-- 1 ubuntu ubuntu  2181 Jan 13 04:24 align.py
-rw-rw-r-- 1 ubuntu ubuntu  6663 Jan 13 04:21 asr.py
-rw-rw-r-- 1 ubuntu ubuntu  4147 Jan 13 04:18 audio.py
-rw-rw-r-- 1 ubuntu ubuntu 11675 Jan 13 05:38 cli.py
-rw-rw-r-- 1 ubuntu ubuntu  1258 Jan 13 04:16 config.py
-rw-rw-r-- 1 ubuntu ubuntu  6352 Jan 13 05:40 ctc_fit.py
-rw-rw-r-- 1 ubuntu ubuntu   387 Jan 13 04:17 dataset.py
-rw-rw-r-- 1 ubuntu ubuntu  1508 Jan 13 04:37 db.py
-rw-rw-r-- 1 ubuntu ubuntu  7309 Jan 13 05:38 eval.py
-rw-rw-r-- 1 ubuntu ubuntu  7194 Jan 13 05:32 gemini_asr.py
-rw-rw-r-- 1 ubuntu ubuntu  1138 Jan 13 05:29 ids.py
-rw-rw-r-- 1 ubuntu ubuntu    50 Jan 13 04:16 __init__.py
-rw-rw-r-- 1 ubuntu ubuntu   667 Jan 13 04:19 io.py
-rw-rw-r-- 1 ubuntu ubuntu   551 Jan 13 04:20 language.py
-rw-rw-r-- 1 ubuntu ubuntu  2272 Jan 13 04:35 lid.py
-rw-rw-r-- 1 ubuntu ubuntu 15337 Jan 13 05:38 pipeline.py
drwxrwxr-x 2 ubuntu ubuntu  4096 Jan 13 05:40 __pycache__
-rw-rw-r-- 1 ubuntu ubuntu  3887 Jan 13 04:32 r2.py
-rw-rw-r-- 1 ubuntu ubuntu  6414 Jan 13 04:37 rover.py
-rw-rw-r-- 1 ubuntu ubuntu  1120 Jan 13 04:17 tar.py
-rw-rw-r-- 1 ubuntu ubuntu  1448 Jan 13 04:25 textnorm.py
-rw-rw-r-- 1 ubuntu ubuntu  5377 Jan 13 05:37 verify.py


**Inspecting changed files**

I see there are several new or modified files, like ctc_fit.py, gemini_asr.py, and others, along with size changes in textnorm.py. I need to take a closer look at these updated files. The user mentioned that Prompt.md has changed, so I should definitely check that file as well to understand what’s new, what modifications have been made, and how they might impact my work. Let’s get started on this!


Output:
Exit code: 0
Wall time: 0 seconds
Output:
This is a prompt for any **audio-native** model (Gemini, Voxtral, etc.) used as an extra ASR hypothesis.
It is written to minimize hallucinations and keep outputs easy to validate.

`prompt`

`Instruction: You are a strict, verbatim transcription engine.
Input: Audio file attached
Language hint (primary): <language>

Task:
- Transcribe only what is clearly spoken in the audio.
- Do NOT guess or “fill in” missing words. If a word or phrase is unclear, output <unk> for that span.
- Preserve repetitions, filler words, stutters, and false starts exactly as spoken.
- Do NOT correct grammar, spelling, or pronunciation.
- Do NOT translate or summarize.
- Do NOT add punctuation.

Code-switching:
- Use the native script for the primary language.
- If English words are spoken, keep them in Latin script as spoken (do not transliterate).

OUTPUT FORMAT (exact):
Output 1: Transcript
<single transcript line>

Do not output anything else (no extra headings, notes, or additional outputs).`


sed -n '1,260p' maya_transcribe/gemini_asr.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import os
import time
import unicodedata
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from maya_transcribe.asr import ASRResult
from maya_transcribe.textnorm import normalize_text

def _get_api_key() -> str:
    # Support common env var names; `.env` uses GEMINI_KEY in this repo.
    key = (
        os.getenv("GEMINI_KEY")
        or os.getenv("GEMINI_API_KEY")
        or os.getenv("GOOGLE_API_KEY")
        or os.getenv("GOOGLE_GENAI_API_KEY")
    )
    if not key:
        raise RuntimeError("Missing Gemini API key (set GEMINI_KEY or GOOGLE_API_KEY)")
    return key

def _mime_type_for_path(path: Path) -> str:
    ext = path.suffix.lower().lstrip(".")
    if ext == "wav":
        return "audio/wav"
    if ext == "mp3":
        return "audio/mpeg"
    if ext == "flac":
        return "audio/flac"
    if ext in {"m4a", "mp4"}:
        return "audio/mp4"
    if ext in {"ogg", "opus"}:
        return "audio/ogg"
    return "application/octet-stream"

def load_prompt_template(prompt_path: Path | None = None) -> str:
    """
    Load the prompt text from Prompt.md (preferred) or fall back to a built-in template.
    """
    if prompt_path and prompt_path.exists():
        text = prompt_path.read_text(encoding="utf-8")
        start = text.find("`Instruction:")
        if start != -1:
            start += 1  # drop the leading backtick
            end = text.rfind("`")
            if end != -1 and end > start:
                return text[start:end].strip()
            return text[start:].strip()
        return text.strip()

    return (
        "Instruction: You are a strict, verbatim transcription engine.\n"
        "Input: Audio file attached\n"
        "Language hint (primary): <language>\n\n"
        "Task:\n"
        "- Transcribe only what is clearly spoken in the audio.\n"
        "- Do NOT guess or “fill in” missing words. If unclear, output <unk>.\n"
        "- Preserve repetitions, filler words, stutters, and false starts.\n"
        "- Do NOT correct grammar, spelling, or pronunciation.\n"
        "- Do NOT translate or summarize.\n"
        "- Do NOT add punctuation.\n\n"
        "Code-switching:\n"
        "- Use the native script for the primary language.\n"
        "- If English words are spoken, keep them in Latin script as spoken.\n\n"
        "OUTPUT FORMAT (exact):\n"
        "Output 1: Transcript\n"
        "<single transcript line>\n"
        "Do not output anything else.\n"
    ).strip()

def parse_strict_output(text: str) -> str:
    """
    Extract the transcript line from the strict output format.
    """
    text = (text or "").replace("\r\n", "\n").replace("\r", "\n").strip()
    if not text:
        raise ValueError("empty_response")

    lines = [ln.strip() for ln in text.split("\n") if ln.strip()]
    if not lines:
        raise ValueError("empty_response")

    if lines[0] == "Output 1: Transcript":
        if len(lines) != 2:
            raise ValueError("unexpected_extra_lines")
        transcript = lines[1].strip()
    else:
        # Some model variants return just the transcript; accept only if it's a single non-empty line.
        if len(lines) != 1:
            raise ValueError("unexpected_extra_lines")
        transcript = lines[0].strip()

    if not transcript:
        raise ValueError("empty_transcript")

    # Safety: reject if model added extra outputs.
    lowered = text.lower()
    if "output 2" in lowered or "output 3" in lowered or "output 4" in lowered:
        raise ValueError("extra_outputs")

    # Prompt forbids punctuation; treat punctuation as invalid output.
    for ch in transcript:
        if unicodedata.category(ch).startswith("P"):
            raise ValueError("punctuation_forbidden")

    return transcript

@dataclass(frozen=True)
class GeminiConfig:
    model: str = "gemini-2.0-flash"
    temperature: float = 0.0
    max_output_tokens: int = 512
    response_mime_type: str = "text/plain"
    retries: int = 3
    initial_backoff_sec: float = 1.0

class GeminiASR:
    def __init__(self, config: GeminiConfig | None = None, prompt_template: str | None = None) -> None:
        try:
            from google import genai
            from google.genai import types
        except Exception as e:  # pragma: no cover
            raise RuntimeError("GeminiASR requires `google-genai` (pip install google-genai)") from e

        self._genai = genai
        self._types = types
        self.config = config or GeminiConfig()
        self._client = genai.Client(api_key=_get_api_key())
        self._prompt_template = prompt_template or load_prompt_template(Path("Prompt.md"))

    @property
    def name(self) -> str:
        return f"gemini:{self.config.model}"

    def transcribe(self, audio_path: Path, language: str | None = None) -> ASRResult:
        language = language or "unknown"
        prompt = self._prompt_template.replace("<language>", language)
        audio_bytes = audio_path.read_bytes()
        mime_type = _mime_type_for_path(audio_path)

        cfg = self._types.GenerateContentConfig(
            temperature=self.config.temperature,
            max_output_tokens=self.config.max_output_tokens,
            response_mime_type=self.config.response_mime_type,
        )

        last_err: Exception | None = None
        for attempt in range(self.config.retries + 1):
            try:
                response = self._client.models.generate_content(
                    model=self.config.model,
                    contents=[
                        self._types.Part.from_text(text=prompt),
                        self._types.Part.from_bytes(data=audio_bytes, mime_type=mime_type),
                    ],
                    config=cfg,
                )
                raw_text = getattr(response, "text", None) or ""
                transcript = parse_strict_output(raw_text)

                # Normalization is performed later in the pipeline; keep the raw line.
                return ASRResult(
                    model=self.name,
                    text=transcript,
                    language=language,
                    language_probability=None,
                    segments=[],
                    meta={
                        "mime_type": mime_type,
                        "temperature": self.config.temperature,
                        "max_output_tokens": self.config.max_output_tokens,
                        "response_mime_type": self.config.response_mime_type,
                        "prompt_language": language,
                        "raw_output": raw_text,
                    },
                )
            except Exception as e:
                last_err = e
                if attempt >= self.config.retries:
                    break
                backoff = self.config.initial_backoff_sec * (2**attempt)
                time.sleep(backoff)

        raise RuntimeError(f"gemini_transcribe_failed: {type(last_err).__name__}: {last_err}")

def safe_normalized_transcript(text: str, lang: str | None) -> str:
    """
    Convenience helper: strict normalize for alignment/gating.
    """
    return normalize_text(text, lang=lang)


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import hashlib
from dataclasses import dataclass
from pathlib import Path

@dataclass(frozen=True)
class SegmentIdentity:
    segment_id: str
    base_id: str
    relpath: str
    uid: str

def segment_identity(src: Path, root: Path, seen: dict[str, int] | None = None) -> SegmentIdentity:
    """
    Produce a stable segment identity and prevent collisions when stems repeat.
    - `base_id`: `src.stem`
    - `relpath`: path relative to `root` (or absolute as fallback)
    - `uid`: sha1(relpath) prefix
    - `segment_id`: base_id unless it collides; then base_id-uid
    """
    base_id = src.stem
    try:
        relpath = src.relative_to(root).as_posix()
    except Exception:
        relpath = str(src)

    uid = hashlib.sha1(relpath.encode("utf-8")).hexdigest()[:10]
    segment_id = base_id

    if seen is None:
        segment_id = f"{base_id}-{uid}"
    else:
        n = int(seen.get(base_id, 0))
        seen[base_id] = n + 1
        if n > 0:
            segment_id = f"{base_id}-{uid}"

    return SegmentIdentity(segment_id=segment_id, base_id=base_id, relpath=relpath, uid=uid)


**Evaluating file uniqueness**

I need to consider if there's a file with a stem equal to base_id-uid, which is rare but could cause conflicts. To avoid this, I should always append uid or use a separate mapping for uniqueness. Generally, stems likely follow the SPEAKER_XX pattern, which usually doesn't include a hash. However, I've identified a small bug: the seen dictionary counts occurrences without recognizing the root. If base_id repeats, it could lead to issues with segment identity based on uid. Now, I need to take a look at the new version of pipeline.py.


**Reviewing code components**

I plan to read the relevant portions of the code, focusing on the start of the file, segment_id logic, LID failfast, Gemini integration, CTC-fit scoring, selecting the final transcript, and gating. It seems crucial to open these specific parts to ensure I understand how everything connects. I hope I won't get too lost in the details, but I’m excited to see how these pieces come together. Let’s dive into those sections!


sed -n '1,260p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Total output lines: 260
Output:
from __future__ import annotations

import json
import time
from dataclasses import asdict
from pathlib import Path
from typing import Any

from rich.progress import track

from maya_transcribe.asr import ASRResult, FasterWhisperASR, TransformersWhisperASR
from maya_transcribe.audio import compute_qc, ffprobe, normalize_audio
from maya_transcribe.ids import segment_identity
from maya_transcribe.lid import LIDResult, MMSLID
from maya_transcribe.rover import rover_combine
from maya_transcribe.textnorm import normalize_text
from maya_transcribe.verify import verify_segment

def _iter_audio_files(segments_dir: Path) -> list[Path]:
    audio_files: list[Path] = []
    for ext in ("*.wav", "*.mp3", "*.m4a", "*.flac", "*.ogg", "*.opus"):
        audio_files.extend(sorted(segments_dir.rglob(ext)))
    return sorted(set(audio_files))

def _asr_summary(result: ASRResult) -> dict[str, Any]:
    avg_logprobs = [s.avg_logprob for s in result.segments if s.avg_logprob is not None]
    no_speech_probs = [s.no_speech_prob for s in result.segments if s.no_speech_prob is not None]
    compression_ratios = [s.compression_ratio for s in result.segments if s.compression_ratio is not None]

    def mean(xs: list[float]) -> float | None:
        return (sum(xs) / len(xs)) if xs else None

    return {
        "language": result.language,
        "language_probability": result.language_probability,
        "avg_logprob_mean": mean([float(x) for x in avg_logprobs]),
        "no_speech_prob_mean": mean([float(x) for x in no_speech_probs]),
        "compression_ratio_mean": mean([float(x) for x in compression_ratios]),
    }

def run_pipeline(
    *,
    segments_dir: Path,
    out_dir: Path,
    lid_model_id: str = "facebook/mms-lid-256",
    whisper_model: str = "large-v3",
    extra_transformers_whisper_models: list[str] | None = None,
    gemini_model: str | None = None,
    gemini_prompt_path: Path | None = None,
    gemini_max_output_tokens: int = 512,
    gemini_hard_only: bool = True,
    ctc_model: str | None = None,
    ctc_model_map: dict[str, str] | None = None,
    require_lid_backend: bool = True,
    trim_silence: bool = True,
    silence_threshold_db: float = -40.0,
    min_silence_sec: float = 0.15,
    limit: int | None = None,
    thresholds: dict[str, Any] | None = None,
) -> Path:
    """
    End-to-end run: preprocess -> LID -> ASR ensemble -> normalize -> ROVER -> verify.
    Writes `out_dir/results.jsonl` and returns that path.
    """
    thresholds = thresholds or {}
    if (ctc_model or ctc_model_map) and "require_ctc" not in thresholds:
        thresholds["require_ctc"] = True
    if (ctc_model or ctc_model_map) and "min_ctc_margin" not in thresholds:
        thresholds["min_ctc_margin"] = 0.05
    out_dir.mkdir(parents=True, exist_ok=True)

    audio_files = _iter_audio_files(segments_dir)
    if not audio_files:
        raise RuntimeError(f"No audio files found under: {segments_dir}")
    if limit is not None:
        audio_files = audio_files[: max(0, int(limit))]

    # Instantiate models once.
    lid_backend: MMSLID | None
    try:
        lid_backend = MMSLID(model_id=lid_model_id)
    except Exception as e:
        if require_lid_backend:
            raise RuntimeError(
                "Failed to initialize MMS-LID. Install torch/transformers deps (pip install -e '.[torch]') "
                "or run with require_lid_backend=False."
            ) from e
        lid_backend = None

    whisper_backend = FasterWhisperASR(model_size_or_path=whisper_model)

    extra_transformers_whisper_models = extra_transformers_whisper_models or []
    transformers_backends: list[TransformersWhisperASR] = []
    for model_id in extra_transformers_whisper_models:
        transformers_backends.append(TransformersWhisperASR(model_id=model_id))

    ctc_scorer = None
    rank_by_ctc = None
    if ctc_model or ctc_model_map:
        from maya_transcribe.ctc_fit import CTCFitScorer, rank_candidates_by_ctc

        ctc_scorer = CTCFitScorer(model_map=ctc_model_map, default_model=ctc_model)
        rank_by_ctc = rank_candidates_by_ctc

    gemini_backend = None
    if gemini_model:
        try:
            from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template

            prompt = load_prompt_template(gemini_prompt_path or Path("Prompt.md"))
            gemini_backend = GeminiASR(
                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
                prompt_template=prompt,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e

    results_path = out_dir / "results.jsonl"
    summary_path = out_dir / "summary.json"

    counts = {"accept": 0, "human": 0, "reject": 0, "error": 0}
    timings_total: list[float] = []

    seen: dict[str, int] = {}
    with results_path.open("w", encoding="utf-8") as f:
        for src in track(audio_files, desc…1147 chars truncated…        if lid and lid.predictions:
                    top1 = lid.predictions[0]
                    top2p = lid.predictions[1].prob if len(lid.predictions) > 1 else 0.0
                    if float(top1.prob) >= float(thresholds.get("lid_min_prob", 0.90)) and (
                        float(top1.prob) - float(top2p)
                    ) >= float(thresholds.get("lid_min_margin", 0.20)):
                        route_lang = top1.code

                # 3) ASR ensemble
                t_asr0 = time.perf_counter()
                asr_results: dict[str, ASRResult] = {}

                # Whisper: deterministic + a small sampling diversity pass.
                det = whisper_backend.transcribe(wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0)
                primary_asr_key = f"{det.model}|det"
                asr_results[primary_asr_key] = det
                samp = whisper_backend.transcribe(
                    wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2
                )
                asr_results[f"{samp.model}|samp_t0.2"] = samp

                for tf_backend in transformers_backends:
                    res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
                    asr_results[res.model] = res

                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
                if gemini_backend is not None:
                    run_gemini = True
                    if gemini_hard_only:
                        det_sum = _asr_summary(det)
                        avg_lp = det_sum.get("avg_logprob_mean")
                        comp = det_sum.get("compression_ratio_mean")
                        no_speech = det_sum.get("no_speech_prob_mean")
                        run_gemini = bool(
                            route_lang is None
                            or (no_speech is not None and float(no_speech) > 0.5)
                            or (avg_lp is not None and float(avg_lp) < float(thresholds.get("min_avg_logprob_mean", -1.0)))
                            or (
                                comp is not None
                                and float(comp) > float(thresholds.get("max_compression_ratio_mean", 2.4))
                            )
                        )

                    if run_gemini:
                        try:
                            g = gemini_backend.transcribe(wav_path, language=route_lang)
                            asr_results[g.model] = g
                        except Exception:
                            pass

                t_asr1 = time.perf_counter()

                # 4) Normalize hypotheses
                t_norm0 = time.perf_counter()
                norm_hyps: dict[str, str] = {}
                for k, res in asr_results.items():
                    lang_hint = res.language or route_lang
                    norm_hyps[k] = normalize_text(res.text, lang=lang_hint)
                t_norm1 = time.perf_counter()

                # 5) ROVER
                t_rover0 = time.perf_counter()
                rover = rover_combine(norm_hyps)
                t_rover1 = time.perf_counter()

                # 5.5) Audio-grounded fit scoring (optional): choose best candidate by CTC loss.
                t_ctc0 = time.perf_counter()
                ctc_info = None
                final_text = rover.text
                final_source = "rover"
                if ctc_scorer is not None and rank_by_ctc is not None:
                    candidates = {"rover": rover.text, **norm_hyps}
                    ctc_info = rank_by_ctc(ctc_scorer, wav_path, candidates=candidates, lang=route_lang)
                    best_key = ctc_info.get("best_key") if isinstance(ctc_info, dict) else None
                    if best_key and best_key in candidates:
                        final_text = candidates[best_key]
                        final_source = "rover" if best_key == "rover" else f"asr:{best_key}"
                t_ctc1 = time.perf_counter()

                # 6) Verify
                t_ver0 = time.perf_counter()
                asr_summaries = {k: _asr_summary(v) for k, v in asr_results.items()}
                verification = verify_segment(
                    qc=asdict(qc),
                    lid=lid,
                    final_text=final_text,
                    rover={
                        "text": rover.text,
                        "disagreement_rate": rover.disagreement_rate,
                        "skeleton_key": rover.skeleton_key,
                    },
                    asr_summaries=asr_summaries,
                    primary_asr_key=primary_asr_key,
                    ctc=ctc_info,
                    thresholds=thresholds,
                )
                t_ver1 = time.perf_counter()

                record = {
                    "segment_id": segment_id,
                    "base_id": ident.base_id,
                    "relpath": ident.relpath,


sed -n '260,560p' maya_transcribe/pipeline.py