Output:
Chunk ID: 1412eb
Wall time: 0.0823 seconds
Process exited with code 1
Original token count: 2247
Output:
usage: vastai [-h] [--url URL] [--retry RETRY] [--raw] [--explain] [--curl] [--api-key API_KEY] [--version] command ...

positional arguments:
 command                      command to run. one of:
  help                        print this help message
  attach ssh                  Attach an ssh key to an instance. This will allow you to connect to the instance with the ssh key
  cancel copy                 Cancel a remote copy in progress, specified by DST id
  cancel sync                 Cancel a remote copy in progress, specified by DST id
  change bid                  Change the bid price for a spot/interruptible instance
  clone volume                Clone an existing volume
  copy                        Copy directories between instances and/or local
  cloud copy                  Copy files/folders to and from cloud providers
  take snapshot               Schedule a snapshot of a running container and push it to your repo in a container registry
  create api-key              Create a new api-key with restricted permissions. Can be sent to other users and teammates
  create cluster              Create Vast cluster
  create env-var              Create a new user environment variable
  create ssh-key              Create a new ssh-key
  create workergroup          Create a new autoscale group
  create endpoint             Create a new endpoint group
  create instance             Create a new instance
  create subaccount           Create a subaccount
  create team                 Create a new team
  create team-role            Add a new role to your team
  create template             Create a new template
  create volume               Create a new volume
  create network-volume       Create a new network volume
  create overlay              Creates overlay network on top of a physical cluster
  delete api-key              Remove an api-key
  delete ssh-key              Remove an ssh-key
  delete scheduled-job        Delete a scheduled job
  delete cluster              Delete Cluster
  delete workergroup          Delete a workergroup group
  delete endpoint             Delete an endpoint group
  delete env-var              Delete a user environment variable
  delete overlay              Deletes overlay and removes all of its associated instances
  delete template             Delete a Template
  delete volume               Delete a volume
  destroy instance            Destroy an instance (irreversible, deletes data)
  destroy instances           Destroy a list of instances (irreversible, deletes data)
  destroy team                Destroy your team
  detach ssh                  Detach an ssh key from an instance
  execute                     Execute a (constrained) remote command on a machine
  get endpt-logs              Fetch logs for a specific serverless endpoint group
  get wrkgrp-logs             Fetch logs for a specific serverless worker group group
  invite member               Invite a team member
  join cluster                Join Machine to Cluster
  join overlay                Adds instance to an overlay network
  label instance              Assign a string label to an instance
  launch instance             Launch the top instance from the search offers based on the given parameters
  logs                        Get the logs for an instance
  prepay instance             Deposit credits into reserved instance
  reboot instance             Reboot (stop/start) an instance
  recycle instance            Recycle (destroy/create) an instance
  remove member               Remove a team member
  remove team-role            Remove a role from your team
  reports                     Get the user reports for a given machine
  reset api-key               Reset your api-key (get new key from website)
  start instance              Start a stopped instance
  start instances             Start a list of instances
  stop instance               Stop a running instance
  stop instances              Stop a list of instances
  search benchmarks           Search for benchmark results using custom query
  search invoices             Search for benchmark results using custom query
  search offers               Search for instance types using custom query
  search templates            Search for template results using custom query
  search volumes              Search for volume offers using custom query
  search network-volumes      Search for network volume offers using custom query
  set api-key                 Set api-key (get your api-key from the console/CLI)
  set user                    Update user data from json file
  ssh-url                     ssh url helper
  scp-url                     scp url helper
  show api-key                Show an api-key
  show api-keys               List your api-keys associated with your account
  show audit-logs             Display account's history of important actions
  show scheduled-jobs         Display the list of scheduled jobs
  show ssh-keys               List your ssh keys associated with your account
  show workergroups           Display user's current workergroups
  show endpoints              Display user's current endpoint groups
  show connections            Display user's cloud connections
  show deposit                Display reserve deposit info for an instance
  show earnings               Get machine earning history reports
  show env-vars               Show user environment variables
  show invoices               Get billing history reports
  show instance               Display user's current instances
  show instances              Display user's current instances
  show ipaddrs                Display user's history of ip addresses
  show clusters               Show clusters associated with your account.
  show overlays               Show overlays associated with your account.
  show subaccounts            Get current subaccounts
  show members                Show your team members
  show team-role              Show your team role
  show team-roles             Show roles for a team
  show user                   Get current user data
  show volumes                Show stats on owned volumes.
  remove-machine-from-cluster
                              Removes machine from cluster
  transfer credit             Transfer credits to another account
  update workergroup          Update an existing autoscale group
  update endpoint             Update an existing endpoint group
  update env-var              Update an existing user environment variable
  update instance             Update recreate an instance from a new/updated template
  update team-role            Update an existing team role
  update template             Update an existing template
  update ssh-key              Update an existing ssh key
  cancel maint                [Host] Cancel maint window
  cleanup machine             [Host] Remove all expired storage instances from the machine, freeing up space
  delete machine              [Host] Delete machine if the machine is not being used by clients. host jobs on their own machines are disregarded and machine is force deleted.
  list machine                [Host] list a machine for rent
  list machines               [Host] list machines for rent
  add network-disk            [Host] Add Network Disk to Physical Cluster.
  show network-disks          Show network disks associated with your account.
  list network-volume         [Host] list disk space for rent as a network volume
  unlist network-volume       Unlists network volume offer
  list volume                 [Host] list disk space for rent as a volume on a machine
  list volumes                [Host] list disk space for rent as a volume on machines
  unlist volume               [Host] unlist volume offer
  remove defjob               [Host] Delete default jobs
  set defjob                  [Host] Create default jobs for a machine
  set min-bid                 [Host] Set the minimum bid/rental price for a machine
  schedule maint              [Host] Schedule upcoming maint window
  show machine                [Host] Show hosted machines
  show machines               [Host] Show hosted machines
  show maints                 [Host] Show maintenance information for host machines
  unlist machine              [Host] Unlist a listed machine
  self-test machine           [Host] Perform a self-test on the specified machine

options:
 -h, --help                   show this help message and exit
 --url URL                    server REST api url
 --retry RETRY                retry limit
 --raw                        output machine-readable json
 --explain                    output verbose explanation of mapping of CLI calls to HTTPS API endpoints
 --curl                       show a curl equivalency to the call
 --api-key API_KEY            api key. defaults to using the one stored in /home/ubuntu/.config/vastai/vast_api_key
 --version                    show version

Use 'vast COMMAND --help' for more info about a command


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Planning comprehensive stress test**


I’m going to add a new full-fidelity stress harness (`production/test_stress_full.py`) so we can run deeper HTTP/SSE tests without destabilizing the existing script. It will add route timing, queue-full detection, time-series worker/server snapshots, and optional FLEURS-based WER/CER under load.


**Outlining comprehensive async benchmarking script**


cat > production/test_stress_full.py <<'PY'
#!/usr/bin/env python3
"""
Full-fidelity stress + quality test for Hindi ASR over Vast serverless (HTTP/SSE).

Features:
- Customer-like concurrent clients with multiple requests per client
- Route timing + queue-full detection
- First-token/total latency + server processing time
- Optional FLEURS Hindi audio (real speech) with WER/CER checks
- Time-series snapshots from /metrics and /get_endpoint_workers/
- CSV + JSON outputs suitable for scaling-law fitting
"""

import argparse
import asyncio
import base64
import csv
import io
import json
import math
import os
import random
import re
import struct
import sys
import time
import wave
from collections import Counter
from dataclasses import dataclass, asdict
from datetime import datetime
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple

import aiohttp
import numpy as np
import requests

try:
    import jiwer
except Exception:
    jiwer = None

VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"

def _env_or_dotenv(name: str) -> Optional[str]:
    if os.environ.get(name):
        return os.environ.get(name)
    env_path = Path(__file__).resolve().parent.parent / ".env"
    if env_path.exists():
        for line in env_path.read_text().splitlines():
            if line.startswith(name + "="):
                return line.split("=", 1)[1].strip()
    return None

def get_endpoint_info(endpoint_name: str, account_key: Optional[str] = None) -> Dict[str, Any]:
    key = account_key or _env_or_dotenv("VAST_KEY")
    if not key:
        raise RuntimeError("VAST_KEY not set; pass --endpoint-key and --endpoint-id")
    r = requests.get(
        f"{VAST_API}/endptjobs/",
        headers={"Accept": "application/json", "Authorization": f"Bearer {key}"},
        timeout=30,
    )
    data = r.json()
    results = data.get("results", []) if isinstance(data, dict) else data
    for ep in results or []:
        if ep.get("endpoint_name") == endpoint_name:
            return ep
    raise RuntimeError(f"endpoint '{endpoint_name}' not found")

async def get_route(session: aiohttp.ClientSession, endpoint_name: str, endpoint_key: str, cost: float) -> Tuple[Optional[Dict[str, Any]], float, Optional[str], bool]:
    t0 = time.perf_counter()
    try:
        async with session.post(
            f"{VAST_RUN}/route/",
            json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": cost},
            timeout=aiohttp.ClientTimeout(total=15),
        ) as r:
            data = await r.json(content_type=None)
    except Exception as e:
        return None, (time.perf_counter() - t0) * 1000.0, str(e), False

    dt = (time.perf_counter() - t0) * 1000.0
    if isinstance(data, dict) and "url" in data:
        return data, dt, None, False

    status = ""
    if isinstance(data, dict):
        status = str(data.get("status", ""))
    queue_full = "all worker queues full" in status.lower()
    return None, dt, f"route failed: {data}", queue_full

def wrap_auth(route: Dict[str, Any], payload: Dict[str, Any]) -> Dict[str, Any]:
    auth = {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"] if k in route}
    return {"auth_data": auth, "payload": payload}

def pct(vals: List[float], p: int) -> float:
    if not vals:
        return 0.0
    s = sorted(vals)
    idx = min(int(len(s) * p / 100), len(s) - 1)
    return float(s[idx])

def safe_mean(vals: List[float]) -> float:
    return float(sum(vals) / len(vals)) if vals else 0.0

def normalize_text(text: str) -> str:
    if not text:
        return ""
    text = text.lower().strip()
    text = re.sub(r"<[^>]+>", " ", text)
    text = re.sub(r"[^\u0900-\u097fa-z0-9\s]", " ", text)
    text = re.sub(r"\s+", " ", text).strip()
    return text

def compute_wer_cer(reference: str, hypothesis: str) -> Tuple[Optional[float], Optional[float]]:
    if not jiwer:
        return None, None
    ref = normalize_text(reference)
    hyp = normalize_text(hypothesis)
    if not ref:
        return None, None
    try:
        return float(jiwer.wer(ref, hyp)), float(jiwer.cer(ref, hyp))
    except Exception:
        return None, None

@dataclass
class AudioSample:
    sample_id: str
    duration_s: float
    wav_bytes: bytes
    reference: Optional[str] = None

def _pcm_to_wav_bytes(pcm_int16: np.ndarray, sample_rate: int = 16000) -> bytes:
    bio = io.BytesIO()
    with wave.open(bio, "wb") as wf:
        wf.setnchannels(1)
        wf.setsampwidth(2)
        wf.setframerate(sample_rate)
        wf.writeframes(pcm_int16.astype(np.int16).tobytes())
    return bio.getvalue()

def make_sine_samples(durations: List[float]) -> List[AudioSample]:
    samples: List[AudioSample] = []
    for i, d in enumerate(durations):
        n = int(16000 * d)
        pcm = np.array(
            [
                min(32767, max(-32768, int(16000 * math.sin(2 * math.pi * 440 * k / 16000))))
                for k in range(n)
            ],
            dtype=np.int16,
        )
        wav = _pcm_to_wav_bytes(pcm)
        samples.append(AudioSample(sample_id=f"sine_{i}_{d:.2f}s", duration_s=float(d), wav_bytes=wav, reference=None))
    return samples

def load_fleurs_samples(max_samples: int, min_duration: float, max_duration: float, seed: int) -> List[AudioSample]:
    from datasets import load_dataset

    ds = load_dataset("google/fleurs", "hi_in", split="test")
    rows = list(ds)
    rnd = random.Random(seed)
    rnd.shuffle(rows)

    out: List[AudioSample] = []
    for row in rows:
        audio = row["audio"]
        arr = np.asarray(audio["array"])  # float32 [-1, 1]
        sr = int(audio["sampling_rate"])
        if sr != 16000:
            # FLEURS hi_in is normally 16kHz, but keep explicit guard.
            continue
        duration = len(arr) / float(sr)
        if duration < min_duration or duration > max_duration:
            continue
        pcm = np.clip(arr, -1.0, 1.0)
        pcm = (pcm * 32767.0).astype(np.int16)
        wav = _pcm_to_wav_bytes(pcm, sample_rate=sr)
        out.append(
            AudioSample(
                sample_id=str(row["id"]),
                duration_s=round(duration, 3),
                wav_bytes=wav,
                reference=row.get("transcription"),
            )
        )
        if len(out) >= max_samples:
            break

    if not out:
        raise RuntimeError("No FLEURS samples matched filters")
    return out

@dataclass
class RequestRecord:
    client_idx: int
    req_idx: int
    mode: str
    sample_id: str
    duration_s: float
    route_ms: float
    connect_ms: float
    first_token_ms: float
    total_ms: float
    server_processing_ms: float
    status_code: int
    ok: bool
    queue_full: bool
    error: str
    reference: str
    hypothesis: str
    wer: float
    cer: float

@dataclass
class LevelResult:
    mode: str
    concurrency: int
    requests_per_client: int
    wall_time_s: float
    ok: int
    errors: int
    queue_full_errors: int
    throughput_x: float
    route_ms_p50: float
    route_ms_p95: float
    route_ms_p99: float
    first_token_ms_p50: float
    first_token_ms_p95: float
    first_token_ms_p99: float
    total_ms_p50: float
    total_ms_p95: float
    total_ms_p99: float
    server_proc_ms_p50: float
    server_proc_ms_p95: float
    server_proc_ms_p99: float
    vram_used_gb_max: float
    worker_cur_load_max: float
    worker_cur_load_p95: float
    worker_cur_perf_avg: float
    wer_pct: float
    cer_pct: float
    status_counts: Dict[str, int]

async def fetch_server_metrics_via_endpoint(
    session: aiohttp.ClientSession,
    endpoint_name: str,
    endpoint_key: str,
    route_cost: float,
) -> Dict[str, Any]:
    route, _, err, _ = await get_route(session, endpoint_name, endpoint_key, route_cost)
    if not route:
        return {"error": err}
    worker_url = route["url"].rstrip("/")
    req = wrap_auth(route, {})
    try:
        async with session.post(
            f"{worker_url}/metrics",
            json=req,
            timeout=aiohttp.ClientTimeout(total=12),
        ) as r:
            if r.status == 200:
                return await r.json(content_type=None)
            return {"error": f"metrics status={r.status}"}
    except Exception as e:
        return {"error": str(e)}

async def fetch_endpoint_workers(
    session: aiohttp.ClientSession,
    endpoint_id: int,
    endpoint_key: str,
) -> List[Dict[str, Any]]:
    try:
        async with session.post(
            f"{VAST_RUN}/get_endpoint_workers/",
            json={"id": endpoint_id, "api_key": endpoint_key},
            timeout=aiohttp.ClientTimeout(total=12),
        ) as r:
            data = await r.json(content_type=None)
            if isinstance(data, dict) and "workers" in data:
                return data["workers"]
            return data if isinstance(data, list) else []
    except Exception:
        return []

async def metrics_sampler(
    stop_evt: asyncio.Event,
    interval_s: float,
    session: aiohttp.ClientSession,
    endpoint_name: Optional[str],
    endpoint_key: Optional[str],
    endpoint_id: Optional[int],
    route_cost: float,
    server_samples: List[Dict[str, Any]],
    worker_samples: List[Dict[str, Any]],
):
    while not stop_evt.is_set():
        ts = time.time()
        if endpoint_name and endpoint_key:
            sm = await fetch_server_metrics_via_endpoint(session, endpoint_name, endpoint_key, route_cost)
            server_samples.append({"t": ts, "metrics": sm})
            if endpoint_id is not None:
                workers = await fetch_endpoint_workers(session, endpoint_id, endpoint_key)
                worker_samples.append({"t": ts, "workers": workers})
        await asyncio.sleep(interval_s)

async def run_http_request(
    session: aiohttp.ClientSession,
    sample: AudioSample,
    endpoint_name: Optional[str],
    endpoint_key: Optional[str],
    route_cost: float,
    direct_url: Optional[str],
) -> RequestRecord:
    t0 = time.perf_counter()
    route_ms = 0.0
    queue_full = False
    connect_ms = 0.0
    status_code = 0
    first_token_ms = 0.0
    server_proc_ms = 0.0
    hypothesis = ""
    err = ""

    try:
        if endpoint_name and endpoint_key:
            route, route_ms, rerr, qf = await get_route(session, endpoint_name, endpoint_key, route_cost)
            queue_full = qf
            if not route:
                raise RuntimeError(rerr or "route failed")
            worker_url = route["url"].rstrip("/")
            payload = {
                "audio_base64": base64.b64encode(sample.wav_bytes).decode("ascii"),
                "language": "hi",
                "audio_duration": sample.duration_s,
            }
            req = wrap_auth(route, payload)
            t_conn = time.perf_counter()
            async with session.post(
                f"{worker_url}/v1/audio/transcriptions",
                json=req,
                timeout=aiohttp.ClientTimeout(total=180),
            ) as r:
                connect_ms = (time.perf_counter() - t_conn) * 1000.0
                status_code = int(r.status)
                if r.status in (404, 422):
                    route2, route_ms2, rerr2, qf2 = await get_route(session, endpoint_name, endpoint_key, route_cost)
                    route_ms += route_ms2
                    queue_full = queue_full or qf2
                    if not route2:
                        raise RuntimeError(rerr2 or "route fallback failed")
                    req2 = wrap_auth(route2, payload)
                    worker_url2 = route2["url"].rstrip("/")
                    async with session.post(
                        f"{worker_url2}/v1/audio/transcriptions/json",
                        json=req2,
                        timeout=aiohttp.ClientTimeout(total=180),
                    ) as r2:
                        status_code = int(r2.status)
                        data = await r2.json(content_type=None)
                else:
                    data = await r.json(content_type=None)
        else:
            if not direct_url:
                raise RuntimeError("direct_url required when endpoint is not set")
            form = aiohttp.FormData()
            form.add_field("file", sample.wav_bytes, filename="sample.wav", content_type="audio/wav")
            form.add_field("language", "hi")
            t_conn = time.perf_counter()
            async with session.post(
                f"{direct_url.rstrip('/')}/v1/audio/transcriptions",
                data=form,
                timeout=aiohttp.ClientTimeout(total=180),
            ) as r:
                connect_ms = (time.perf_counter() - t_conn) * 1000.0
                status_code = int(r.status)
                data = await r.json(content_type=None)

        if status_code != 200:
            err = f"HTTP {status_code}: {str(data)[:400]}"
        else:
            hypothesis = str(data.get("text", "") or "")
            if "processing_time" in data:
                try:
                    server_proc_ms = float(data["processing_time"]) * 1000.0
                except Exception:
                    server_proc_ms = 0.0

    except Exception as e:
        err = str(e)

    t1 = time.perf_counter()
    total_ms = (t1 - t0) * 1000.0
    first_token_ms = total_ms  # HTTP batch has no token stream

    ref = sample.reference or ""
    wer_val, cer_val = compute_wer_cer(ref, hypothesis)

    return RequestRecord(
        client_idx=-1,
        req_idx=-1,
        mode="http",
        sample_id=sample.sample_id,
        duration_s=sample.duration_s,
        route_ms=route_ms,
        connect_ms=connect_ms,
        first_token_ms=first_token_ms,
        total_ms=total_ms,
        server_processing_ms=server_proc_ms,
        status_code=status_code,
        ok=(err == "" and status_code == 200),
        queue_full=queue_full,
        error=err,
        reference=ref,
        hypothesis=hypothesis,
        wer=float("nan") if wer_val is None else wer_val,
        cer=float("nan") if cer_val is None else cer_val,
    )

async def run_sse_request(
    session: aiohttp.ClientSession,
    sample: AudioSample,
    endpoint_name: Optional[str],
    endpoint_key: Optional[str],
    route_cost: float,
    direct_url: Optional[str],
) -> RequestRecord:
    t0 = time.perf_counter()
    route_ms = 0.0
    queue_full = False
    connect_ms = 0.0
    status_code = 0
    first_token_ms = 0.0
    hypothesis = ""
    err = ""

    try:
        if endpoint_name and endpoint_key:
            route, route_ms, rerr, qf = await get_route(session, endpoint_name, endpoint_key, route_cost)
            queue_full = qf
            if not route:
                raise RuntimeError(rerr or "route failed")
            worker_url = route["url"].rstrip("/")
            payload = {
                "audio_base64": base64.b64encode(sample.wav_bytes).decode("ascii"),
                "language": "hi",
                "chunk_duration_ms": 160,
                "audio_duration": sample.duration_s,
            }
            req = wrap_auth(route, payload)
            t_conn = time.perf_counter()
            async with session.post(
                f"{worker_url}/v1/audio/transcriptions/stream",
                json=req,
                timeout=aiohttp.ClientTimeout(total=180),
            ) as r:
                connect_ms = (time.perf_counter() - t_conn) * 1000.0
                status_code = int(r.status)
                if status_code != 200:
                    body = await r.text()
                    raise RuntimeError(f"SSE {status_code}: {body[:200]}")

                got_first = False
                last_text = ""
                async for line in r.content:
                    l = line.decode(errors="ignore").strip()
                    if not l.startswith("data: "):
                        continue
                    try:
                        d = json.loads(l[6:])
                    except Exception:
                        continue
                    txt = str(d.get("text", "") or "")
                    if txt:
                        last_text = txt
                        if not got_first:
                            first_token_ms = (time.perf_counter() - t0) * 1000.0
                            got_first = True
                    if d.get("done"):
                        break
                hypothesis = last_text
        else:
            if not direct_url:
                raise RuntimeError("direct_url required when endpoint is not set")
            form = aiohttp.FormData()
            form.add_field("file", sample.wav_bytes, filename="sample.wav", content_type="audio/wav")
            t_conn = time.perf_counter()
            async with session.post(
                f"{direct_url.rstrip('/')}/v1/audio/transcriptions/stream",
                data=form,
                timeout=aiohttp.ClientTimeout(total=180),
            ) as r:
                connect_ms = (time.perf_counter() - t_conn) * 1000.0
                status_code = int(r.status)
                if status_code != 200:
                    body = await r.text()
                    raise RuntimeError(f"SSE {status_code}: {body[:200]}")
                got_first = False
                last_text = ""
                async for line in r.content:
                    l = line.decode(errors="ignore").strip()
                    if not l.startswith("data: "):
                        continue
                    try:
                        d = json.loads(l[6:])
                    except Exception:
                        continue
                    txt = str(d.get("text", "") or "")
                    if txt:
                        last_text = txt
                        if not got_first:
                            first_token_ms = (time.perf_counter() - t0) * 1000.0
                            got_first = True
                    if d.get("done"):
                        break
                hypothesis = last_text
    except Exception as e:
        err = str(e)

    t1 = time.perf_counter()
    total_ms = (t1 - t0) * 1000.0
    if first_token_ms <= 0:
        first_token_ms = total_ms

    ref = sample.reference or ""
    wer_val, cer_val = compute_wer_cer(ref, hypothesis)

    return RequestRecord(
        client_idx=-1,
        req_idx=-1,
        mode="sse",
        sample_id=sample.sample_id,
        duration_s=sample.duration_s,
        route_ms=route_ms,
        connect_ms=connect_ms,
        first_token_ms=first_token_ms,
        total_ms=total_ms,
        server_processing_ms=0.0,
        status_code=status_code,
        ok=(err == "" and status_code == 200),
        queue_full=queue_full,
        error=err,
        reference=ref,
        hypothesis=hypothesis,
        wer=float("nan") if wer_val is None else wer_val,
        cer=float("nan") if cer_val is None else cer_val,
    )

def _series_vram_max(server_samples: List[Dict[str, Any]]) -> float:
    vals: List[float] = []
    for s in server_samples:
        m = s.get("metrics", {})
        if isinstance(m, dict) and "vram_used_gb" in m:
            try:
                vals.append(float(m.get("vram_used_gb", 0.0)))
            except Exception:
                pass
    return max(vals) if vals else 0.0

def _worker_stats(worker_samples: List[Dict[str, Any]]) -> Tuple[float, float, float]:
    loads: List[float] = []
    perfs: List[float] = []
    for s in worker_samples:
        workers = s.get("workers") or []
        for w in workers:
            try:
                loads.append(float(w.get("cur_load", 0.0)))
                perfs.append(float(w.get("cur_perf", 0.0)))
            except Exception:
                pass
    return (max(loads) if loads else 0.0, pct(loads, 95), safe_mean(perfs))

def summarize_level(
    mode: str,
    concurrency: int,
    requests_per_client: int,
    wall_time_s: float,
    records: List[RequestRecord],
    server_samples: List[Dict[str, Any]],
    worker_samples: List[Dict[str, Any]],
) -> LevelResult:
    ok_recs = [r for r in records if r.ok]
    route_ms = [r.route_ms for r in records if r.route_ms > 0]
    first_ms = [r.first_token_ms for r in ok_recs]
    total_ms = [r.total_ms for r in ok_recs]
    proc_ms = [r.server_processing_ms for r in ok_recs if r.server_processing_ms > 0]

    total_audio_ok = sum(r.duration_s for r in ok_recs)
    throughput_x = (total_audio_ok / wall_time_s) if wall_time_s > 0 else 0.0

    wer_vals = [r.wer for r in ok_recs if r.reference and not math.isnan(r.wer)]
    cer_vals = [r.cer for r in ok_recs if r.reference and not math.isnan(r.cer)]

    status_counts: Counter[str] = Counter()
    for r in records:
        k = str(r.status_code if r.status_code else "exception")
        status_counts[k] += 1

    qf = sum(1 for r in records if r.queue_full)

    vram_max = _series_vram_max(server_samples)
    worker_load_max, worker_load_p95, worker_perf_avg = _worker_stats(worker_samples)

    return LevelResult(
        mode=mode,
        concurrency=concurrency,
        requests_per_client=requests_per_client,
        wall_time_s=round(wall_time_s, 3),
        ok=len(ok_recs),
        errors=len(records) - len(ok_recs),
        queue_full_errors=qf,
        throughput_x=round(throughput_x, 3),
        route_ms_p50=round(pct(route_ms, 50), 2),
        route_ms_p95=round(pct(route_ms, 95), 2),
        route_ms_p99=round(pct(route_ms, 99), 2),
        first_token_ms_p50=round(pct(first_ms, 50), 2),
        first_token_ms_p95=round(pct(first_ms, 95), 2),
        first_token_ms_p99=round(pct(first_ms, 99), 2),
        total_ms_p50=round(pct(total_ms, 50), 2),
        total_ms_p95=round(pct(total_ms, 95), 2),
        total_ms_p99=round(pct(total_ms, 99), 2),
        server_proc_ms_p50=round(pct(proc_ms, 50), 2),
        server_proc_ms_p95=round(pct(proc_ms, 95), 2),
        server_proc_ms_p99=round(pct(proc_ms, 99), 2),
        vram_used_gb_max=round(vram_max, 2),
        worker_cur_load_max=round(worker_load_max, 3),
        worker_cur_load_p95=round(worker_load_p95, 3),
        worker_cur_perf_avg=round(worker_perf_avg, 4),
        wer_pct=round(100.0 * safe_mean(wer_vals), 3) if wer_vals else 0.0,
        cer_pct=round(100.0 * safe_mean(cer_vals), 3) if cer_vals else 0.0,
        status_counts=dict(status_counts),
    )

async def run_level(
    mode: str,
    concurrency: int,
    requests_per_client: int,
    samples: List[AudioSample],
    endpoint_name: Optional[str],
    endpoint_key: Optional[str],
    endpoint_id: Optional[int],
    route_cost: float,
    metrics_interval: float,
    direct_url: Optional[str],
    shuffle_samples: bool,
    seed: int,
    inter_request_delay: float,
):
    print("\n" + "=" * 90)
    print(f"Level mode={mode} c={concurrency} reqs/client={requests_per_client}")
    print("=" * 90)

    pool = list(samples)
    if shuffle_samples:
        rnd = random.Random(seed + concurrency)
        rnd.shuffle(pool)

    sample_idx = 0
    sample_lock = asyncio.Lock()

    async def pick_sample() -> AudioSample:
        nonlocal sample_idx
        async with sample_lock:
            s = pool[sample_idx % len(pool)]
            sample_idx += 1
            return s

    timeout = aiohttp.ClientTimeout(total=220)
    connector = aiohttp.TCPConnector(limit=max(200, concurrency * 8), ttl_dns_cache=300)

    records: List[RequestRecord] = []
    server_samples: List[Dict[str, Any]] = []
    worker_samples: List[Dict[str, Any]] = []

    stop_evt = asyncio.Event()

    async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:
        sampler_task = asyncio.create_task(
            metrics_sampler(
                stop_evt,
                metrics_interval,
                session,
                endpoint_name,
                endpoint_key,
                endpoint_id,
                route_cost,
                server_samples,
                worker_samples,
            )
        )

        t0 = time.perf_counter()

        async def client_worker(client_idx: int):
            for req_idx in range(requests_per_client):
                sample = await pick_sample()
                if mode == "http":
                    rec = await run_http_request(
                        session,
                        sample,
                        endpoint_name,
                        endpoint_key,
                        route_cost,
                        direct_url,
                    )
                else:
                    rec = await run_sse_request(
                        session,
                        sample,
                        endpoint_name,
                        endpoint_key,
                        route_cost,
                        direct_url,
                    )
                rec.client_idx = client_idx
                rec.req_idx = req_idx
                records.append(rec)

                if inter_request_delay > 0 and req_idx + 1 < requests_per_client:
                    await asyncio.sleep(inter_request_delay)

        await asyncio.gather(*(client_worker(i) for i in range(concurrency)))
        t1 = time.perf_counter()

        stop_evt.set()
        try:
            await asyncio.wait_for(sampler_task, timeout=10)
        except Exception:
            sampler_task.cancel()

    wall = t1 - t0
    summary = summarize_level(
        mode,
        concurrency,
        requests_per_client,
        wall,
        records,
        server_samples,
        worker_samples,
    )

    print(
        f"OK={summary.ok}/{len(records)} err={summary.errors} qfull={summary.queue_full_errors} "
        f"tput={summary.throughput_x:.2f}x total_p95={summary.total_ms_p95:.0f}ms "
        f"ft_p95={summary.first_token_ms_p95:.0f}ms wer={summary.wer_pct:.2f}%"
    )

    return {
        "summary": asdict(summary),
        "records": [asdict(r) for r in records],
        "server_samples": server_samples,
        "worker_samples": worker_samples,
    }

def write_outputs(results: List[Dict[str, Any]], out_prefix: str):
    out_dir = Path(__file__).resolve().parent.parent / "benchmark_results"
    out_dir.mkdir(parents=True, exist_ok=True)

    csv_path = out_dir / f"{out_prefix}.csv"
    json_path = out_dir / f"{out_prefix}.json"
    details_csv_path = out_dir / f"{out_prefix}_details.csv"

    summary_fields = [
        "mode",
        "concurrency",
        "requests_per_client",
        "wall_time_s",
        "ok",
        "errors",
        "queue_full_errors",
        "throughput_x",
        "route_ms_p50",
        "route_ms_p95",
        "route_ms_p99",
        "first_token_ms_p50",
        "first_token_ms_p95",
        "first_token_ms_p99",
        "total_ms_p50",
        "total_ms_p95",
        "total_ms_p99",
        "server_proc_ms_p50",
        "server_proc_ms_p95",
        "server_proc_ms_p99",
        "vram_used_gb_max",
        "worker_cur_load_max",
        "worker_cur_load_p95",
        "worker_cur_perf_avg",
        "wer_pct",
        "cer_pct",
        "status_counts",
    ]

    with csv_path.open("w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=summary_fields)
        w.writeheader()
        for item in results:
            row = dict(item["summary"])
            row["status_counts"] = json.dumps(row.get("status_counts", {}), ensure_ascii=False)
            w.writerow(row)

    with details_csv_path.open("w", newline="") as f:
        fields = [
            "mode",
            "concurrency",
            "client_idx",
            "req_idx",
            "sample_id",
            "duration_s",
            "route_ms",
            "connect_ms",
            "first_token_ms",
            "total_ms",
            "server_processing_ms",
            "status_code",
            "ok",
            "queue_full",
            "error",
            "wer",
            "cer",
            "reference",
            "hypothesis",
        ]
        w = csv.DictWriter(f, fieldnames=fields)
        w.writeheader()
        for item in results:
            s = item["summary"]
            for rec in item["records"]:
                w.writerow(
                    {
                        "mode": s["mode"],
                        "concurrency": s["concurrency"],
                        **{k: rec.get(k, "") for k in fields if k not in ("mode", "concurrency")},
                    }
                )

    with json_path.open("w") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)

    return str(csv_path), str(details_csv_path), str(json_path)

async def main_async():
    p = argparse.ArgumentParser(description="Full stress + quality tester for Hindi ASR")
    p.add_argument("--url", default=None, help="Direct URL, e.g. http://ip:8000")
    p.add_argument("--endpoint", default=None, help="Vast endpoint name")
    p.add_argument("--endpoint-key", default=None, help="Endpoint API key")
    p.add_argument("--endpoint-id", type=int, default=None, help="Endpoint id")
    p.add_argument("--route-cost", type=float, default=10.0)
    p.add_argument("--mode", choices=["http", "sse", "both"], default="both")
    p.add_argument("--levels", default="1,5,10,25,50,75,100")
    p.add_argument("--requests-per-client", type=int, default=1)
    p.add_argument("--cooldown", type=float, default=5.0)
    p.add_argument("--metrics-interval", type=float, default=2.0)
    p.add_argument("--inter-request-delay", type=float, default=0.0)

    p.add_argument("--audio-source", choices=["sine", "fleurs"], default="fleurs")
    p.add_argument("--audio-duration", type=float, default=10.0, help="Used for sine source")
    p.add_argument("--audio-durations", default="", help="Comma-separated durations for sine source")
    p.add_argument("--sample-pool", type=int, default=120, help="Pool size for fleurs samples")
    p.add_argument("--min-duration", type=float, default=3.0)
    p.add_argument("--max-duration", type=float, default=20.0)
    p.add_argument("--shuffle-samples", action="store_true")
    p.add_argument("--seed", type=int, default=42)
    p.add_argument("--output-prefix", default="")
    args = p.parse_args()

    if not args.url and not args.endpoint:
        print("ERROR: pass --url or --endpoint")
        sys.exit(2)

    endpoint_name = args.endpoint
    endpoint_key = args.endpoint_key
    endpoint_id = args.endpoint_id
    direct_url = args.url.rstrip("/") if args.url else None

    if endpoint_name:
        if not endpoint_key or not endpoint_id:
            info = get_endpoint_info(endpoint_name)
            endpoint_key = endpoint_key or info.get("api_key")
            endpoint_id = endpoint_id or info.get("id")
        if not endpoint_key or endpoint_id is None:
            raise RuntimeError("Could not resolve endpoint_key/endpoint_id")

    if args.audio_source == "sine":
        if args.audio_durations:
            durations = [float(x) for x in args.audio_durations.split(",") if x.strip()]
        else:
            durations = [args.audio_duration]
        samples = make_sine_samples(durations)
    else:
        samples = load_fleurs_samples(
            max_samples=args.sample_pool,
            min_duration=args.min_duration,
            max_duration=args.max_duration,
            seed=args.seed,
        )

    levels = [int(x) for x in args.levels.split(",") if x.strip()]
    modes = [args.mode] if args.mode != "both" else ["http", "sse"]

    print("=== Full Stress Test ===")
    print(f"Target: {'endpoint ' + endpoint_name if endpoint_name else direct_url}")
    print(f"Modes: {modes}")
    print(f"Levels: {levels}")
    print(f"Audio source: {args.audio_source} (pool={len(samples)})")
    print(f"Requests/client: {args.requests_per_client}")

    results: List[Dict[str, Any]] = []
    for mode in modes:
        for lvl in levels:
            item = await run_level(
                mode=mode,
                concurrency=lvl,
                requests_per_client=args.requests_per_client,
                samples=samples,
                endpoint_name=endpoint_name,
                endpoint_key=endpoint_key,
                endpoint_id=endpoint_id,
                route_cost=args.route_cost,
                metrics_interval=args.metrics_interval,
                direct_url=direct_url,
                shuffle_samples=args.shuffle_samples,
                seed=args.seed,
                inter_request_delay=args.inter_request_delay,
            )
            results.append(item)
            if lvl != levels[-1]:
                await asyncio.sleep(args.cooldown)

    ts = datetime.now().strftime("%Y%m%d_%H%M%S")
    prefix = args.output_prefix or f"stress_full_{ts}"
    csv_path, details_path, json_path = write_outputs(results, prefix)

    print("\n" + "=" * 90)
    print("SUMMARY")
    print("=" * 90)
    for item in results:
        s = item["summary"]
        print(
            f"{s['mode']:>4} c={s['concurrency']:>3} ok={s['ok']:>4}/{s['concurrency'] * s['requests_per_client']:<4} "
            f"err={s['errors']:>3} qfull={s['queue_full_errors']:>3} "
            f"p95_total={s['total_ms_p95']:>8.1f}ms p95_ft={s['first_token_ms_p95']:>8.1f}ms "
            f"tput={s['throughput_x']:>6.2f}x wer={s['wer_pct']:>6.2f}%"
        )

    print(f"\nSummary CSV : {csv_path}")
    print(f"Details CSV : {details_path}")
    print(f"Full JSON   : {json_path}")

def main():
    asyncio.run(main_async())

if __name__ == "__main__":
    main()
PY


Output:
Chunk ID: e74eab
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 0
Output:


Summary: auto


**Preparing syntax check**


The new harness is in place. I’m running a fast syntax/lint check and a tiny dry run against the endpoint to validate it before launching the full stress matrix.


python3 -m py_compile production/test_stress_full.py


python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode http --levels 1 --requests-per-client 1 --audio-source fleurs --sample-pool 8 --min-duration 3 --max-duration 12 --metrics-interval 2 --route-cost 10 --output-prefix sanity_full_http_c1


Output:
Chunk ID: fe7977
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 0
Output:


Output:
Chunk ID: 84e131
Wall time: 6.2852 seconds
Process exited with code 0
Original token count: 244
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['http']
Levels: [1]
Audio source: fleurs (pool=8)
Requests/client: 1

==========================================================================================
Level mode=http c=1 reqs/client=1
==========================================================================================
OK=1/1 err=0 qfull=0 tput=6.43x total_p95=1581ms ft_p95=1581ms wer=100.00%

==========================================================================================
SUMMARY
==========================================================================================
http c=  1 ok=   1/1    err=  0 qfull=  0 p95_total=  1581.5ms p95_ft=  1581.5ms tput=  6.43x wer=100.00%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1.json


**Inspecting CSV output details**


sed -n '1,6p' benchmark_results/sanity_full_http_c1_details.csv


Output:
Chunk ID: 93a8e3
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 170
Output:
mode,concurrency,client_idx,req_idx,sample_id,duration_s,route_ms,connect_ms,first_token_ms,total_ms,server_processing_ms,status_code,ok,queue_full,error,wer,cer,reference,hypothesis
http,1,0,0,1734,10.2,148.11158925294876,1432.7744776383042,1581.4585210755467,1581.4585210755467,589.0,200,True,False,,1.0,0.9847328244274809,वैज्ञानिक यह निष्कर्ष निकालने में सक्षम थे कि काला पदार्थ अन्य काले पदार्थ को उसी तरह प्रभावित करता है जिस तरह से आम पदार्थ करता है,वै


**Investigating truncated transcription issue**


python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode sse --levels 1 --requests-per-client 1 --audio-source fleurs --sample-pool 8 --min-duration 3 --max-duration 12 --metrics-interval 2 --route-cost 10 --output-prefix sanity_full_sse_c1


Output:
Chunk ID: 68d210
Wall time: 6.6625 seconds
Process exited with code 0
Original token count: 242
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['sse']
Levels: [1]
Audio source: fleurs (pool=8)
Requests/client: 1

==========================================================================================
Level mode=sse c=1 reqs/client=1
==========================================================================================
OK=1/1 err=0 qfull=0 tput=6.50x total_p95=1569ms ft_p95=1051ms wer=100.00%

==========================================================================================
SUMMARY
==========================================================================================
 sse c=  1 ok=   1/1    err=  0 qfull=  0 p95_total=  1569.1ms p95_ft=  1051.0ms tput=  6.50x wer=100.00%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_sse_c1.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_sse_c1_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_sse_c1.json


sed -n '1,6p' benchmark_results/sanity_full_sse_c1_details.csv


Output:
Chunk ID: c07f78
Wall time: 0.0503 seconds
Process exited with code 0
Original token count: 169
Output:
mode,concurrency,client_idx,req_idx,sample_id,duration_s,route_ms,connect_ms,first_token_ms,total_ms,server_processing_ms,status_code,ok,queue_full,error,wer,cer,reference,hypothesis
sse,1,0,0,1734,10.2,141.45752508193254,837.6300176605582,1050.9953554719687,1569.0886294469237,0.0,200,True,False,,1.0,0.9847328244274809,वैज्ञानिक यह निष्कर्ष निकालने में सक्षम थे कि काला पदार्थ अन्य काले पदार्थ को उसी तरह प्रभावित करता है जिस तरह से आम पदार्थ करता है, वै


**Testing sample selection and quality**


python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode http --levels 1 --requests-per-client 10 --audio-source fleurs --sample-pool 20 --min-duration 3 --max-duration 15 --shuffle-samples --seed 7 --metrics-interval 2 --output-prefix sanity_full_http_c1_r10


Output:
Chunk ID: 2135ab
Wall time: 22.4340 seconds
Process exited with code 0
Original token count: 248
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['http']
Levels: [1]
Audio source: fleurs (pool=20)
Requests/client: 10

==========================================================================================
Level mode=http c=1 reqs/client=10
==========================================================================================
OK=10/10 err=0 qfull=0 tput=7.38x total_p95=1762ms ft_p95=1762ms wer=44.54%

==========================================================================================
SUMMARY
==========================================================================================
http c=  1 ok=  10/10   err=  0 qfull=  0 p95_total=  1762.2ms p95_ft=  1762.2ms tput=  7.38x wer= 44.54%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1_r10.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1_r10_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sanity_full_http_c1_r10.json


**Inspecting initial WER results**


sed -n '1,14p' benchmark_results/sanity_full_http_c1_r10_details.csv


Output:
Chunk ID: 0fc8ed
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1799
Output:
mode,concurrency,client_idx,req_idx,sample_id,duration_s,route_ms,connect_ms,first_token_ms,total_ms,server_processing_ms,status_code,ok,queue_full,error,wer,cer,reference,hypothesis
http,1,0,0,1803,11.04,142.5088131800294,1552.816436626017,1695.9975305944681,1695.9975305944681,607.0,200,True,False,,0.15,0.05319148936170213,आपके हाथों की नमी बाहरी परतों के साथ क्रिया करेगी जो अजीब महसूस करेगी और एक तरह का आवरण बनाएगी,आपके हाथों की नमी बाहरी पड़तों के साथ क्रिया करेगी जो अजीव महसूस करेगी और एक तरह का आवरण बनायेगी
http,1,0,1,1770,8.88,34.80921778827906,1302.1319452673197,1337.4793687835336,1337.4793687835336,491.0,200,True,False,,0.15789473684210525,0.04819277108433735,यह तर्कसंगत लग रहा है क्योंकि ऐसा महसूस नहीं होता है कि पृथ्वी चल रही है है कि नहीं,यह तर्क संगत लग रहा है क्योंकि ऐसा महसूस नहीं होता है कि पृथ्वी चल रही है कि नहीं
http,1,0,2,1933,13.8,34.743351861834526,1726.6709441319108,1762.224138714373,1762.224138714373,766.0,200,True,False,,0.4583333333333333,0.2066115702479339,अंत में कई ऐसी छोटी बिल्लियाँ लूज़ पेट कैट्स सहित हैं जो कीट रोडंट्स छिपकलियों और पक्षियों जैसे कई छोटे जीवों को खाती हैं,<s> और अब अंत में कई ऐसी छोटी बिलियां लूस बेड कैट सहित हैं जो किट रोड एंड छिपकलियों और पक्षियों जैसे कई छोटे जीवों को खात
http,1,0,3,1660,13.74,35.100908018648624,1692.5665913149714,1728.423680178821,1728.423680178821,740.0,200,True,False,,0.5,0.21428571428571427,रूमानियत में सांस्कृतिक नियतिवाद का बड़ा तत्व था जो गोएथे फिच्ते और श्लेगल जैसे लेखकों से लिया गया,हमारी में सांस्कृतिक नियुक्ति वात्रा बड़ा महत्व था जो गोई थे फिचते और स्लेकल जैसे लेखको से लिया गया
http,1,0,4,1683,12.18,34.76431220769882,1670.2902484685183,1705.6707870215178,1705.6707870215178,680.0,200,True,False,,0.13157894736842105,0.0658682634730539,यहूदी और गैर-यहूदी की तरह अभी भी ऐसे कई पुरुष और महिलाएं जीवित हैं जो उस समय बचने में सफल रहे और कई जिनके प्रियजनों को मार दिया गया या उनके मरने तक उनसे काम करवाया गया,यूदी और गैर योदी की तरह अभी भी ऐसे कई पुरूष महिलाएं जीवित हैं जो उस समय बचने में सफल रहे और कई जिनके प्रियजनों को मार दिया गया या उनके मरने तक उनसे काम करवाया
http,1,0,5,1728,10.02,34.283872693777084,1324.2454286664724,1359.1045094653964,1359.1045094653964,537.0,200,True,False,,0.8571428571428571,0.6119402985074627,गोभी का जूस अम्लीय या आधारभूत क्षारीय रसायन के आधार पर रंग बदलता है,गो भी कास अमलीय या आधारभूत र
http,1,0,6,1709,10.86,34.43153202533722,1537.9801467061043,1573.0068935081363,1573.0068935081363,622.0,200,True,False,,0.0,0.0,यह रिपोर्ट इराक के प्रति प्रशासन की वर्तमान नीति के लगभग सभी पहलुओं के प्रति बेहद आलोचनात्मक है और यह इसकी दिशा में तत्काल परिवर्तन का आग्रह करती है,यह रिपोर्ट इराक के प्रति प्रशासन की वर्तमान नीति के लगभग सभी पहलुओं के प्रति बेहद आलोचनात्मक है और यह इसकी दिशा में तत्काल परिवर्तन का आग्रह करती है
http,1,0,7,1996,14.28,36.160582676529884,1718.2979388162494,1755.5067148059607,1755.5067148059607,799.0,200,True,False,,0.2857142857142857,0.07975460122699386,पुरुषों के स्टैंडिंग सुपर-जी में ऑस्ट्रेलिया के मिशेल गौरले ग्यारहवें स्थान पर रहे चेक प्रतियोगी ओल्डरिच जेलिनेक पुरुषों के सिटिंग सुपर-जी में सोलहवें स्थान पर रहे,पुरूषों के स्टैंडिंग सुपर जीम में ऑस्ट्रेलिया के मिशेल गौरले ग्यारहवें स्थान पर रहे चेक प्रतियोगी ऑल ड्रिज जैलीनेक पुरूषों के सेटिंग सुपर जी में सोल्बे स्थान पर रहे
http,1,0,8,1849,12.24,36.19276266545057,1613.2682925090194,1650.1999832689762,1650.1999832689762,667.0,200,True,False,,1.0,0.9915966386554622,अगर आपने फ़िल्म नेशनल ट्रेज़र देखी है तो आपको लगेगा कि स्वतंत्रता के घोषणा-पत्र के पीछे एक ख़ज़ाने का नक्शा लिखा गया था,अ
http,1,0,9,1845,13.32,34.82307121157646,1691.0846317186952,1726.6849325969815,1726.6849325969815,717.0,200,True,False,,0.9130434782608695,0.7256637168141593,महाराजा और महारानी के ख़िलाफ़ चीखते-चिल्लाते लोगों की भीड़ से घिरी एक गाड़ी में सफर करते हुए वे लोग पेरिस वापस आए,महाराणा और महाराने के खिलाफ चीख ने


**Investigating alternative batch transcription**


Early validation surfaced an important quality risk: some HTTP/SSE responses are truncated, causing high WER spikes. I’m pausing the large sweep briefly to inspect the pipeline API and fix this before we collect final production metrics; otherwise scaling numbers would be based on a regressed quality state.


sed -n '1,320p' nemotron_asr/pipelines.py


sed -n '320,760p' nemotron_asr/pipelines.py


rg -n "class .*Pipeline|def run\(|transcribe|streaming|open_streaming_session|process_streaming_batch|final_transcript|partial_transcript" nemotron_asr/pipelines.py -n


Output:
Chunk ID: 2fdd0d
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 3634
Output:
from __future__ import annotations

import os
import re
from abc import abstractmethod
from typing import TYPE_CHECKING, Iterable, Any
import time
import math
import numpy as np
import torch
from omegaconf import DictConfig, OmegaConf
from torch import Tensor

from nemo.collections.asr.inference.model_wrappers.asr_inference_wrapper import ASRInferenceWrapper
from nemo.collections.asr.inference.pipelines.pipeline_interface import PipelineInterface
from nemo.collections.asr.inference.streaming.buffering.audio_bufferer import BatchedAudioBufferer
from nemo.collections.asr.inference.streaming.buffering.cache_feature_bufferer import BatchedCacheFeatureBufferer
from nemo.collections.asr.inference.streaming.buffering.feature_bufferer import BatchedFeatureBufferer
from nemo.collections.asr.inference.streaming.framing.multi_stream import ContinuousBatchedRequestStreamer
from nemo.collections.asr.inference.streaming.framing.request import FeatureBuffer, Frame, Request
from nemo.collections.asr.inference.streaming.framing.request_options import ASRRequestOptions
from nemo.collections.asr.inference.streaming.state.state import StreamingState
from nemo.collections.asr.inference.streaming.text.text_processing import StreamingTextProcessor
from nemo.collections.asr.inference.utils.bpe_decoder import BPEDecoder
from nemo.collections.asr.inference.utils.context_manager import CacheAwareContextManager
from nemo.collections.asr.inference.utils.enums import RequestType
from nemo.collections.asr.inference.utils.pipeline_utils import (
    check_existance_of_required_attributes,
    get_leading_punctuation_regex_pattern,
    ids_to_text_without_stripping,
    get_confidence_utils,
)
from nemo.collections.asr.inference.utils.progressbar import ProgressBar
from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec
from nemo.collections.asr.inference.pipelines.base_pipeline import TranscribeStepOutput
from nemo.collections.asr.inference.model_wrappers.cache_aware_rnnt_inference_wrapper import CacheAwareRNNTInferenceWrapper
from nemo.collections.asr.inference.streaming.decoders.greedy.greedy_rnnt_decoder import RNNTGreedyDecoder
from nemo.collections.asr.inference.streaming.endpointing.greedy.greedy_rnnt_endpointing import RNNTGreedyEndpointing
from nemo.collections.asr.inference.streaming.state.cache_aware_rnnt_state import CacheAwareRNNTStreamingState
from nemo.collections.asr.inference.utils.endpointing_utils import millisecond_to_frames
from nemo.collections.asr.parts.utils.rnnt_utils import Hypothesis
from nemo.collections.asr.inference.factory.buffered_pipeline_builder import BufferedPipelineBuilder
from nemo.collections.asr.inference.utils.enums import PipelineType
from nemo.collections.asr.inference.factory.base_builder import BaseBuilder
from nemo.collections.asr.inference.pipelines.cache_aware_ctc_pipeline import CacheAwareCTCPipeline
from nemo.collections.asr.inference.utils.enums import ASRDecodingType
from nemo.collections.asr.parts.submodules.ctc_decoding import CTCDecodingConfig
from nemo.collections.asr.parts.submodules.rnnt_decoding import RNNTDecodingConfig
from nemo.utils import logging

if TYPE_CHECKING:
    from nemo.collections.asr.inference.itn.inverse_normalizer import AlignmentPreservingInverseNormalizer
    from nemo.collections.asr.inference.nmt.llm_translator import LLMTranslator

class BaseStreamingPipeline(PipelineInterface):
    """
    Base class for all pipelines.
    """

    def __init__(self):
        """Initialize state pool to store the state for each stream"""
        self._state_pool: dict[int, StreamingState] = {}

    def get_state(self, stream_id: int) -> StreamingState:
        """Retrieve state for a given stream ID."""
        return self._state_pool.get(stream_id, None)

    def get_states(self, stream_ids: Iterable[int]) -> list[StreamingState]:
        """Retrieve states for a list of stream IDs."""
        return [self.get_state(stream_id) for stream_id in stream_ids]

    def delete_state(self, stream_id: int) -> None:
        """Delete the state from the state pool."""
        if stream_id in self._state_pool:
            del self._state_pool[stream_id]

    def delete_states(self, stream_ids: Iterable[int]) -> None:
        """Delete states for a list of stream IDs."""
        for stream_id in stream_ids:
            self.delete_state(stream_id)

    def init_state(self, stream_id: int, options: ASRRequestOptions) -> StreamingState:
        """Initialize the state of the stream"""
        if stream_id not in self._state_pool:
            state = self.create_state(options)
            self._state_pool[stream_id] = state
        return self._state_pool[stream_id]

    def reset_session(self) -> None:
        """Reset the frame buffer and internal state pool"""
        self._state_pool.clear()

    def open_session(self) -> None:
        """Start a new session by resetting the internal state pool"""
        self.reset_session()

    def close_session(self) -> None:
        """Close the session by resetting the internal state pool"""
        self.reset_session()

    def open_streaming_session(self, options: ASRRequestOptions | None = None) -> int:
        """
        Open a new streaming session for real-time audio
        Args:
            options (ASRRequestOptions | None): Optional options for the stream
        Returns:
            int: The stream ID for this session
        """
        if not hasattr(self, '_streaming_request_generator'):
            raise RuntimeError("Pipeline not configured for streaming. Call init_streaming_request_generator() first.")

        # Create default options if none provided
        if options is None:
            options = ASRRequestOptions()

        return self._streaming_request_generator.open_stream(options)

    def append_streaming_audio(self, stream_id: int, audio_samples) -> None:
        """
        Append audio samples to an active streaming session
        Args:
            stream_id (int): The stream ID
            audio_samples: Audio samples (torch.Tensor or np.ndarray)
        """
        if not hasattr(self, '_streaming_request_generator'):
            raise RuntimeError("Pipeline not configured for streaming.")
        self._streaming_request_generator.append_audio(stream_id, audio_samples)

    def close_streaming_session(self, stream_id: int) -> None:
        """
        Mark a streaming session as ended (stops accepting new audio).
        State cleanup is deferred until all buffered frames are processed.

        NOTE: This method no longer immediately deletes state. Instead, it just
        marks the stream as ended in the request generator. The actual cleanup
        of _state_pool and context_manager happens later when the inference loop
        confirms all frames have been processed.

        Args:
            stream_id (int): The stream ID to close
        """
        # Close the stream in the request generator (marks as ended, stops accepting audio)
        if hasattr(self, '_streaming_request_generator'):
            self._streaming_request_generator.close_stream(stream_id)

        # State cleanup is now deferred - DO NOT delete from _state_pool here
        # The caller (e.g., cache_aware_pipeline_modal.py) should handle cleanup
        # after confirming all buffered frames have been processed

    async def process_streaming_batch(self):
        """
        Process the next batch of ready frames from streaming sessions.
        This is async to work with WebSocket handlers, but calls sync inference internally.
        Returns:
            tuple: (requests, outputs) where:
                - requests: list[Request] - The frames/feature buffers that were processed
                - outputs: list[TranscribeStepOutput] - Transcription outputs
            Returns ([], []) if no frames ready
        """
        if not hasattr(self, '_streaming_request_generator'):
            raise RuntimeError("Pipeline not configured for streaming.")

        try:
            # Try to get ready frames (non-blocking)
            requests = next(self._streaming_request_generator)
            # Process the batch (sync inference)
            # t0 = time.perf_counter()
            outputs = self.transcribe_step(requests)
            # t1 = time.perf_counter()
            # print(f"   🧹 Transcribe step took {t1 - t0} seconds")
            return requests, outputs
        except StopIteration:
            # All streams ended and consumed
            return [], []
        except Exception as e:
            # Check if it's NotEnoughDataException (can't import type here)
            if type(e).__name__ == 'NotEnoughDataException':
                # No frames ready yet
                return [], []
            # Re-raise other exceptions
            raise

    @abstractmethod
    def transcribe_step_for_frames(self, frames: list[Frame]) -> None:
        """Transcribe a step for frames"""
        pass

    @abstractmethod
    def transcribe_step_for_feature_buffers(self, fbuffers: list[FeatureBuffer]) -> None:
        """Transcribe a step for feature buffers"""
        pass

    @abstractmethod
    def get_sep(self) -> str:
        """Return the separator for the text postprocessor."""
        pass

    def translate_step(self, states: list[StreamingState], step_outputs: list[TranscribeStepOutput]) -> None:
        """
        Translate step
        Args:
            states (list[StreamingState]): List of StreamingState objects.
            step_outputs (list[TranscribeStepOutput]): List of TranscribeStepOutput objects.
        """
        src_langs, tgt_langs = [], []
        asr_transcripts, current_prefixes, previous_translations = [], [], []
        final_transcript_mask = []
        states_to_translate = []

        src_contexts, tgt_contexts = [], []
        for state, step_output in zip(states, step_outputs):
            if not state.options.enable_nmt:
                continue

            src_lang = state.options.source_language
            tgt_lang = state.options.target_language
            if not src_lang or not tgt_lang:
                raise ValueError("Source and target languages must be set when NMT is enabled")

            final = step_output.final_transcript
            partial = step_output.partial_transcript
            if not (final.strip() or partial.strip()):
                continue

            transcript = final or partial
            is_final = bool(final)
            prev_translation, prefix = state.previous_translation_info

            states_to_translate.append((state, step_output))
            src_langs.append(src_lang)
            tgt_langs.append(tgt_lang)
            asr_transcripts.append(transcript)
            current_prefixes.append(prefix)
            previous_translations.append(prev_translation)
            final_transcript_mask.append(is_final)

            src_context, tgt_context = state.previous_context
            src_contexts.append(src_context)
            tgt_contexts.append(tgt_context)

        if len(states_to_translate) == 0:
            return

        translations = self.nmt_model.translate(
            asr_transcripts, current_prefixes, src_langs, tgt_langs, src_contexts, tgt_contexts
        )
        new_prefixes = self.nmt_model.get_prefixes(asr_transcripts, translations, previous_translations)

        for (state, step_output), translation, new_prefix, prev_prefix, is_final in zip(
            states_to_translate, translations, new_prefixes, current_prefixes, final_transcript_mask
        ):
            if is_final:
                step_output.final_translation = translation
                step_output.partial_translation = ""
                state.cleanup_translation_info_after_eou()
                state.set_translation_context(step_output.final_transcript, translation)
                new_prefix = translation
            else:
                step_output.partial_translation = translation
                step_output.final_translation = ""
                state.set_translation_info(translation, new_prefix)

            lcp = os.path.commonprefix([prev_prefix, new_prefix])
            step_output.current_step_translation = new_prefix[len(lcp) :]

    def transcribe_step(self, requests: list[Request]) -> list[TranscribeStepOutput]:
        """
        Transcribe step
        Args:
            requests (list[Request]): List of Request objects.
        Returns:
            list[TranscribeStepOutput]: List of TranscribeStepOutput objects.
        """

        # Initialize the state if it is the first request for the stream
        states = []
        for request in requests:
            if request.is_first:
                self.init_state(request.stream_id, request.options)
            states.append(self.get_state(request.stream_id))

        # Perform the transcribe step for the frames or feature buffers
        if isinstance(requests[0], Frame):
            self.transcribe_step_for_frames(frames=requests)
        elif isinstance(requests[0], FeatureBuffer):
            self.transcribe_step_for_feature_buffers(fbuffers=requests)
        else:
            raise ValueError(f"Invalid request type: {type(requests[0])}")

        # Create current step output for each request
        outputs = []
        sep = self.get_sep()
        for request, state in zip(requests, states):
            step_output = TranscribeStepOutput.from_state(state=state, request=request, sep=sep)
            outputs.append(step_output)

        # Perform the translation step
        if self.nmt_enabled:
            self.translate_step(states=states, step_outputs=outputs)

        # Cleanup the states after the response is sent
        # If last request, delete state from the state pool to free memory
        for state, request in zip(states, requests):
            state.cleanup_after_response()
            if request.is_last:
                self.delete_state(request.stream_id)
        return outputs

    def copy_asr_model_attributes(self, asr_model: ASRInferenceWrapper) -> None:
        """
        Copy the attributes from the ASR model
        Args:
            asr_model (ASRInferenceWrapper): ASR model to copy the attributes from.
        """
        self.asr_model = asr_model
        self.tokenizer = asr_model.tokenizer
        self.device = asr_model.device
        self.supports_punctuation = asr_model.supports_punctuation()
        self.asr_supported_puncts = asr_model.supported_punctuation()


Output:
Chunk ID: 0946dc
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 4930
Output:
        self.asr_supported_puncts = asr_model.supported_punctuation()
        self.leading_regex_pattern = get_leading_punctuation_regex_pattern(self.asr_supported_puncts)
        self.blank_id = asr_model.get_blank_id()
        self.vocabulary = asr_model.get_vocabulary()
        self.sep = asr_model.word_separator
        self.underscore_id = asr_model.underscore_id
        self.punctuation_ids = asr_model.punctuation_ids
        self.language_token_ids = asr_model.language_token_ids
        self.preprocessor, self.preprocessor_config = asr_model.create_preprocessor()
        self.subsampling_factor = asr_model.get_subsampling_factor()
        self.window_stride = asr_model.get_window_stride()
        self.model_stride_in_secs = asr_model.get_model_stride(in_secs=True)
        self.model_stride_in_milliseconds = asr_model.get_model_stride(in_milliseconds=True)

    def update_partial_transcript(
        self, requests: list[Request], tokenizer: TokenizerSpec, leading_regex_pattern: str
    ) -> None:
        """
        Update partial and current step transcripts from the state.
        Args:
            requests (list[Request]): List of Request objects.
            tokenizer (TokenizerSpec): Used to convert tokens into text
            leading_regex_pattern (str): Regex pattern for the punctuation marks.
        """
        word_separator = self.get_sep()
        for request in requests:
            state = self.get_state(request.stream_id)
            # state tokens represent all tokens accumulated since the EOU
            # incomplete segment tokens are the remaining tokens on the right side of the buffer after EOU
            all_tokens = state.tokens + state.incomplete_segment_tokens
            if len(all_tokens) > 0:
                pt_string = ids_to_text_without_stripping(all_tokens, tokenizer, word_separator)
                if leading_regex_pattern:
                    pt_string = re.sub(leading_regex_pattern, r'\1', pt_string)
                state.partial_transcript = pt_string
            else:
                state.partial_transcript = ""

            current_step_tokens = state.current_step_tokens
            if len(current_step_tokens) > 0:
                step_transcript = ids_to_text_without_stripping(current_step_tokens, tokenizer, word_separator)
                state.current_step_transcript = step_transcript
            else:
                state.current_step_transcript = ""

    def init_bpe_decoder(self) -> None:
        """Initialize the BPE decoder"""
        check_existance_of_required_attributes(
            self,
            [
                'vocabulary',
                'tokenizer',
                'confidence_aggregator',
                'asr_supported_puncts',
                'word_boundary_tolerance',
                'model_stride_in_secs',
            ],
        )

        self.bpe_decoder = BPEDecoder(
            vocabulary=self.vocabulary,
            tokenizer=self.tokenizer,
            confidence_aggregator=self.confidence_aggregator,
            asr_supported_puncts=self.asr_supported_puncts,
            word_boundary_tolerance=self.word_boundary_tolerance,
            token_duration_in_secs=self.model_stride_in_secs,
        )

    def init_text_processor(
        self,
        cfg: DictConfig,
        itn_model: AlignmentPreservingInverseNormalizer | None,
    ) -> None:
        """
        Initialize the text processor.
        Args:
            cfg: (DictConfig) Configuration parameters.
            itn_model: (AlignmentPreservingInverseNormalizer | None) Inverse Text Normalization model.
        """
        check_existance_of_required_attributes(
            self,
            [
                'asr_supported_puncts',
                'supports_punctuation',
                'confidence_aggregator',
                'sep',
            ],
        )

        self.text_processor = StreamingTextProcessor(
            itn_cfg=cfg.itn,
            itn_model=itn_model,
            asr_supported_puncts=self.asr_supported_puncts,
            asr_supports_punctuation=self.supports_punctuation,
            confidence_aggregator=self.confidence_aggregator,
            sep=self.sep,
            enable_pnc=cfg.enable_pnc,
            enable_itn=cfg.enable_itn,
        )

    def init_nmt_model(self, nmt_model: LLMTranslator | None) -> None:
        """
        Initialize the Translation model.
        Args:
            nmt_model: (LLMTranslator | None) LLM based translation model.
        """
        self.nmt_model = nmt_model
        self.nmt_enabled = nmt_model is not None

    def init_bufferer_for_buffered_streaming(self) -> None:
        """Initialize the bufferer."""
        check_existance_of_required_attributes(
            self,
            [
                'request_type',
                'sample_rate',
                'buffer_size_in_secs',
                'preprocessor_config',
                'device',
            ],
        )

        if self.request_type is RequestType.FEATURE_BUFFER:
            # Feature buffering: It will be used when the input is feature buffers
            self.bufferer = BatchedFeatureBufferer(
                sample_rate=self.sample_rate,
                buffer_size_in_secs=self.buffer_size_in_secs,
                preprocessor_cfg=self.preprocessor_config,
                device=self.device,
            )
        elif self.request_type is RequestType.FRAME:
            # Audio buffering: It will be used when the input is audio frames
            self.bufferer = BatchedAudioBufferer(
                sample_rate=self.sample_rate, buffer_size_in_secs=self.buffer_size_in_secs
            )
        else:
            raise ValueError(f"Unknown request type: {self.request_type}")

    def init_bufferer_for_cache_aware_streaming(self) -> None:
        """Initialize the bufferer for cache-aware streaming."""
        check_existance_of_required_attributes(
            self,
            [
                'num_slots',
                'use_feat_cache',
                'chunk_size_in_secs',
                'buffer_size_in_secs',
                'sample_rate',
                'preprocessor_config',
                'device',
            ],
        )

        if self.use_feat_cache:
            # Only calculate mel-spec features for last chunk
            chunk_size_for_feature_buffer = self.chunk_size_in_secs
        else:
            # Calculate mel-spec features for the whole buffer
            chunk_size_for_feature_buffer = self.buffer_size_in_secs

        self.bufferer = BatchedCacheFeatureBufferer(
            num_slots=self.num_slots,
            sample_rate=self.sample_rate,
            buffer_size_in_secs=self.buffer_size_in_secs,
            chunk_size_in_secs=chunk_size_for_feature_buffer,
            preprocessor_cfg=self.preprocessor_config,
            device=self.device,
        )

    def init_context_manager(self) -> None:
        """Initialize the context manager."""
        check_existance_of_required_attributes(self, ['asr_model', 'num_slots', 'use_cache'])
        self.context_manager = CacheAwareContextManager(
            cache_aware_model=self.asr_model, num_slots=self.num_slots, use_cache=self.use_cache
        )

class CacheAwareRNNTStreamingPipeline(BaseStreamingPipeline):
    """Cache Aware RNNT pipeline."""

    def __init__(
        self,
        cfg: DictConfig,
        asr_model: CacheAwareRNNTInferenceWrapper,
        itn_model: AlignmentPreservingInverseNormalizer | None = None,
        nmt_model: LLMTranslator | None = None,
    ):
        """
        Initialize the CacheAwareRNNTStreamingPipeline.
        Args:
            cfg: (DictConfig) Configuration parameters.
            asr_model: (CacheAwareRNNTInferenceWrapper) ASR model.
            itn_model: (AlignmentPreservingInverseNormalizer | None) Inverse Text Normalization model.
            nmt_model: (LLMTranslator | None) LLM based translation model.
        """
        self.copy_asr_model_attributes(asr_model)
        self.init_parameters(cfg)
        self.init_context_manager()
        self.init_bufferer_for_cache_aware_streaming()
        self.conf_func, self.confidence_aggregator = get_confidence_utils(cfg.confidence)
        self.init_bpe_decoder()
        self.init_greedy_rnnt_decoder()
        self.init_endpointer()
        self.init_text_processor(cfg, itn_model)
        self.init_nmt_model(nmt_model)
        super().__init__()

    def init_parameters(self, cfg: DictConfig) -> None:
        """
        Initialize the parameters.
        Args:
            cfg: (DictConfig) Configuration parameters.
        """
        if cfg.streaming.att_context_size is not None:
            self.asr_model.set_default_att_context_size(att_context_size=cfg.streaming.att_context_size)

        self.sample_rate = cfg.streaming.sample_rate
        self.asr_output_granularity = cfg.asr_output_granularity
        self.pre_encode_cache_size = self.asr_model.get_pre_encode_cache_size()
        self.model_chunk_size = self.asr_model.get_chunk_size()
        if isinstance(self.model_chunk_size, list):
            self.model_chunk_size = self.model_chunk_size[1]

        self.use_cache = cfg.streaming.use_cache
        self.use_feat_cache = cfg.streaming.use_feat_cache

        if cfg.streaming.get("chunk_size_in_secs", None) is not None:
            self.chunk_size_in_secs = cfg.streaming.chunk_size_in_secs
            self.tokens_per_frame = math.ceil(
                np.trunc(self.chunk_size_in_secs / self.window_stride) / self.subsampling_factor
            )
            # overwrite the encoder streaming params with proper shift size for cache aware streaming
            self.asr_model.setup_streaming_params(
                chunk_size=self.model_chunk_size // self.subsampling_factor, shift_size=self.tokens_per_frame
            )
        else:
            self.chunk_size_in_secs = self.model_chunk_size * self.window_stride
            self.tokens_per_frame = math.ceil(self.model_chunk_size / self.subsampling_factor)

        if isinstance(self.pre_encode_cache_size, list):
            self.pre_encode_cache_size = self.pre_encode_cache_size[1]
        self.pre_encode_cache_size_in_secs = self.pre_encode_cache_size * self.window_stride

        # Context Manager
        self.batch_size = cfg.streaming.batch_size
        self.num_slots = cfg.streaming.num_slots
        if self.num_slots < self.batch_size:
            raise ValueError(
                f"Number of slots in the context manager must be >= batch_size: {self.num_slots} < {self.batch_size}"
            )
        model_chunk_size_in_secs = self.model_chunk_size * self.window_stride

        if self.use_cache:
            # if using cache, we need to pad some samples for pre_encode
            self.buffer_size_in_secs = self.pre_encode_cache_size_in_secs + model_chunk_size_in_secs
            self.drop_left_context = None
            self.valid_out_len = None
        else:
            # if not using cache, we need to keep left context in buffer, but no extra padding in pre_encode
            left_context_size = self.asr_model.get_att_context_size()[0]
            if left_context_size < 0:
                raise ValueError(f"Left context size should not be a negative value: {left_context_size}")
            self.buffer_size_in_secs = (
                model_chunk_size_in_secs + left_context_size * self.subsampling_factor * self.window_stride
            )
            self.drop_left_context = left_context_size
            self.valid_out_len = self.tokens_per_frame

        self.stop_history_eou_in_milliseconds = cfg.endpointing.stop_history_eou
        self.residue_tokens_at_end = cfg.endpointing.residue_tokens_at_end
        self.word_boundary_tolerance = cfg.streaming.word_boundary_tolerance
        self.return_tail_result = cfg.return_tail_result

        self.request_type = RequestType.from_str(cfg.streaming.request_type)
        if self.request_type is not RequestType.FRAME:
            raise ValueError(f"Request type {self.request_type} is not supported for cache-aware streaming.")

    def init_greedy_rnnt_decoder(self) -> None:
        """Initialize the RNNT decoder."""
        check_existance_of_required_attributes(self, ['vocabulary', 'conf_func'])
        self.greedy_rnnt_decoder = RNNTGreedyDecoder(vocabulary=self.vocabulary, conf_func=self.conf_func)

    def init_endpointer(self) -> None:
        """Initialize the endpointer."""
        check_existance_of_required_attributes(
            self,
            [
                'vocabulary',
                'model_stride_in_milliseconds',
                'stop_history_eou_in_milliseconds',
                'residue_tokens_at_end',
            ],
        )

        self.endpointer = RNNTGreedyEndpointing(
            vocabulary=self.vocabulary,
            ms_per_timestep=self.model_stride_in_milliseconds,
            stop_history_eou=self.stop_history_eou_in_milliseconds,
            residue_tokens_at_end=self.residue_tokens_at_end,
        )

    def create_state(self, options: ASRRequestOptions) -> CacheAwareRNNTStreamingState:
        """
        Create new empty state.
        Args:
            options: (ASRRequestOptions) Request options for particular stream.
        Returns:
            (CacheAwareRNNTStreamingState) New empty state.
        """
        state = CacheAwareRNNTStreamingState()
        state.set_global_offset(0)
        new_options = options.augment_with_defaults(
            default_enable_itn=self.text_processor.is_itn_enabled(),
            default_enable_pnc=self.text_processor.is_pnc_enabled(),
            default_enable_nmt=self.nmt_enabled,
            default_source_language=self.nmt_model.source_language if self.nmt_enabled else None,
            default_target_language=self.nmt_model.target_language if self.nmt_enabled else None,
            default_stop_history_eou=self.stop_history_eou_in_milliseconds,
            default_asr_output_granularity=self.asr_output_granularity,
        )

        eou_label_buffer_size = 0
        if new_options.stop_history_eou > 0:
            eou_label_buffer_size = millisecond_to_frames(
                new_options.stop_history_eou, math.ceil(self.model_stride_in_milliseconds)
            )
            eou_label_buffer_size += self.residue_tokens_at_end
        state.setup_label_buffer(eou_label_buffer_size, self.blank_id)
        state.set_previous_hypothesis(None)
        state.set_options(new_options)
        return state

    def get_sep(self) -> str:
        """Return the separator for the text processor."""
        return self.sep

    def preprocess(self, buffers: list[Tensor], right_paddings: list[int] | None = None) -> tuple[Tensor, Tensor]:
        """
        Preprocess the feature buffers by stacking them and computing the lengths
        Args:
            buffers: (list[Tensor]) List of feature buffers.
            right_paddings: (list[int] | None) List of right paddings.
        Returns:
            (tuple[Tensor, Tensor]) Processed feature buffers and their lengths.
        """
        feature_buffers = [f_buffer.unsqueeze_(0) for f_buffer in buffers]
        feature_buffer_lens = torch.tensor([f_buffer.shape[2] for f_buffer in feature_buffers], device=self.device)
        if right_paddings is not None:
            right_paddings = torch.tensor(right_paddings, device=feature_buffer_lens.device)
            feature_buffer_lens = feature_buffer_lens - right_paddings
        feature_buffers = torch.cat(feature_buffers).to(self.device)
        return feature_buffers, feature_buffer_lens

    def run_greedy_decoder(self, state: CacheAwareRNNTStreamingState, frame: Frame, hyp: Hypothesis) -> bool:
        """
        Run the greedy RNNT decoder on the hypothesis and update the state
        Args:
            state: (CacheAwareRNNTStreamingState) The state of the stream
            frame: (Frame) The current frame
            hyp: (Hypothesis) The hypothesis of the current frame
        Returns:
            (bool) Whether EOU is detected.
        """
        eou_detected = frame.is_last
        cur_output, cur_labels, new_offset = self.greedy_rnnt_decoder(
            global_timestamps=hyp.timestamp,
            tokens=hyp.y_sequence,
            length=self.tokens_per_frame,
            offset=state.offset,
        )
        state.set_offset(new_offset)

        # cur labels contains blank tokens as well, it is needed for EOU detection
        state.update_label_buffer(cur_labels)

        if not eou_detected:
            emissions = state.get_label_buffer()
            pivot_point = len(emissions) - 1
            eou_detected, _ = self.endpointer.detect_eou_near_pivot(
                emissions, pivot_point, stop_history_eou=state.options.stop_history_eou
            )

        state.update_state(cur_output, eou_detected=eou_detected)
        return eou_detected

    def cache_aware_transcribe_step(
        self,
        frames: list[Frame],
        features: list[Tensor],
        right_paddings: list[int],
        ready_state_ids: set,
        keep_all_outputs: bool = False,
    ) -> None:
        """
        Cache Aware Transcribe Step
        It receives a list of frames and features and do the following:

        1. Preprocess the features by stacking them and computing the lengths
        2. Collecting previous hypotheses for stateful decoding
        3. Get the context and mapping from the context manager for cache aware streaming
        4. Perform a streaming step with the ASR model
        5. Update the cache and reset the cache slots for the streams that has ended
        6. Update the previous hypothesis and reset the previous hypothesis for the streams that has ended
        7. Perform greedy RNNT decoding to get the best hypothesis and update the states
        8. Update the ready states to indicate that the state is ready for text post-processing
        Args:
            frames: (list[Frame]) List of frames to transcribe.
            features: (list[Tensor]) List of feature buffers.
            right_paddings: (list[int] | None) List of right paddings.
            ready_state_ids: (set) Set of ready state IDs.
            keep_all_outputs: (bool) Whether to keep all outputs or not.
        """

        feature_buffers, feature_buffer_lens = self.preprocess(features, right_paddings)
        states, stream_ids, eos_flags = [], [], []
        for frame in frames:
            states.append(self.get_state(frame.stream_id))
            stream_ids.append(frame.stream_id)
            eos_flags.append(frame.is_last)

        previous_hypotheses = [state.get_previous_hypothesis() for state in states]
        context, mapping = self.context_manager.get_context(stream_ids)

        drop_extra_pre_encoded = 0 if not self.use_cache else self.asr_model.drop_extra_pre_encoded
        best_hyp, new_context = self.asr_model.stream_step(
            processed_signal=feature_buffers,
            processed_signal_length=feature_buffer_lens,
            context=context,
            previous_hypotheses=previous_hypotheses,
            drop_extra_pre_encoded=drop_extra_pre_encoded,
            keep_all_outputs=keep_all_outputs,
            drop_left_context=self.drop_left_context,
            valid_out_len=self.valid_out_len,
        )

        # update the cache and reset the cache slots for the streams that has ended
        self.context_manager.update_cache(stream_ids, new_context, mapping)
        self.context_manager.reset_slots(stream_ids, eos_flags)

        # update the previous hypothesis and reset the previous hypothesis for the streams that has ended
        for state, hyp, eos in zip(states, best_hyp, eos_flags):


Output:
Chunk ID: 386dde
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 1775
Output:
17:from nemo.collections.asr.inference.streaming.buffering.audio_bufferer import BatchedAudioBufferer
18:from nemo.collections.asr.inference.streaming.buffering.cache_feature_bufferer import BatchedCacheFeatureBufferer
19:from nemo.collections.asr.inference.streaming.buffering.feature_bufferer import BatchedFeatureBufferer
20:from nemo.collections.asr.inference.streaming.framing.multi_stream import ContinuousBatchedRequestStreamer
21:from nemo.collections.asr.inference.streaming.framing.request import FeatureBuffer, Frame, Request
22:from nemo.collections.asr.inference.streaming.framing.request_options import ASRRequestOptions
23:from nemo.collections.asr.inference.streaming.state.state import StreamingState
24:from nemo.collections.asr.inference.streaming.text.text_processing import StreamingTextProcessor
38:from nemo.collections.asr.inference.streaming.decoders.greedy.greedy_rnnt_decoder import RNNTGreedyDecoder
39:from nemo.collections.asr.inference.streaming.endpointing.greedy.greedy_rnnt_endpointing import RNNTGreedyEndpointing
40:from nemo.collections.asr.inference.streaming.state.cache_aware_rnnt_state import CacheAwareRNNTStreamingState
56:class BaseStreamingPipeline(PipelineInterface):
102:    def open_streaming_session(self, options: ASRRequestOptions | None = None) -> int:
104:        Open a new streaming session for real-time audio
110:        if not hasattr(self, '_streaming_request_generator'):
111:            raise RuntimeError("Pipeline not configured for streaming. Call init_streaming_request_generator() first.")
117:        return self._streaming_request_generator.open_stream(options)
119:    def append_streaming_audio(self, stream_id: int, audio_samples) -> None:
121:        Append audio samples to an active streaming session
126:        if not hasattr(self, '_streaming_request_generator'):
127:            raise RuntimeError("Pipeline not configured for streaming.")
128:        self._streaming_request_generator.append_audio(stream_id, audio_samples)
130:    def close_streaming_session(self, stream_id: int) -> None:
132:        Mark a streaming session as ended (stops accepting new audio).
144:        if hasattr(self, '_streaming_request_generator'):
145:            self._streaming_request_generator.close_stream(stream_id)
151:    async def process_streaming_batch(self):
153:        Process the next batch of ready frames from streaming sessions.
161:        if not hasattr(self, '_streaming_request_generator'):
162:            raise RuntimeError("Pipeline not configured for streaming.")
166:            requests = next(self._streaming_request_generator)
169:            outputs = self.transcribe_step(requests)
185:    def transcribe_step_for_frames(self, frames: list[Frame]) -> None:
190:    def transcribe_step_for_feature_buffers(self, fbuffers: list[FeatureBuffer]) -> None:
208:        final_transcript_mask = []
221:            final = step_output.final_transcript
222:            partial = step_output.partial_transcript
236:            final_transcript_mask.append(is_final)
251:            states_to_translate, translations, new_prefixes, current_prefixes, final_transcript_mask
257:                state.set_translation_context(step_output.final_transcript, translation)
267:    def transcribe_step(self, requests: list[Request]) -> list[TranscribeStepOutput]:
283:        # Perform the transcribe step for the frames or feature buffers
285:            self.transcribe_step_for_frames(frames=requests)
287:            self.transcribe_step_for_feature_buffers(fbuffers=requests)
334:    def update_partial_transcript(
354:                state.partial_transcript = pt_string
356:                state.partial_transcript = ""
429:    def init_bufferer_for_buffered_streaming(self) -> None:
458:    def init_bufferer_for_cache_aware_streaming(self) -> None:
459:        """Initialize the bufferer for cache-aware streaming."""
497:class CacheAwareRNNTStreamingPipeline(BaseStreamingPipeline):
518:        self.init_bufferer_for_cache_aware_streaming()
533:        if cfg.streaming.att_context_size is not None:
534:            self.asr_model.set_default_att_context_size(att_context_size=cfg.streaming.att_context_size)
536:        self.sample_rate = cfg.streaming.sample_rate
543:        self.use_cache = cfg.streaming.use_cache
544:        self.use_feat_cache = cfg.streaming.use_feat_cache
546:        if cfg.streaming.get("chunk_size_in_secs", None) is not None:
547:            self.chunk_size_in_secs = cfg.streaming.chunk_size_in_secs
551:            # overwrite the encoder streaming params with proper shift size for cache aware streaming
552:            self.asr_model.setup_streaming_params(
564:        self.batch_size = cfg.streaming.batch_size
565:        self.num_slots = cfg.streaming.num_slots
590:        self.word_boundary_tolerance = cfg.streaming.word_boundary_tolerance
593:        self.request_type = RequestType.from_str(cfg.streaming.request_type)
595:            raise ValueError(f"Request type {self.request_type} is not supported for cache-aware streaming.")
705:    def cache_aware_transcribe_step(
719:        3. Get the context and mapping from the context manager for cache aware streaming
720:        4. Perform a streaming step with the ASR model
726:            frames: (list[Frame]) List of frames to transcribe.
774:    def transcribe_step_for_feature_buffers(self, fbuffers: list[FeatureBuffer]) -> None:
776:        raise NotImplementedError("Feature buffer type is not supported for cache aware streaming.")
778:    def transcribe_step_for_frames(self, frames: list[Frame]) -> None:
780:        Transcribes the frames in a streaming manner.
784:            frames: (list[Frame]) List of frames to transcribe.
807:                self.cache_aware_transcribe_step(
812:                self.cache_aware_transcribe_step(
821:        self.update_partial_transcript(frames, self.tokenizer, self.leading_regex_pattern)
824:    def init_streaming_request_generator(self):
826:        Initialize the streaming request generator for real-time audio.
827:        This should be called before using streaming methods.
831:        self._streaming_request_generator = StreamingBatchedRequestStreamer(
844:class CacheAwareStreamingPipelineBuilder(BaseBuilder):
853:        Build the cache aware streaming pipeline based on the config.
864:            raise ValueError("Cache aware CTC pipeline is not implemented for streaming.")
866:        raise ValueError("Invalid asr decoding type for cache aware streaming. Need to be one of ['CTC', 'RNNT']")
897:        This loads the model, wraps it in CacheAwareRNNTInferenceWrapper for streaming.
987:        Build the cache aware RNNT streaming pipeline based on the config.
1015:        Build the cache aware CTC streaming pipeline based on the config.
1037:class StreamingPipelineBuilder:
1038:    """Router for building the streaming pipeline based on the pipeline type."""
1080:            raise ValueError("Buffered pipeline is not implemented for streaming.")
1084:            raise ValueError(f"Invalid streaming pipeline type: {cfg.pipeline_type}")


Summary: auto


**Inspecting pipeline processing methods**


sed -n '760,900p' nemotron_asr/pipelines.py