Output:
Chunk ID: fa6600
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 135
Output:
production/onstart.sh
production/test_stress_full.py
production/test_stress.py
production/worker.py
production/server.py
production/test_smoke.py
production/Dockerfile
production/requirements.txt
production/Dockerfile.slim
production/deploy.py
production/__pycache__/server.cpython-310.pyc
production/__pycache__/test_stress_full.cpython-310.pyc
production/__pycache__/test_stress.cpython-310.pyc
production/__pycache__/deploy.cpython-310.pyc
production/__pycache__/worker.cpython-310.pyc
production/__pycache__/test_smoke.cpython-310.pyc


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Inspecting production script files**


I confirmed the project isn’t a git repo at the top-level path, so I’ll edit directly in `production/` and validate by execution. Next I’m reading the current smoke/stress/deploy scripts to wire dynamic WS port discovery and stabilize the WS benchmark loop.


sed -n '1,260p' production/test_smoke.py


sed -n '1,320p' production/test_stress.py


Output:
Chunk ID: ce7616
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 3031
Output:
#!/usr/bin/env python3
"""
Smoke test for Hindi ASR production server.
Tests: health, HTTP batch, HTTP SSE stream, WebSocket streaming.
Usage:
  python test_smoke.py --url http://worker-ip:8000
  python test_smoke.py --endpoint hindi-asr-prod --route-cost 10
"""
import os, sys, json, time, struct, asyncio, argparse, math, base64
from pathlib import Path
import requests
from urllib.parse import urlparse
import aiohttp

def generate_test_wav(duration_s=3.0, freq_hz=440, sample_rate=16000):
    """Generate sine wave as WAV bytes + raw PCM"""
    n = int(sample_rate * duration_s)
    pcm = struct.pack(f'<{n}h', *[min(32767, max(-32768, int(16000*math.sin(2*math.pi*freq_hz*i/sample_rate)))) for i in range(n)])
    hdr = struct.pack('<4sI4s4sIHHIIHH4sI', b'RIFF', 36+len(pcm), b'WAVE', b'fmt ', 16, 1, 1, sample_rate, sample_rate*2, 2, 16, b'data', len(pcm))
    return pcm, hdr + pcm

VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"

def _env_or_dotenv(name: str):
    if os.environ.get(name):
        return os.environ.get(name)
    env_path = Path(__file__).resolve().parent.parent / ".env"
    if env_path.exists():
        for line in env_path.read_text().splitlines():
            if line.startswith(name + "="):
                return line.split("=", 1)[1].strip()
    return None

def get_endpoint_info(endpoint_name: str):
    account_key = _env_or_dotenv("VAST_KEY")
    if not account_key:
        raise RuntimeError("VAST_KEY not set; pass --endpoint-key")
    r = requests.get(
        f"{VAST_API}/endptjobs/",
        headers={"Accept": "application/json", "Authorization": f"Bearer {account_key}"},
        timeout=30,
    )
    data = r.json()
    results = data.get("results", []) if isinstance(data, dict) else data
    for ep in results or []:
        if ep.get("endpoint_name") == endpoint_name:
            return ep
    raise RuntimeError(f"endpoint '{endpoint_name}' not found")

def get_endpoint_key(endpoint_name: str, endpoint_key: str | None):
    if endpoint_key:
        return endpoint_key
    return get_endpoint_info(endpoint_name).get("api_key")

def get_endpoint_id(endpoint_name: str):
    return get_endpoint_info(endpoint_name).get("id")

def get_route(endpoint_name: str, endpoint_key: str, cost: float):
    payload = {"endpoint": endpoint_name, "api_key": endpoint_key, "cost": cost}
    r = requests.post(f"{VAST_RUN}/route/", json=payload, timeout=30)
    data = r.json()
    if "url" not in data:
        raise RuntimeError(f"route failed: {data}")
    return data

def get_endpoint_workers(endpoint_name: str, endpoint_key: str):
    eid = get_endpoint_id(endpoint_name)
    r = requests.post(
        f"{VAST_RUN}/get_endpoint_workers/",
        json={"id": eid, "api_key": endpoint_key},
        timeout=30,
        headers={"Content-Type": "application/json"},
    )
    data = r.json()
    if isinstance(data, dict) and "workers" in data:
        return data["workers"]
    return data if isinstance(data, list) else []

def wrap_auth(route: dict, payload: dict):
    auth = {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"] if k in route}
    return {"auth_data": auth, "payload": payload}

async def test_health(url, endpoint_name=None, endpoint_key=None):
    print("\n=== Health ===")
    if endpoint_name:
        try:
            workers = await asyncio.to_thread(get_endpoint_workers, endpoint_name, endpoint_key)
            ok = len(workers) > 0
            print(f"  [{'PASS' if ok else 'FAIL'}] workers={len(workers)}")
            return ok
        except Exception as e:
            print(f"  [FAIL] {e}")
            return False

    async with aiohttp.ClientSession() as s:
        t0 = time.perf_counter()
        async with s.get(f"{url}/health") as r:
            dt = (time.perf_counter()-t0)*1000
            d = await r.json()
            ok = r.status == 200
            print(f"  [{'PASS' if ok else 'FAIL'}] {r.status} {dt:.0f}ms {json.dumps(d)}")
            return ok

async def test_http_batch(url, endpoint_name=None, endpoint_key=None, route_cost=10.0):
    print("\n=== HTTP Batch ===")
    _, wav = generate_test_wav(3.0)
    if endpoint_name:
        audio_b64 = base64.b64encode(wav).decode("ascii")
        async with aiohttp.ClientSession() as s:
            t0 = time.perf_counter()
            try:
                route = await asyncio.to_thread(get_route, endpoint_name, endpoint_key, route_cost)
                worker_url = route["url"].rstrip("/")
                payload = {"audio_base64": audio_b64, "language": "hi"}
                req = wrap_auth(route, payload)
                async with s.post(f"{worker_url}/v1/audio/transcriptions", json=req, timeout=aiohttp.ClientTimeout(total=120)) as r:
                    dt = (time.perf_counter()-t0)*1000
                    if r.status == 200:
                        d = await r.json()
                        print(f"  [PASS] {dt:.0f}ms text='{d.get('text','')[:80]}' proc={d.get('processing_time',0)}s")
                        return True
                    # Fallback for older images
                    if r.status in (404, 422):
                        route = await asyncio.to_thread(get_route, endpoint_name, endpoint_key, route_cost)
                        worker_url = route["url"].rstrip("/")
                        req = wrap_auth(route, payload)
                        async with s.post(f"{worker_url}/v1/audio/transcriptions/json", json=req, timeout=aiohttp.ClientTimeout(total=120)) as r2:
                            dt = (time.perf_counter()-t0)*1000
                            if r2.status == 200:
                                d = await r2.json()
                                print(f"  [PASS] {dt:.0f}ms text='{d.get('text','')[:80]}' proc={d.get('processing_time',0)}s (json)")
                                return True
                            print(f"  [FAIL] {r2.status} {(await r2.text())[:200]}")
                            return False
                    print(f"  [FAIL] {r.status} {(await r.text())[:200]}")
                    return False
            except Exception as e:
                print(f"  [FAIL] {e}"); return False

    async with aiohttp.ClientSession() as s:
        form = aiohttp.FormData()
        form.add_field('file', wav, filename='test.wav', content_type='audio/wav')
        form.add_field('language', 'hi')
        t0 = time.perf_counter()
        try:
            async with s.post(f"{url}/v1/audio/transcriptions", data=form, timeout=aiohttp.ClientTimeout(total=60)) as r:
                dt = (time.perf_counter()-t0)*1000
                if r.status == 200:
                    d = await r.json()
                    print(f"  [PASS] {dt:.0f}ms text='{d.get('text','')[:80]}' proc={d.get('processing_time',0)}s")
                    return True
                print(f"  [FAIL] {r.status} {(await r.text())[:200]}")
                return False
        except Exception as e:
            print(f"  [FAIL] {e}"); return False

async def test_http_stream(url, endpoint_name=None, endpoint_key=None, route_cost=10.0):
    print("\n=== HTTP Stream (SSE) ===")
    _, wav = generate_test_wav(3.0)
    if endpoint_name:
        audio_b64 = base64.b64encode(wav).decode("ascii")
        async with aiohttp.ClientSession() as s:
            t0 = time.perf_counter()
            try:
                route = await asyncio.to_thread(get_route, endpoint_name, endpoint_key, route_cost)
                worker_url = route["url"].rstrip("/")
                payload = {"audio_base64": audio_b64, "language": "hi", "chunk_duration_ms": 160}
                req = wrap_auth(route, payload)
                async with s.post(f"{worker_url}/v1/audio/transcriptions/stream", json=req, timeout=aiohttp.ClientTimeout(total=120)) as r:
                    if r.status != 200:
                        print(f"  [FAIL] {r.status}")
                        return False
                    events = []
                    async for line in r.content:
                        l = line.decode().strip()
                        if l.startswith("data: "):
                            d = json.loads(l[6:])
                            events.append(d)
                            if d.get("done"):
                                break
                    dt = (time.perf_counter()-t0)*1000
                    print(f"  [PASS] {len(events)} events {dt:.0f}ms")
                    if events:
                        print(f"  Final: '{events[-1].get('text','')[:80]}'")
                    return True
            except Exception as e:
                print(f"  [FAIL] {e}"); return False

    async with aiohttp.ClientSession() as s:
        form = aiohttp.FormData()
        form.add_field('file', wav, filename='test.wav', content_type='audio/wav')
        t0 = time.perf_counter()
        try:
            async with s.post(f"{url}/v1/audio/transcriptions/stream", data=form, timeout=aiohttp.ClientTimeout(total=60)) as r:
                if r.status != 200:
                    print(f"  [FAIL] {r.status}"); return False
                events = []
                async for line in r.content:
                    l = line.decode().strip()
                    if l.startswith("data: "):
                        d = json.loads(l[6:])
                        events.append(d)
                        if d.get("done"): break
                dt = (time.perf_counter()-t0)*1000
                print(f"  [PASS] {len(events)} events {dt:.0f}ms")
                if events: print(f"  Final: '{events[-1].get('text','')[:80]}'")
                return True
        except Exception as e:
            print(f"  [FAIL] {e}"); return False

async def test_websocket(url, endpoint_name=None, endpoint_key=None, route_cost=10.0, ws_port=None):
    print("\n=== WebSocket ===")
    if endpoint_name:
        if not ws_port:
            print("  [SKIP] ws_port not provided for endpoint mode")
            return True
        try:
            route = await asyncio.to_thread(get_route, endpoint_name, endpoint_key, route_cost)
            parsed = urlparse(route["url"])
            scheme = "wss" if parsed.scheme == "https" else "ws"
            host = parsed.hostname
            ws_url = f"{scheme}://{host}:{ws_port}/ws"
        except Exception as e:
            print(f"  [FAIL] {e}")
            return False
    else:
        ws_url = url.replace("http://","ws://").replace("https://","wss://") + "/ws"
    pcm, _ = generate_test_wav(3.0)
    try:
        async with aiohttp.ClientSession() as s:
            t0 = time.perf_counter()
            async with s.ws_connect(ws_url, timeout=30) as ws:
                ct = (time.perf_counter()-t0)*1000
                print(f"  Connected {ct:.0f}ms")
                ready = await asyncio.wait_for(ws.receive(), timeout=10)
                if ready.type == aiohttp.WSMsgType.TEXT:
                    print(f"  Ready: {ready.data}")
                chunk_size = int(0.160*16000*2)
                ft = None; msgs = []
                async def send():
                    for i in range(0, len(pcm), chunk_size):
                        await ws.send_bytes(pcm[i:i+chunk_size])
                        await asyncio.sleep(0.160)
                    await ws.send_str("END")
                async def recv():
                    nonlocal ft
                    while True:
                        m = await asyncio.wait_for(ws.receive(), timeout=15)
                        if m.type == aiohttp.WSMsgType.TEXT:
                            if m.data == "END": break
                            d = json.loads(m.data)
                            if ft is None and d.get("text"): ft = time.perf_counter()
                            msgs.append(d)
                        elif m.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING): break
                await asyncio.gather(send(), recv())
                dt = (time.perf_counter()-t0)*1000
                ft_ms = (ft-t0)*1000 if ft else -1
                print(f"  [PASS] {len(msgs)} msgs {dt:.0f}ms first_token={ft_ms:.0f}ms")
                if msgs:


Output:
Chunk ID: cabbd7
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 3341
Output:
#!/usr/bin/env python3
"""
Production Stress Test for Hindi ASR - Simulates Real Customer Usage

Captures ALL metrics needed for scaling law decisions:
- Connection time (p50/p95/p99)
- First token latency (p50/p95/p99)
- End-to-end latency
- WER/CER (if reference text available)
- Throughput (streams * realtime)
- GPU utilization (via /metrics endpoint)
- Error rate
- Drift (latency accumulation over time)

Usage:
    python test_stress.py --url http://ip:8000 --mode ws --levels 1,10,50,100
    python test_stress.py --url http://ip:8000 --mode http --levels 1,5,10,20
    python test_stress.py --url http://ip:8000 --mode sse --levels 1,10,50
    python test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10
"""
import os, sys, json, time, math, struct, asyncio, argparse, csv, base64
from pathlib import Path
import requests
from urllib.parse import urlparse
from collections import defaultdict
from datetime import datetime
import aiohttp

def generate_test_audio(duration_s=10.0, freq_hz=440, sample_rate=16000):
    """Generate test audio: sine wave as raw PCM + WAV"""
    n = int(sample_rate * duration_s)
    pcm = struct.pack(f'<{n}h', *[
        min(32767, max(-32768, int(16000 * math.sin(2 * math.pi * freq_hz * i / sample_rate))))
        for i in range(n)
    ])
    hdr = struct.pack(
        '<4sI4s4sIHHIIHH4sI',
        b'RIFF', 36 + len(pcm), b'WAVE', b'fmt ', 16,
        1, 1, sample_rate, sample_rate * 2, 2, 16,
        b'data', len(pcm),
    )
    return pcm, hdr + pcm

VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"

def _env_or_dotenv(name: str):
    if os.environ.get(name):
        return os.environ.get(name)
    env_path = Path(__file__).resolve().parent.parent / ".env"
    if env_path.exists():
        for line in env_path.read_text().splitlines():
            if line.startswith(name + "="):
                return line.split("=", 1)[1].strip()
    return None

def get_endpoint_info(endpoint_name: str):
    account_key = _env_or_dotenv("VAST_KEY")
    if not account_key:
        raise RuntimeError("VAST_KEY not set; pass --endpoint-key")
    r = requests.get(
        f"{VAST_API}/endptjobs/",
        headers={"Accept": "application/json", "Authorization": f"Bearer {account_key}"},
        timeout=30,
    )
    data = r.json()
    results = data.get("results", []) if isinstance(data, dict) else data
    for ep in results or []:
        if ep.get("endpoint_name") == endpoint_name:
            return ep
    raise RuntimeError(f"endpoint '{endpoint_name}' not found")

def get_endpoint_key(endpoint_name: str, endpoint_key: str | None):
    if endpoint_key:
        return endpoint_key
    return get_endpoint_info(endpoint_name).get("api_key")

def get_endpoint_id(endpoint_name: str):
    return get_endpoint_info(endpoint_name).get("id")

def wrap_auth(route: dict, payload: dict):
    auth = {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"] if k in route}
    return {"auth_data": auth, "payload": payload}

def pct(arr, p):
    if not arr: return 0
    s = sorted(arr)
    return s[min(int(len(s) * p / 100), len(s) - 1)]

class StressMetrics:
    """Collects and reports all metrics for a stress test level"""
    def __init__(self, concurrency, mode, audio_duration_s):
        self.concurrency = concurrency
        self.mode = mode
        self.audio_duration_s = audio_duration_s
        self.start_time = None
        self.end_time = None
        # Per-stream metrics
        self.connect_times_ms = []
        self.first_token_times_ms = []
        self.total_times_ms = []
        self.messages_per_stream = []
        self.errors = []
        self.texts = []
        # Server metrics snapshots
        self.server_metrics_before = None
        self.server_metrics_after = None

    def summary(self):
        ok = self.concurrency - len(self.errors)
        wall_time = (self.end_time - self.start_time) if self.end_time else 0
        throughput_x = 0.0
        if wall_time > 0:
            throughput_x = (ok * self.audio_duration_s) / wall_time
        return {
            "concurrency": self.concurrency,
            "mode": self.mode,
            "ok": ok,
            "errors": len(self.errors),
            "wall_time_s": round(wall_time, 2),
            "connect_ms_p50": round(pct(self.connect_times_ms, 50), 1),
            "connect_ms_p95": round(pct(self.connect_times_ms, 95), 1),
            "connect_ms_p99": round(pct(self.connect_times_ms, 99), 1),
            "first_token_ms_p50": round(pct(self.first_token_times_ms, 50), 1),
            "first_token_ms_p95": round(pct(self.first_token_times_ms, 95), 1),
            "first_token_ms_p99": round(pct(self.first_token_times_ms, 99), 1),
            "total_ms_p50": round(pct(self.total_times_ms, 50), 1),
            "total_ms_p95": round(pct(self.total_times_ms, 95), 1),
            "total_ms_p99": round(pct(self.total_times_ms, 99), 1),
            "msgs_per_stream_avg": round(sum(self.messages_per_stream) / max(len(self.messages_per_stream), 1), 1),
            "throughput_x": round(throughput_x, 2),
        }

async def fetch_server_metrics(url=None, endpoint_name=None, endpoint_key=None, route_cost=10.0):
    """Fetch /metrics either directly or through serverless PyWorker."""
    if endpoint_name:
        try:
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    route = await r.json()
            if "url" not in route:
                return {}
            worker_url = route["url"].rstrip("/")
            req = wrap_auth(route, {})
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{worker_url}/metrics",
                    json=req,
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    if r.status == 200:
                        return await r.json()
        except Exception:
            return {}
        return {}

    if not url:
        return {}
    try:
        async with aiohttp.ClientSession() as s:
            async with s.get(f"{url}/metrics", timeout=aiohttp.ClientTimeout(total=5)) as r:
                if r.status == 200:
                    return await r.json()
    except Exception:
        pass
    return {}

async def fetch_endpoint_workers(endpoint_name, endpoint_key):
    """Fetch worker list for endpoint from Vast run API"""
    try:
        eid = get_endpoint_id(endpoint_name)
        async with aiohttp.ClientSession() as s:
            async with s.post(
                f"{VAST_RUN}/get_endpoint_workers/",
                json={"id": eid, "api_key": endpoint_key},
                timeout=aiohttp.ClientTimeout(total=10),
            ) as r:
                data = await r.json()
                if isinstance(data, dict) and "workers" in data:
                    return data["workers"]
                return data if isinstance(data, list) else []
    except Exception:
        return []

async def run_ws_stream(url, pcm_bytes, stream_idx, metrics, duration_s,
                        endpoint_name=None, endpoint_key=None, route_cost=10.0, ws_port=None):
    """Run a single WebSocket stream at real-time pace"""
    if endpoint_name:
        if not ws_port:
            metrics.errors.append({"stream": stream_idx, "error": "ws_port required for endpoint mode"})
            return
        try:
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    route = await r.json()
            if "url" not in route:
                metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
                return
            parsed = urlparse(route["url"])
            scheme = "wss" if parsed.scheme == "https" else "ws"
            ws_url = f"{scheme}://{parsed.hostname}:{ws_port}/ws"
        except Exception as e:
            metrics.errors.append({"stream": stream_idx, "error": str(e)})
            return
    else:
        ws_url = url.replace("http://", "ws://").replace("https://", "wss://") + "/ws"
    chunk_size = int(0.160 * 16000 * 2)  # 160ms chunks
    t_start = time.perf_counter()

    try:
        async with aiohttp.ClientSession() as s:
            async with s.ws_connect(ws_url, timeout=30) as ws:
                t_connected = time.perf_counter()
                metrics.connect_times_ms.append((t_connected - t_start) * 1000)

                # Wait for ready
                ready = await asyncio.wait_for(ws.receive(), timeout=10)

                first_token = None
                msgs = []

                async def send():
                    for i in range(0, len(pcm_bytes), chunk_size):
                        await ws.send_bytes(pcm_bytes[i:i + chunk_size])
                        await asyncio.sleep(0.160)  # Real-time pacing
                    await ws.send_str("END")

                async def recv():
                    nonlocal first_token
                    while True:
                        try:
                            m = await asyncio.wait_for(ws.receive(), timeout=30)
                            if m.type == aiohttp.WSMsgType.TEXT:
                                if m.data == "END":
                                    break
                                d = json.loads(m.data)
                                if first_token is None and d.get("text"):
                                    first_token = time.perf_counter()
                                msgs.append(d)
                            elif m.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING, aiohttp.WSMsgType.CLOSED):
                                break
                        except asyncio.TimeoutError:
                            break

                await asyncio.gather(send(), recv())

                t_done = time.perf_counter()
                if first_token:
                    metrics.first_token_times_ms.append((first_token - t_start) * 1000)
                metrics.total_times_ms.append((t_done - t_start) * 1000)
                metrics.messages_per_stream.append(len(msgs))
                if msgs:
                    last_text = ""
                    for m in reversed(msgs):
                        if m.get("text"):
                            last_text = m["text"]
                            break
                    metrics.texts.append(last_text)

    except Exception as e:
        metrics.errors.append({"stream": stream_idx, "error": str(e)})

async def run_http_batch(url, wav_bytes, stream_idx, metrics,
                         endpoint_name=None, endpoint_key=None, route_cost=10.0):
    """Run a single HTTP batch transcription"""
    t_start = time.perf_counter()
    try:
        async with aiohttp.ClientSession() as s:
            if endpoint_name:
                audio_b64 = base64.b64encode(wav_bytes).decode("ascii")
                # Get route
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as rr:
                    route = await rr.json()
                if "url" not in route:
                    metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
                    return
                worker_url = route["url"].rstrip("/")
                payload = {"audio_base64": audio_b64, "language": "hi"}
                req = wrap_auth(route, payload)
                async with s.post(
                    f"{worker_url}/v1/audio/transcriptions",
                    json=req,
                    timeout=aiohttp.ClientTimeout(total=120),
                ) as r:
                    t_done = time.perf_counter()
                    metrics.connect_times_ms.append(0)
                    if r.status == 200:
                        d = await r.json()
                        metrics.total_times_ms.append((t_done - t_start) * 1000)
                        metrics.first_token_times_ms.append((t_done - t_start) * 1000)
                        metrics.messages_per_stream.append(1)
                        metrics.texts.append(d.get("text", ""))
                    elif r.status in (404, 422):
                        # Fallback for older images
                        async with s.post(
                            f"{VAST_RUN}/route/",
                            json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                            timeout=aiohttp.ClientTimeout(total=10),
                        ) as rr2:
                            route2 = await rr2.json()
                        if "url" not in route2:
                            metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route2}"})


Output:
Chunk ID: a5cde7
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1509
Output:
#!/usr/bin/env python3
"""
Vast.ai Serverless Deployment for Hindi ASR.
Usage: python deploy.py {create|status|workers|logs|route|destroy}
"""
import os, sys, json, argparse, requests
sys.path.insert(0, os.path.dirname(__file__))
try:
    from dotenv import load_dotenv
    load_dotenv(os.path.join(os.path.dirname(__file__), "..", ".env"))
except ImportError:
    pass

VAST_API_KEY = os.environ.get("VAST_KEY", "")
VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"
ENDPOINT_NAME = os.environ.get("VAST_ENDPOINT_NAME", "hindi-asr-prod")
DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v10-production")
TEMPLATE_HASH = os.environ.get("VAST_TEMPLATE_HASH", "")
LAUNCH_ARGS = os.environ.get("VAST_LAUNCH_ARGS", "-p 8000:8000")

ENDPOINT_CONFIG = {
    "endpoint_name": ENDPOINT_NAME,
    "min_load": 40,
    "target_util": 0.65,
    "cold_mult": 2.5,
    "cold_workers": 2,
    "max_workers": 16,
}

SEARCH_PARAMS = {
    "verified": {"eq": True}, "rentable": {"eq": True},
    "rented": {"eq": False}, "gpu_ram": {"gte": 40},
    "num_gpus": {"eq": 1}, "reliability": {"gte": 0.95},
    "inet_down": {"gte": 200},
}

def headers():
    return {"Accept": "application/json", "Content-Type": "application/json",
            "Authorization": f"Bearer {VAST_API_KEY}"}

def get_endpoints():
    try:
        r = requests.get(f"{VAST_API}/endptjobs/", headers=headers())
        data = r.json()
        if isinstance(data, dict):
            return data.get("results", [])
        if isinstance(data, list):
            return data
    except Exception:
        pass
    return []

def get_endpoint_info():
    for ep in get_endpoints():
        if ep.get("endpoint_name") == ENDPOINT_NAME:
            return ep
    return None

def create_endpoint():
    print(f"Creating endpoint: {ENDPOINT_NAME}")
    r = requests.post(f"{VAST_API}/endptjobs/", headers=headers(), json=ENDPOINT_CONFIG)
    d = r.json()
    if d.get("success"):
        print(f"  Created: id={d['result']}")
        return d["result"]
    print(f"  Error: {d}")
    return get_endpoint_id()

def get_endpoint_id():
    ep = get_endpoint_info()
    return ep.get("id") if ep else None

def get_endpoint_key():
    ep = get_endpoint_info()
    return ep.get("api_key") if ep else None

def get_template_hash():
    if TEMPLATE_HASH:
        return TEMPLATE_HASH
    # Try to infer from existing workergroup
    try:
        r = requests.get(f"{VAST_API}/workergroups/", headers=headers())
        data = r.json()
        results = data.get("results", []) if isinstance(data, dict) else data
        for wg in results or []:
            if wg.get("endpoint_name") == ENDPOINT_NAME or wg.get("endpoint_id") == get_endpoint_id():
                return wg.get("template_hash") or ""
    except Exception:
        pass
    return ""

def create_workergroup(eid=None):
    eid = eid or get_endpoint_id()
    if not eid: print("No endpoint found"); return
    print(f"Creating workergroup for endpoint {eid}")
    payload = {
        "endpoint_id": eid,
        "gpu_ram": 40,
    }
    template_hash = get_template_hash()
    if template_hash:
        payload["template_hash"] = template_hash
        payload["search_params"] = ""
    else:
        payload["search_params"] = json.dumps(SEARCH_PARAMS)
    if LAUNCH_ARGS:
        payload["launch_args"] = LAUNCH_ARGS
    r = requests.post(f"{VAST_API}/workergroups/", headers=headers(), json=payload)
    d = r.json()
    print(f"  {'Created: id=' + str(d['result']) if d.get('success') else 'Error: ' + str(d)}")

def get_workers():
    eid = get_endpoint_id()
    ekey = get_endpoint_key()
    if not eid or not ekey:
        return []
    r = requests.post(f"{VAST_RUN}/get_endpoint_workers/",
        json={"id": eid, "api_key": ekey},
        headers={"Content-Type": "application/json"})
    try:
        d = r.json()
        return d if isinstance(d, list) else d.get("workers", [])
    except: return []

def get_route(cost=10.0):
    ekey = get_endpoint_key()
    if not ekey:
        return {"error": "endpoint api_key not found"}
    r = requests.post(f"{VAST_RUN}/route/",
        json={"endpoint": ENDPOINT_NAME, "api_key": ekey, "cost": cost},
        headers={"Content-Type": "application/json"})
    return r.json()

def show_status():
    eid = get_endpoint_id()
    print(f"Endpoint: {ENDPOINT_NAME} (id={eid})")
    if not eid: print("  NOT DEPLOYED"); return
    workers = get_workers()
    print(f"  Workers: {len(workers)}")
    for w in workers:
        print(f"    id={w.get('id')} status={w.get('status')} load={w.get('cur_load',0):.1f}")
    route = get_route()
    print(f"  Route: {route.get('url', route)}")

def destroy():
    eid = get_endpoint_id()
    if not eid: print("Nothing to destroy"); return
    print(f"Destroying endpoint {eid}")
    requests.put(f"{VAST_API}/endptjobs/{eid}/", headers=headers(), json={"status": "stopped"})
    requests.delete(f"{VAST_API}/endptjobs/{eid}/", headers=headers())
    print("  Done")

def main():
    if not VAST_API_KEY: print("ERROR: VAST_KEY not set"); sys.exit(1)
    p = argparse.ArgumentParser()
    p.add_argument("action", choices=["create","status","workers","logs","route","destroy"])
    a = p.parse_args()
    if a.action == "create":
        eid = create_endpoint()
        if eid: create_workergroup(eid)
    elif a.action == "status": show_status()
    elif a.action == "workers": print(json.dumps(get_workers(), indent=2))
    elif a.action == "logs":
        eid = get_endpoint_id()
        if eid:
            ekey = get_endpoint_key()
            r = requests.post(f"{VAST_RUN}/get_endpoint_logs/",
                json={"id": eid, "api_key": ekey, "tail": 2000},
                headers={"Content-Type": "application/json"})
            for k,v in r.json().items():
                if v: print(f"\n=== {k} ===\n{v}")
    elif a.action == "route": print(json.dumps(get_route(), indent=2))
    elif a.action == "destroy": destroy()

if __name__ == "__main__": main()


Summary: auto


**Planning full file retrieval**


nl -ba production/test_smoke.py | sed -n '240,520p'


nl -ba production/test_stress.py | sed -n '300,760p'


nl -ba production/server.py | sed -n '1,360p'


Output:
Chunk ID: 56439c
Wall time: 0.0518 seconds
Process exited with code 0
Original token count: 833
Output:
   240	                ft = None; msgs = []
   241	                async def send():
   242	                    for i in range(0, len(pcm), chunk_size):
   243	                        await ws.send_bytes(pcm[i:i+chunk_size])
   244	                        await asyncio.sleep(0.160)
   245	                    await ws.send_str("END")
   246	                async def recv():
   247	                    nonlocal ft
   248	                    while True:
   249	                        m = await asyncio.wait_for(ws.receive(), timeout=15)
   250	                        if m.type == aiohttp.WSMsgType.TEXT:
   251	                            if m.data == "END": break
   252	                            d = json.loads(m.data)
   253	                            if ft is None and d.get("text"): ft = time.perf_counter()
   254	                            msgs.append(d)
   255	                        elif m.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING): break
   256	                await asyncio.gather(send(), recv())
   257	                dt = (time.perf_counter()-t0)*1000
   258	                ft_ms = (ft-t0)*1000 if ft else -1
   259	                print(f"  [PASS] {len(msgs)} msgs {dt:.0f}ms first_token={ft_ms:.0f}ms")
   260	                if msgs:
   261	                    for m in reversed(msgs):
   262	                        if m.get("text"): print(f"  Last: '{m['text'][:80]}'"); break
   263	                return True
   264	    except Exception as e:
   265	        print(f"  [FAIL] {e}"); return False
   266
   267	async def main():
   268	    p = argparse.ArgumentParser()
   269	    p.add_argument("--url", default=None, help="Direct server URL (http://ip:port)")
   270	    p.add_argument("--endpoint", default=None, help="Vast endpoint name (uses /route + auth)")
   271	    p.add_argument("--endpoint-key", default=None, help="Endpoint API key (optional if VAST_KEY set)")
   272	    p.add_argument("--route-cost", type=float, default=10.0)
   273	    p.add_argument("--ws-port", type=int, default=None, help="Direct WS port on worker (if exposed)")
   274	    p.add_argument("--http-only", action="store_true")
   275	    p.add_argument("--ws-only", action="store_true")
   276	    a = p.parse_args()
   277	    if not a.url and not a.endpoint:
   278	        print("ERROR: provide --url or --endpoint")
   279	        sys.exit(2)
   280	    url = a.url.rstrip("/") if a.url else ""
   281	    endpoint_key = get_endpoint_key(a.endpoint, a.endpoint_key) if a.endpoint else None
   282	    results = {"health": await test_health(url, a.endpoint, endpoint_key)}
   283	    if not a.ws_only:
   284	        results["http_batch"] = await test_http_batch(url, a.endpoint, endpoint_key, a.route_cost)
   285	        results["http_stream"] = await test_http_stream(url, a.endpoint, endpoint_key, a.route_cost)
   286	    if not a.http_only:
   287	        results["websocket"] = await test_websocket(url, a.endpoint, endpoint_key, a.route_cost, a.ws_port)
   288	    print("\n=== SUMMARY ===")
   289	    ok = all(results.values())
   290	    for k,v in results.items(): print(f"  {k:20s} [{'PASS' if v else 'FAIL'}]")
   291	    print(f"\n{'All passed!' if ok else 'FAILURES detected'}")
   292	    sys.exit(0 if ok else 1)
   293
   294	if __name__ == "__main__": asyncio.run(main())


Output:
Chunk ID: ebde09
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 4780
Output:
   300	                    json=req,
   301	                    timeout=aiohttp.ClientTimeout(total=120),
   302	                ) as r:
   303	                    t_done = time.perf_counter()
   304	                    metrics.connect_times_ms.append(0)
   305	                    if r.status == 200:
   306	                        d = await r.json()
   307	                        metrics.total_times_ms.append((t_done - t_start) * 1000)
   308	                        metrics.first_token_times_ms.append((t_done - t_start) * 1000)
   309	                        metrics.messages_per_stream.append(1)
   310	                        metrics.texts.append(d.get("text", ""))
   311	                    elif r.status in (404, 422):
   312	                        # Fallback for older images
   313	                        async with s.post(
   314	                            f"{VAST_RUN}/route/",
   315	                            json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
   316	                            timeout=aiohttp.ClientTimeout(total=10),
   317	                        ) as rr2:
   318	                            route2 = await rr2.json()
   319	                        if "url" not in route2:
   320	                            metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route2}"})
   321	                            return
   322	                        worker_url = route2["url"].rstrip("/")
   323	                        req = wrap_auth(route2, payload)
   324	                        async with s.post(
   325	                            f"{worker_url}/v1/audio/transcriptions/json",
   326	                            json=req,
   327	                            timeout=aiohttp.ClientTimeout(total=120),
   328	                        ) as r2:
   329	                            t_done = time.perf_counter()
   330	                            if r2.status == 200:
   331	                                d = await r2.json()
   332	                                metrics.total_times_ms.append((t_done - t_start) * 1000)
   333	                                metrics.first_token_times_ms.append((t_done - t_start) * 1000)
   334	                                metrics.messages_per_stream.append(1)
   335	                                metrics.texts.append(d.get("text", ""))
   336	                            else:
   337	                                metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r2.status}"})
   338	                    else:
   339	                        metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r.status}"})
   340	                return
   341
   342	            form = aiohttp.FormData()
   343	            form.add_field('file', wav_bytes, filename='test.wav', content_type='audio/wav')
   344	            form.add_field('language', 'hi')
   345
   346	            async with s.post(
   347	                f"{url}/v1/audio/transcriptions",
   348	                data=form,
   349	                timeout=aiohttp.ClientTimeout(total=120),
   350	            ) as r:
   351	                t_done = time.perf_counter()
   352	                metrics.connect_times_ms.append(0)  # N/A for HTTP
   353	                if r.status == 200:
   354	                    d = await r.json()
   355	                    metrics.total_times_ms.append((t_done - t_start) * 1000)
   356	                    metrics.first_token_times_ms.append((t_done - t_start) * 1000)
   357	                    metrics.messages_per_stream.append(1)
   358	                    metrics.texts.append(d.get("text", ""))
   359	                else:
   360	                    metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r.status}"})
   361	    except Exception as e:
   362	        metrics.errors.append({"stream": stream_idx, "error": str(e)})
   363
   364
   365	async def run_sse_stream(url, wav_bytes, stream_idx, metrics,
   366	                         endpoint_name=None, endpoint_key=None, route_cost=10.0):
   367	    """Run a single HTTP SSE stream"""
   368	    t_start = time.perf_counter()
   369	    try:
   370	        async with aiohttp.ClientSession() as s:
   371	            if endpoint_name:
   372	                audio_b64 = base64.b64encode(wav_bytes).decode("ascii")
   373	                async with s.post(
   374	                    f"{VAST_RUN}/route/",
   375	                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
   376	                    timeout=aiohttp.ClientTimeout(total=10),
   377	                ) as rr:
   378	                    route = await rr.json()
   379	                if "url" not in route:
   380	                    metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
   381	                    return
   382	                worker_url = route["url"].rstrip("/")
   383	                payload = {"audio_base64": audio_b64, "language": "hi", "chunk_duration_ms": 160}
   384	                req = wrap_auth(route, payload)
   385	                async with s.post(
   386	                    f"{worker_url}/v1/audio/transcriptions/stream",
   387	                    json=req,
   388	                    timeout=aiohttp.ClientTimeout(total=120),
   389	                ) as r:
   390	                    if r.status != 200:
   391	                        metrics.errors.append({"stream": stream_idx, "error": f"SSE {r.status}"})
   392	                        return
   393	                    first_token = None
   394	                    msgs = 0
   395	                    async for line in r.content:
   396	                        l = line.decode().strip()
   397	                        if l.startswith("data: "):
   398	                            d = json.loads(l[6:])
   399	                            if first_token is None and d.get("text"):
   400	                                first_token = time.perf_counter()
   401	                            msgs += 1
   402	                            if d.get("done"):
   403	                                break
   404	                    t_done = time.perf_counter()
   405	                    if first_token:
   406	                        metrics.first_token_times_ms.append((first_token - t_start) * 1000)
   407	                    metrics.total_times_ms.append((t_done - t_start) * 1000)
   408	                    metrics.messages_per_stream.append(msgs)
   409	                return
   410
   411	            form = aiohttp.FormData()
   412	            form.add_field('file', wav_bytes, filename='test.wav', content_type='audio/wav')
   413	            async with s.post(
   414	                f"{url}/v1/audio/transcriptions/stream",
   415	                data=form,
   416	                timeout=aiohttp.ClientTimeout(total=120),
   417	            ) as r:
   418	                if r.status != 200:
   419	                    metrics.errors.append({"stream": stream_idx, "error": f"SSE {r.status}"})
   420	                    return
   421	                first_token = None
   422	                msgs = 0
   423	                async for line in r.content:
   424	                    l = line.decode().strip()
   425	                    if l.startswith("data: "):
   426	                        d = json.loads(l[6:])
   427	                        if first_token is None and d.get("text"):
   428	                            first_token = time.perf_counter()
   429	                        msgs += 1
   430	                        if d.get("done"):
   431	                            break
   432	                t_done = time.perf_counter()
   433	                if first_token:
   434	                    metrics.first_token_times_ms.append((first_token - t_start) * 1000)
   435	                metrics.total_times_ms.append((t_done - t_start) * 1000)
   436	                metrics.messages_per_stream.append(msgs)
   437	    except Exception as e:
   438	        metrics.errors.append({"stream": stream_idx, "error": str(e)})
   439
   440
   441	async def run_stress_level(url, concurrency, mode, audio_duration_s=10.0,
   442	                           endpoint_name=None, endpoint_key=None, route_cost=10.0, ws_port=None):
   443	    """Run a single stress test level"""
   444	    print(f"\n{'='*60}")
   445	    print(f"  Stress Level: c={concurrency} mode={mode} audio={audio_duration_s}s")
   446	    print(f"{'='*60}")
   447
   448	    pcm, wav = generate_test_audio(duration_s=audio_duration_s)
   449	    metrics = StressMetrics(concurrency, mode, audio_duration_s)
   450
   451	    # Capture server metrics before
   452	    if endpoint_name:
   453	        metrics.server_metrics_before = await fetch_server_metrics(
   454	            endpoint_name=endpoint_name,
   455	            endpoint_key=endpoint_key,
   456	            route_cost=route_cost,
   457	        )
   458	    else:
   459	        metrics.server_metrics_before = await fetch_server_metrics(url=url)
   460
   461	    metrics.start_time = time.perf_counter()
   462
   463	    # Launch concurrent streams
   464	    tasks = []
   465	    for i in range(concurrency):
   466	        if mode == "ws":
   467	            tasks.append(run_ws_stream(url, pcm, i, metrics, audio_duration_s,
   468	                                       endpoint_name, endpoint_key, route_cost, ws_port))
   469	        elif mode == "http":
   470	            tasks.append(run_http_batch(url, wav, i, metrics, endpoint_name, endpoint_key, route_cost))
   471	        elif mode == "sse":
   472	            tasks.append(run_sse_stream(url, wav, i, metrics, endpoint_name, endpoint_key, route_cost))
   473
   474	    await asyncio.gather(*tasks)
   475	    metrics.end_time = time.perf_counter()
   476
   477	    # Brief pause then capture server metrics after
   478	    await asyncio.sleep(1)
   479	    if endpoint_name:
   480	        metrics.server_metrics_after = await fetch_server_metrics(
   481	            endpoint_name=endpoint_name,
   482	            endpoint_key=endpoint_key,
   483	            route_cost=route_cost,
   484	        )
   485	    else:
   486	        metrics.server_metrics_after = await fetch_server_metrics(url=url)
   487
   488	    # Print results
   489	    s = metrics.summary()
   490	    print(f"\n  Results:")
   491	    print(f"    OK/Total: {s['ok']}/{s['concurrency']}  Errors: {s['errors']}")
   492	    print(f"    Wall time: {s['wall_time_s']}s")
   493	    print(f"    Connect:     p50={s['connect_ms_p50']}ms  p95={s['connect_ms_p95']}ms  p99={s['connect_ms_p99']}ms")
   494	    print(f"    First token: p50={s['first_token_ms_p50']}ms  p95={s['first_token_ms_p95']}ms  p99={s['first_token_ms_p99']}ms")
   495	    print(f"    Total:       p50={s['total_ms_p50']}ms  p95={s['total_ms_p95']}ms  p99={s['total_ms_p99']}ms")
   496	    print(f"    Msgs/stream: {s['msgs_per_stream_avg']}")
   497	    print(f"    Throughput:  {s['throughput_x']}x realtime")
   498
   499	    if metrics.server_metrics_after:
   500	        sm = metrics.server_metrics_after
   501	        if endpoint_name:
   502	            if isinstance(sm, dict):
   503	                inf = sm.get("inference", {})
   504	                print(f"    Server GPU:  p50={inf.get('gpu_ms_p50',0)}ms p99={inf.get('gpu_ms_p99',0)}ms")
   505	                print(f"    Server VRAM: {sm.get('vram_used_gb',0)}GB/{sm.get('vram_total_gb',0)}GB")
   506	                http = sm.get("http", {})
   507	                print(f"    HTTP:        total={http.get('requests_total',0)} ok={http.get('requests_ok',0)} err={http.get('requests_err',0)}")
   508	            elif isinstance(sm, list):
   509	                for w in sm:
   510	                    print(f"    Worker {w.get('id')} status={w.get('status')} cur_load={w.get('cur_load',0):.1f} cur_perf={w.get('cur_perf',0):.1f}")
   511	        else:
   512	            inf = sm.get("inference", {})
   513	            print(f"    Server GPU:  p50={inf.get('gpu_ms_p50',0)}ms p99={inf.get('gpu_ms_p99',0)}ms")
   514	            print(f"    Server VRAM: {sm.get('vram_used_gb',0)}GB/{sm.get('vram_total_gb',0)}GB")
   515
   516	    if metrics.errors:
   517	        print(f"    Errors: {metrics.errors[:5]}")
   518
   519	    return metrics
   520
   521
   522	async def main():
   523	    p = argparse.ArgumentParser(description="ASR Production Stress Test")
   524	    p.add_argument("--url", default=None, help="Server URL (http://ip:port)")
   525	    p.add_argument("--endpoint", default=None, help="Vast endpoint name (uses /route + auth)")
   526	    p.add_argument("--endpoint-key", default=None, help="Endpoint API key (optional if VAST_KEY set)")
   527	    p.add_argument("--route-cost", type=float, default=10.0)
   528	    p.add_argument("--ws-port", type=int, default=None, help="Direct WS port on worker (if exposed)")
   529	    p.add_argument("--mode", default="ws", choices=["ws", "http", "sse", "both"])
   530	    p.add_argument("--levels", default="1,5,10,25,50,75,100", help="Comma-separated concurrency levels")
   531	    p.add_argument("--audio-duration", type=float, default=10.0, help="Audio duration per stream (seconds)")
   532	    p.add_argument("--output", default=None, help="Output CSV file for results")
   533	    p.add_argument("--cooldown", type=float, default=5.0, help="Seconds between levels")
   534	    a = p.parse_args()
   535
   536	    if not a.url and not a.endpoint:
   537	        print("ERROR: provide --url or --endpoint")
   538	        sys.exit(2)
   539	    url = a.url.rstrip("/") if a.url else ""
   540	    levels = [int(x) for x in a.levels.split(",")]
   541	    modes = [a.mode] if a.mode != "both" else ["ws", "http"]
   542
   543	    endpoint_key = get_endpoint_key(a.endpoint, a.endpoint_key) if a.endpoint else None
   544
   545	    print(f"=== Hindi ASR Stress Test ===")
   546	    print(f"URL: {url if url else 'via endpoint'}")
   547	    if a.endpoint:
   548	        print(f"Endpoint: {a.endpoint}")
   549	    print(f"Modes: {modes}")
   550	    print(f"Levels: {levels}")
   551	    print(f"Audio: {a.audio_duration}s per stream")
   552
   553	    # Verify server is up
   554	    if a.endpoint:
   555	        workers = await fetch_endpoint_workers(a.endpoint, endpoint_key)
   556	        if not workers:
   557	            print("Serverless endpoint has no active workers")
   558	            sys.exit(1)
   559	        print(f"Endpoint ready: workers={len(workers)}")
   560	    else:
   561	        health = await fetch_server_metrics(url=url)
   562	        if not health:
   563	            # Try health endpoint directly
   564	            try:
   565	                async with aiohttp.ClientSession() as s:
   566	                    async with s.get(f"{url}/health", timeout=aiohttp.ClientTimeout(total=5)) as r:
   567	                        if r.status != 200:
   568	                            print(f"Server not ready: {r.status}")
   569	                            sys.exit(1)
   570	                        print(f"Server ready: {await r.json()}")
   571	            except Exception as e:
   572	                print(f"Cannot reach server: {e}")
   573	                sys.exit(1)
   574
   575	    all_results = []
   576
   577	    for mode in modes:
   578	        for level in levels:
   579	            metrics = await run_stress_level(
   580	                url,
   581	                level,
   582	                mode,
   583	                a.audio_duration,
   584	                a.endpoint,
   585	                endpoint_key,
   586	                a.route_cost,
   587	                a.ws_port,
   588	            )
   589	            all_results.append({"level": level, "mode": mode, "metrics": metrics})
   590
   591	            # Cooldown between levels
   592	            if level != levels[-1]:
   593	                print(f"\n  Cooldown {a.cooldown}s...")
   594	                await asyncio.sleep(a.cooldown)
   595
   596	    # Write CSV output
   597	    output_file = a.output or f"stress_results_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv"
   598	    output_path = os.path.join(os.path.dirname(__file__), "..", "benchmark_results", output_file)
   599	    os.makedirs(os.path.dirname(output_path), exist_ok=True)
   600
   601	    with open(output_path, "w", newline="") as f:
   602	        writer = csv.DictWriter(f, fieldnames=[
   603	            "mode", "concurrency", "ok", "errors", "wall_time_s",
   604	            "connect_p50", "connect_p95", "connect_p99",
   605	            "first_token_p50", "first_token_p95", "first_token_p99",
   606	            "total_p50", "total_p95", "total_p99",
   607	            "msgs_avg", "throughput_x",
   608	        ])
   609	        writer.writeheader()
   610	        for r in all_results:
   611	            s = r["metrics"].summary()
   612	            writer.writerow({
   613	                "mode": s["mode"], "concurrency": s["concurrency"],
   614	                "ok": s["ok"], "errors": s["errors"], "wall_time_s": s["wall_time_s"],
   615	                "connect_p50": s["connect_ms_p50"], "connect_p95": s["connect_ms_p95"], "connect_p99": s["connect_ms_p99"],
   616	                "first_token_p50": s["first_token_ms_p50"], "first_token_p95": s["first_token_ms_p95"], "first_token_p99": s["first_token_ms_p99"],
   617	                "total_p50": s["total_ms_p50"], "total_p95": s["total_ms_p95"], "total_p99": s["total_ms_p99"],
   618	                "msgs_avg": s["msgs_per_stream_avg"], "throughput_x": s["throughput_x"],
   619	            })
   620	    print(f"\nResults saved to: {output_path}")
   621
   622	    # Also save full JSON with server metrics
   623	    json_path = output_path.replace(".csv", ".json")
   624	    with open(json_path, "w") as f:
   625	        json.dump([{
   626	            "level": r["level"], "mode": r["mode"],
   627	            "summary": r["metrics"].summary(),
   628	            "server_before": r["metrics"].server_metrics_before,
   629	            "server_after": r["metrics"].server_metrics_after,
   630	            "errors": r["metrics"].errors,
   631	        } for r in all_results], f, indent=2)
   632	    print(f"Full metrics saved to: {json_path}")
   633
   634	    # Print summary table
   635	    print(f"\n{'='*80}")
   636	    print(f"  STRESS TEST SUMMARY")
   637	    print(f"{'='*80}")
   638	    print(f"{'Mode':>6} {'Conc':>5} {'OK':>4} {'Err':>4} {'FT p50':>8} {'FT p99':>8} {'Tot p50':>8} {'Msgs':>6}")
   639	    print(f"{'-'*6:>6} {'-'*5:>5} {'-'*4:>4} {'-'*4:>4} {'-'*8:>8} {'-'*8:>8} {'-'*8:>8} {'-'*6:>6}")
   640	    for r in all_results:
   641	        s = r["metrics"].summary()
   642	        print(f"{s['mode']:>6} {s['concurrency']:>5} {s['ok']:>4} {s['errors']:>4} "
   643	              f"{s['first_token_ms_p50']:>7.0f}ms {s['first_token_ms_p99']:>7.0f}ms "
   644	              f"{s['total_ms_p50']:>7.0f}ms {s['msgs_per_stream_avg']:>5.1f}")
   645
   646
   647	if __name__ == "__main__":
   648	    asyncio.run(main())


Output:
Chunk ID: 1986cb
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 3753
Output:
     1	"""
     2	Production Hindi ASR Server - Standalone FastAPI
     3	Supports: HTTP batch, HTTP streaming (SSE), WebSocket streaming
     4	Designed for Vast.ai serverless deployment with PyWorker integration.
     5
     6	Architecture:
     7	  - NeMo cache-aware RNNT streaming pipeline (Nemotron-0.6B fine-tuned Hindi)
     8	  - Event-driven batching (50ms max wait, immediate dispatch when batch fills)
     9	  - CPU-based Silero VAD gating (filters silence before GPU, ~3x effective capacity)
    10	  - Per-stream state management with fairness scheduling
    11	  - Backpressure: rejects new connections when at capacity (fast-fail > silent degrade)
    12
    13	Performance (A100-SXM4-80GB, from prior benchmarks):
    14	  - Safe zone: 1-50 concurrent streams (WER 22-31%, FT <211ms)
    15	  - Optimal: 50-100 streams (WER 31-33%, FT <402ms)
    16	  - Edge: 100-140 streams (WER 33-37%)
    17	  - VRAM: 10.4GB constant (compute-bound, not memory-bound)
    18	"""
    19
    20	import os
    21	import sys
    22	import time
    23	import json
    24	import wave
    25	import asyncio
    26	import logging
    27	import tempfile
    28	import signal
    29	import base64
    30	from io import BytesIO
    31	from dataclasses import dataclass, field
    32	from typing import Optional, List
    33	from collections import deque
    34
    35	import numpy as np
    36	import torch
    37	import uvicorn
    38	from fastapi import FastAPI, WebSocket, WebSocketDisconnect, HTTPException, Request
    39	from fastapi.responses import JSONResponse, StreamingResponse
    40	from starlette.websockets import WebSocketState
    41
    42	# === Logging setup ===
    43	LOG_FILE = os.environ.get("MODEL_LOG", "/var/log/model/server.log")
    44	os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)
    45
    46	logging.basicConfig(
    47	    level=logging.INFO,
    48	    format="%(asctime)s [%(levelname)s] %(message)s",
    49	    handlers=[
    50	        logging.FileHandler(LOG_FILE),
    51	        logging.StreamHandler(sys.stdout),
    52	    ],
    53	)
    54	logger = logging.getLogger("asr_server")
    55
    56	# === Configuration ===
    57
    58	MODEL_PATH = os.environ.get("MODEL_PATH", "/model/final_model.nemo")
    59	HF_MODEL_REPO = os.environ.get("HF_MODEL_REPO", "BayAreaBoys/nemotron-hindi")
    60	HF_MODEL_FILE = os.environ.get("HF_MODEL_FILE", "final_model.nemo")
    61	SERVER_PORT = int(os.environ.get("SERVER_PORT", "8000"))
    62
    63	# Production limits
    64	MAX_ACTIVE_STREAMS = int(os.environ.get("MAX_ACTIVE_STREAMS", "100"))
    65	MAX_BATCH_WAIT_MS = float(os.environ.get("MAX_BATCH_WAIT_MS", "50"))
    66	ENABLE_VAD = os.environ.get("ENABLE_VAD", "true").lower() == "true"
    67	VAD_THRESHOLD = float(os.environ.get("VAD_THRESHOLD", "0.5"))
    68
    69
    70	@dataclass
    71	class PipelineConfig:
    72	    """Pipeline configuration — matches NeMo's expected OmegaConf structure"""
    73	    model_name: str = ""
    74	    device: str = "cuda"
    75	    device_id: int = 0
    76	    compute_dtype: str = "bfloat16"
    77	    use_amp: bool = True
    78	    # Decoding
    79	    decoding_strategy: str = "greedy_batch"
    80	    use_cuda_graph_decoder: bool = False
    81	    max_symbols: int = 10
    82	    # Streaming
    83	    sample_rate: int = 16000
    84	    batch_size: int = 512
    85	    att_context_size: list = field(default_factory=lambda: [70, 6])
    86	    use_cache: bool = True
    87	    use_feat_cache: bool = True
    88	    num_slots: int = 1024
    89	    # Endpointing
    90	    stop_history_eou: int = 800
    91	    residue_tokens_at_end: int = 2
    92
    93
    94	# === Global server state ===
    95	class ASRServerState:
    96	    """
    97	    Centralized server state. Single instance per process.
    98	    WHY: Avoids global variable soup, enables clean shutdown.
    99	    """
   100	    def __init__(self):
   101	        self.pipeline = None
   102	        self.cfg = None
   103	        self.vad = None
   104	        self.start_time = None
   105
   106	        # Stream management
   107	        self.client_queues: dict[int, asyncio.Queue] = {}
   108	        self.client_queues_lock = asyncio.Lock()
   109	        self.stream_audio_timestamps: dict[int, float] = {}
   110	        self.stream_timestamps_lock = asyncio.Lock()
   111	        self.streams_pending_cleanup: set[int] = set()
   112	        self.streams_cleanup_lock = asyncio.Lock()
   113
   114	        # Event-driven batching
   115	        self._batch_ready_event = asyncio.Event()
   116	        self.inference_task = None
   117	        self.inference_running = False
   118
   119	        # Metrics (rolling window for percentile calculation)
   120	        self.metrics = {
   121	            "total_batches": 0,
   122	            "total_outputs": 0,
   123	            "total_gpu_ms": 0.0,
   124	            "total_route_ms": 0.0,
   125	            "batch_sizes": deque(maxlen=500),
   126	            "gpu_times_ms": deque(maxlen=500),
   127	            "cycle_times_ms": deque(maxlen=500),
   128	            "empty_cycles": 0,
   129	            "vad_filtered_chunks": 0,
   130	            "http_requests_total": 0,
   131	            "http_requests_ok": 0,
   132	            "http_requests_err": 0,
   133	            "ws_connections_total": 0,
   134	            "ws_connections_active": 0,
   135	            "ws_connections_rejected": 0,
   136	        }
   137	        self.model_loaded = False
   138
   139
   140	state = ASRServerState()
   141
   142
   143	# === FastAPI App ===
   144	app = FastAPI(
   145	    title="Hindi ASR Production Server",
   146	    description="Cache-aware streaming ASR for Hindi (Nemotron-0.6B fine-tuned)",
   147	    version="1.0.0",
   148	)
   149
   150
   151	# ============================================================
   152	# Model Loading
   153	# ============================================================
   154
   155	def download_model_if_needed() -> str:
   156	    """Download model from HuggingFace if not found locally. Returns local path."""
   157	    if os.path.exists(MODEL_PATH):
   158	        logger.info(f"Model found at {MODEL_PATH}")
   159	        return MODEL_PATH
   160
   161	    logger.info(f"Model not found at {MODEL_PATH}, downloading from {HF_MODEL_REPO}...")
   162	    from huggingface_hub import hf_hub_download
   163	    hf_token = os.environ.get("HF_TOKEN")
   164	    model_path = hf_hub_download(
   165	        repo_id=HF_MODEL_REPO,
   166	        filename=HF_MODEL_FILE,
   167	        token=hf_token,
   168	        cache_dir="/model",
   169	    )
   170	    logger.info(f"Model downloaded to {model_path}")
   171	    return model_path
   172
   173
   174	def build_pipeline_config(model_path: str):
   175	    """Build OmegaConf config for NeMo pipeline"""
   176	    from omegaconf import OmegaConf
   177	    from dataclasses import dataclass, field
   178	    from typing import Optional, List
   179
   180	    # Full config structure matching CacheAwarePipelineConfig from nemotron_asr
   181	    @dataclass
   182	    class BoostingTreeConfig:
   183	        model_path: Optional[str] = None
   184	        key_phrases_file: Optional[str] = None
   185	        key_phrases_list: Optional[List[str]] = None
   186	        source_lang: str = "en"
   187
   188	    @dataclass
   189	    class GreedyDecodingConfig:
   190	        use_cuda_graph_decoder: bool = False
   191	        max_symbols: int = 10
   192	        ngram_lm_model: Optional[str] = None
   193	        ngram_lm_alpha: float = 0.0
   194	        boosting_tree: BoostingTreeConfig = field(default_factory=BoostingTreeConfig)
   195	        boosting_tree_alpha: float = 0.0
   196
   197	    @dataclass
   198	    class DecodingConfig:
   199	        strategy: str = "greedy_batch"
   200	        preserve_alignments: bool = False
   201	        fused_batch_size: int = -1
   202	        greedy: GreedyDecodingConfig = field(default_factory=GreedyDecodingConfig)
   203
   204	    @dataclass
   205	    class ASRConfig:
   206	        model_name: str = model_path
   207	        device: str = "cuda"
   208	        device_id: int = 0
   209	        compute_dtype: str = "bfloat16"
   210	        use_amp: bool = True
   211	        decoding: DecodingConfig = field(default_factory=DecodingConfig)
   212
   213	    @dataclass
   214	    class ITNConfig:
   215	        input_case: str = "lower_cased"
   216	        whitelist: Optional[str] = None
   217	        overwrite_cache: bool = False
   218	        max_number_of_permutations_per_split: int = 729
   219	        left_padding_size: int = 4
   220	        batch_size: int = 32
   221	        n_jobs: int = 16
   222
   223	    @dataclass
   224	    class ConfidenceConfig:
   225	        exclude_blank: bool = True
   226	        aggregation: str = "mean"
   227	        method_cfg: dict = field(default_factory=lambda: {
   228	            "name": "entropy",
   229	            "entropy_type": "tsallis",
   230	            "alpha": 0.5,
   231	            "entropy_norm": "exp",
   232	        })
   233
   234	    @dataclass
   235	    class EndpointingConfig:
   236	        stop_history_eou: int = 800
   237	        residue_tokens_at_end: int = 2
   238
   239	    @dataclass
   240	    class StreamingConfig:
   241	        sample_rate: int = 16000
   242	        batch_size: int = 512
   243	        word_boundary_tolerance: int = 4
   244	        att_context_size: List[int] = field(default_factory=lambda: [70, 6])
   245	        use_cache: bool = True
   246	        use_feat_cache: bool = True
   247	        chunk_size_in_secs: Optional[float] = None
   248	        request_type: str = "frame"
   249	        num_slots: int = 1024
   250	        exhaustive_batching: bool = False
   251	        batching_delay_secs: float = 0.300
   252
   253	    @dataclass
   254	    class MetricsASRConfig:
   255	        gt_text_attr_name: str = "text"
   256	        clean_groundtruth_text: bool = False
   257	        langid: str = "hi"
   258	        use_cer: bool = True
   259	        ignore_capitalization: bool = True
   260	        ignore_punctuation: bool = True
   261	        strip_punc_space: bool = False
   262
   263	    @dataclass
   264	    class MetricsConfig:
   265	        asr: MetricsASRConfig = field(default_factory=MetricsASRConfig)
   266
   267	    @dataclass
   268	    class CacheAwarePipelineConfig:
   269	        asr: ASRConfig = field(default_factory=ASRConfig)
   270	        itn: ITNConfig = field(default_factory=ITNConfig)
   271	        nmt: Optional[dict] = None
   272	        confidence: ConfidenceConfig = field(default_factory=ConfidenceConfig)
   273	        endpointing: EndpointingConfig = field(default_factory=EndpointingConfig)
   274	        streaming: StreamingConfig = field(default_factory=StreamingConfig)
   275	        matmul_precision: str = "high"
   276	        log_level: int = 20
   277	        pipeline_type: str = "cache_aware"
   278	        asr_decoding_type: str = "rnnt"
   279	        audio_file: Optional[str] = None
   280	        output_filename: Optional[str] = None
   281	        output_dir: Optional[str] = None
   282	        enable_pnc: bool = False
   283	        enable_itn: bool = False
   284	        enable_nmt: bool = False
   285	        asr_output_granularity: str = "segment"
   286	        cache_dir: Optional[str] = None
   287	        lang: Optional[str] = "hi"
   288	        return_tail_result: bool = False
   289	        calculate_wer: bool = True
   290	        calculate_bleu: bool = True
   291	        metrics: MetricsConfig = field(default_factory=MetricsConfig)
   292
   293	    config = CacheAwarePipelineConfig()
   294	    config.asr.model_name = model_path
   295	    return OmegaConf.structured(config)
   296
   297
   298	async def load_model():
   299	    """Load model and build pipeline. Called at startup."""
   300	    from omegaconf import OmegaConf
   301
   302	    # Silence chatty NeMo logs
   303	    logging.getLogger("nemo_logger").setLevel(logging.WARNING)
   304
   305	    logger.info("=== Starting model load ===")
   306	    model_path = download_model_if_needed()
   307	    state.cfg = build_pipeline_config(model_path)
   308
   309	    logger.info(f"Building pipeline: model={model_path}")
   310	    logger.info(f"  att_context_size={state.cfg.streaming.att_context_size}")
   311	    logger.info(f"  batch_size={state.cfg.streaming.batch_size}")
   312
   313	    # Build pipeline (reuses nemotron_asr pipeline code)
   314	    from pipelines import StreamingPipelineBuilder
   315	    state.pipeline = StreamingPipelineBuilder.build_pipeline(state.cfg)
   316
   317	    # Initialize streaming support
   318	    state.pipeline.init_streaming_request_generator()
   319	    logger.info("Streaming request generator initialized")
   320
   321	    # Warmup GPU (3 iterations)
   322	    logger.info("Warming up GPU (3 iterations)...")
   323	    from asr_utils import preprocess_audio
   324	    from nemo.collections.asr.inference.streaming.framing.request_options import ASRRequestOptions
   325
   326	    WARMUP_URL = "https://github.com/voxserv/audio_quality_testing_samples/raw/refs/heads/master/mono_44100/156550__acclivity__a-dream-within-a-dream.wav"
   327	    try:
   328	        audio_bytes = preprocess_audio(WARMUP_URL, target_sample_rate=16000)
   329	    except Exception as e:
   330	        # If warmup audio fails to download, generate silence
   331	        logger.warning(f"Warmup audio download failed: {e}, using silence")
   332	        audio_bytes = np.zeros(16000 * 3, dtype=np.int16).tobytes()  # 3s silence
   333
   334	    chunk_size = int(0.080 * 16000 * 2)  # 80ms chunks, 16kHz, 16-bit
   335
   336	    for warmup_iter in range(1, 4):
   337	        options = ASRRequestOptions()
   338	        options.asr_output_granularity = "word"
   339	        stream_id = state.pipeline.open_streaming_session(options=options)
   340
   341	        for i in range(0, len(audio_bytes), chunk_size):
   342	            chunk = audio_bytes[i:i + chunk_size]
   343	            state.pipeline.append_streaming_audio(stream_id, chunk)
   344	            await state.pipeline.process_streaming_batch()
   345
   346	        state.pipeline.close_streaming_session(stream_id)
   347	        try:
   348	            await state.pipeline.process_streaming_batch()
   349	        except Exception:
   350	            pass
   351
   352	        # Cleanup warmup stream
   353	        if stream_id in state.pipeline._state_pool:
   354	            del state.pipeline._state_pool[stream_id]
   355	        if stream_id in state.pipeline._streaming_request_generator.streams:
   356	            state.pipeline._streaming_request_generator.streams.pop(stream_id, None)
   357
   358	        logger.info(f"  Warmup {warmup_iter}/3 complete")
   359
   360	    # Initialize VAD


**Noting server infrastructure needs**


I’ve identified the immediate technical gap: endpoint-mode WS still assumes a fixed host port, but Vast serverless maps `18000` to a random host port per worker. I’m now patching the smoke/stress tools to auto-discover the mapped WS host port from the worker instance API and then re-running WS tests.


**Inspecting server websocket routes**


rg -n "@app\.websocket|/ws|metrics|audio_timestamp|att_context_size|chunk|VAD|silero|process_streaming_batch|open_streaming_session" production/server.py | sed -n '1,240p'


nl -ba production/server.py | sed -n '360,860p'


nl -ba production/worker.py | sed -n '1,420p'


Output:
Chunk ID: b0467a
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 1695
Output:
9:  - CPU-based Silero VAD gating (filters silence before GPU, ~3x effective capacity)
66:ENABLE_VAD = os.environ.get("ENABLE_VAD", "true").lower() == "true"
67:VAD_THRESHOLD = float(os.environ.get("VAD_THRESHOLD", "0.5"))
85:    att_context_size: list = field(default_factory=lambda: [70, 6])
109:        self.stream_audio_timestamps: dict[int, float] = {}
120:        self.metrics = {
129:            "vad_filtered_chunks": 0,
244:        att_context_size: List[int] = field(default_factory=lambda: [70, 6])
247:        chunk_size_in_secs: Optional[float] = None
291:        metrics: MetricsConfig = field(default_factory=MetricsConfig)
310:    logger.info(f"  att_context_size={state.cfg.streaming.att_context_size}")
334:    chunk_size = int(0.080 * 16000 * 2)  # 80ms chunks, 16kHz, 16-bit
339:        stream_id = state.pipeline.open_streaming_session(options=options)
341:        for i in range(0, len(audio_bytes), chunk_size):
342:            chunk = audio_bytes[i:i + chunk_size]
343:            state.pipeline.append_streaming_audio(stream_id, chunk)
344:            await state.pipeline.process_streaming_batch()
348:            await state.pipeline.process_streaming_batch()
360:    # Initialize VAD
361:    if ENABLE_VAD:
362:        from vad import SileroVAD
363:        state.vad = SileroVAD(threshold=VAD_THRESHOLD, sample_rate=16000)
364:        logger.info(f"VAD enabled (threshold={VAD_THRESHOLD})")
366:        logger.info("VAD disabled")
385:    stream_id = state.pipeline.open_streaming_session(options=options)
387:    # Use pipeline's native chunk size if available; fallback to 80ms
388:    frame_secs = getattr(state.pipeline, "chunk_size_in_secs", 0.08) or 0.08
395:        # Feed audio in frame-sized chunks
397:            chunk = pcm_bytes[i:i + frame_bytes]
398:            state.pipeline.append_streaming_audio(stream_id, chunk)
400:                requests, outputs = await state.pipeline.process_streaming_batch()
420:                requests, outputs = await state.pipeline.process_streaming_batch()
459:    Returns (audio_bytes, language, response_format, chunk_duration_ms).
478:        chunk_duration_ms = int(body.get("chunk_duration_ms", 160) or 160)
479:        return audio_data, language, response_format, chunk_duration_ms
494:    chunk_duration_ms = int(form.get("chunk_duration_ms", 160) or 160)
495:    return audio_data, language, response_format, chunk_duration_ms
537:def _build_metrics_payload():
538:    """Shared metrics payload for GET/POST /metrics."""
539:    m = state.metrics
570:            "vad_filtered_chunks": m["vad_filtered_chunks"],
597:@app.get("/metrics")
598:async def metrics_get():
600:    Detailed metrics endpoint for monitoring/alerting.
603:    return _build_metrics_payload()
606:@app.post("/metrics")
607:async def metrics_post(_request: Request):
609:    return _build_metrics_payload()
619:    state.metrics["http_requests_total"] += 1
621:        state.metrics["http_requests_err"] += 1
625:        audio_data, language, _response_format, _chunk_duration_ms = await _parse_audio_request(request)
632:        state.metrics["http_requests_ok"] += 1
641:        state.metrics["http_requests_err"] += 1
655:    state.metrics["http_requests_total"] += 1
658:        state.metrics["http_requests_err"] += 1
665:        audio_data, language, response_format, _chunk_duration_ms = await _parse_audio_request(request)
687:        state.metrics["http_requests_ok"] += 1
694:        state.metrics["http_requests_err"] += 1
708:    state.metrics["http_requests_total"] += 1
711:        state.metrics["http_requests_err"] += 1
715:        audio_data, language, response_format, chunk_duration_ms = await _parse_audio_request(request)
728:            stream_id = state.pipeline.open_streaming_session(options=options)
730:            chunk_size = int(chunk_duration_ms / 1000.0 * 16000 * 2)  # bytes per chunk
731:            att_ctx = state.cfg.streaming.att_context_size[1] + 1
739:                for i in range(0, len(pcm_bytes), chunk_size):
740:                    chunk = pcm_bytes[i:i + chunk_size]
741:                    audio_buffer += chunk
749:                        requests, outputs = await state.pipeline.process_streaming_batch()
778:                        requests, outputs = await state.pipeline.process_streaming_batch()
811:        state.metrics["http_requests_ok"] += 1
821:        state.metrics["http_requests_err"] += 1
830:@app.websocket("/ws")
836:      Client -> Server: binary audio chunks (raw PCM 16-bit 16kHz mono)
843:    state.metrics["ws_connections_total"] += 1
848:        state.metrics["ws_connections_rejected"] += 1
854:    state.metrics["ws_connections_active"] += 1
862:        stream_id = state.pipeline.open_streaming_session()
869:        state.metrics["ws_connections_active"] -= 1
880:        """Receive audio chunks from client, feed to pipeline"""
881:        att_ctx = state.cfg.streaming.att_context_size[1] + 1
905:                        # VAD gate: skip silence before GPU
907:                            state.metrics["vad_filtered_chunks"] += 1
974:        state.metrics["ws_connections_active"] -= 1
997:        state.stream_audio_timestamps[stream_id] = 0.0
1007:            state.stream_audio_timestamps.pop(stream_id, None)
1024:        audio_ts = output_dict["audio_timestamp"]
1033:            "audio_timestamp": audio_ts,
1087:    metrics_report_interval = 100
1115:                timestamps = dict(state.stream_audio_timestamps)
1121:                    requests, outputs = await state.pipeline.process_streaming_batch()
1128:                    state.metrics["total_batches"] += 1
1129:                    state.metrics["total_outputs"] += len(outputs)
1130:                    state.metrics["total_gpu_ms"] += gpu_ms
1131:                    state.metrics["batch_sizes"].append(len(outputs))
1132:                    state.metrics["gpu_times_ms"].append(gpu_ms)
1143:                            "audio_timestamp": timestamps.get(output.stream_id, 0.0),
1151:                            state.metrics["empty_cycles"] += 1
1158:                        if sid in state.stream_audio_timestamps:
1159:                            state.stream_audio_timestamps[sid] = ts
1163:                state.metrics["total_route_ms"] += (time.perf_counter() - route_start) * 1000
1168:                state.metrics["cycle_times_ms"].append(cycle_ms)
1170:                # Periodic metrics log
1171:                if state.metrics["total_batches"] % metrics_report_interval == 0:
1172:                    _log_metrics(active)
1181:def _log_metrics(active_clients: int):
1182:    """Structured metrics log"""
1183:    m = state.metrics
1203:        f"vad_skip={m['vad_filtered_chunks']}"
1226:    logger.info(f"  VAD: {'enabled' if ENABLE_VAD else 'disabled'}")