You are Codex, a coding agent based on GPT-5. You and the user share the same workspace and collaborate to achieve the user's goals.

# Personality

You are a deeply pragmatic, effective software engineer. You take engineering quality seriously, and collaboration comes through as direct, factual statements. You communicate efficiently, keeping the user clearly informed about ongoing actions without unnecessary detail.

## Values
You are guided by these core values:
- Clarity: You communicate reasoning explicitly and concretely, so decisions and tradeoffs are easy to evaluate upfront.
- Pragmatism: You keep the end goal and momentum in mind, focusing on what will actually work and move things forward to achieve the user's goal.
- Rigor: You expect technical arguments to be coherent and defensible, and you surface gaps or weak assumptions politely with emphasis on creating clarity and moving the task forward.

## Interaction Style
You communicate concisely and respectfully, focusing on the task at hand. You always prioritize actionable guidance, clearly stating assumptions, environment prerequisites, and next steps. Unless explicitly asked, you avoid excessively verbose explanations about your work.

You avoid cheerleading, motivational language, or artificial reassurance, or any kind of fluff. You don't comment on user requests, positively or negatively, unless there is reason for escalation. You don't feel like you need to fill the space with words, you stay concise and communicate what is necessary for user collaboration - not more, not less.

## Escalation
You may challenge the user to raise their technical bar, but you never patronize or dismiss their concerns. When presenting an alternative approach or solution to the user, you explain the reasoning behind the approach, so your thoughts are demonstrably correct. You maintain a pragmatic mindset when discussing these tradeoffs, and so are willing to work with the user after concerns have been noted.

# General

- When searching for text or files, prefer using `rg` or `rg --files` respectively because `rg` is much faster than alternatives like `grep`. (If the `rg` command is not found, then use alternatives.)
- Parallelize tool calls whenever possible - especially file reads, such as `cat`, `rg`, `sed`, `ls`, `git show`, `nl`, `wc`. Use `multi_tool_use.parallel` to parallelize tool calls and only this.

## Editing constraints

- Default to ASCII when editing or creating files. Only introduce non-ASCII or other Unicode characters when there is a clear justification and the file already uses them.
- Add succinct code comments that explain what is going on if code is not self-explanatory. You should not add comments like "Assigns the value to the variable", but a brief comment might be useful ahead of a complex code block that the user would otherwise have to spend time parsing out. Usage of these comments should be rare.
- Try to use apply_patch for single file edits, but it is fine to explore other options to make the edit if it does not work well. Do not use apply_patch for changes that are auto-generated (i.e. generating package.json or running a lint or format command like gofmt) or when scripting is more efficient (such as search and replacing a string across a codebase).
- Do not use Python to read/write files when a simple shell command or apply_patch would suffice.
- You may be in a dirty git worktree.
    * NEVER revert existing changes you did not make unless explicitly requested, since these changes were made by the user.
    * If asked to make a commit or code edits and there are unrelated changes to your work or changes that you didn't make in those files, don't revert those changes.
    * If the changes are in files you've touched recently, you should read carefully and understand how you can work with the changes rather than reverting them.
    * If the changes are in unrelated files, just ignore them and don't revert them.
- Do not amend a commit unless explicitly requested to do so.
- While you are working, you might notice unexpected changes that you didn't make. If this happens, STOP IMMEDIATELY and ask the user how they would like to proceed.
- **NEVER** use destructive commands like `git reset --hard` or `git checkout --` unless specifically requested or approved by the user.
- You struggle using the git interactive console. **ALWAYS** prefer using non-interactive git commands.

## Special user requests

- If the user makes a simple request (such as asking for the time) which you can fulfill by running a terminal command (such as `date`), you should do so.
- If the user asks for a "review", default to a code review mindset: prioritise identifying bugs, risks, behavioural regressions, and missing tests. Findings must be the primary focus of the response - keep summaries or overviews brief and only after enumerating the issues. Present findings first (ordered by severity with file/line references), follow with open questions or assumptions, and offer a change-summary only as a secondary detail. If no findings are discovered, state that explicitly and mention any residual risks or testing gaps.

## Frontend tasks

When doing frontend design tasks, avoid collapsing into "AI slop" or safe, average-looking layouts.
Aim for interfaces that feel intentional, bold, and a bit surprising.
- Typography: Use expressive, purposeful fonts and avoid default stacks (Inter, Roboto, Arial, system).
- Color & Look: Choose a clear visual direction; define CSS variables; avoid purple-on-white defaults. No purple bias or dark mode bias.
- Motion: Use a few meaningful animations (page-load, staggered reveals) instead of generic micro-motions.
- Background: Don't rely on flat, single-color backgrounds; use gradients, shapes, or subtle patterns to build atmosphere.
- Overall: Avoid boilerplate layouts and interchangeable UI patterns. Vary themes, type families, and visual languages across outputs.
- Ensure the page loads properly on both desktop and mobile

Exception: If working within an existing website or design system, preserve the established patterns, structure, and visual language.

# Working with the user

You interact with the user through a terminal. You have 2 ways of communicating with the users:
- Share intermediary updates in `commentary` channel.
- After you have completed all your work, send a message to the `final` channel.
You are producing plain text that will later be styled by the program you run in. Formatting should make results easy to scan, but not feel mechanical. Use judgment to decide how much structure adds value. Follow the formatting rules exactly.

## Autonomy and persistence
Persist until the task is fully handled end-to-end within the current turn whenever feasible: do not stop at analysis or partial fixes; carry changes through implementation, verification, and a clear explanation of outcomes unless the user explicitly pauses or redirects you.

Unless the user explicitly asks for a plan, asks a question about the code, is brainstorming potential solutions, or some other intent that makes it clear that code should not be written, assume the user wants you to make code changes or run tools to solve the user's problem. In these cases, it's bad to output your proposed solution in a message, you should go ahead and actually implement the change. If you encounter challenges or blockers, you should attempt to resolve them yourself.

## Formatting rules

- You may format with GitHub-flavored Markdown.
- Structure your answer if necessary, the complexity of the answer should match the task. If the task is simple, your answer should be a one-liner. Order sections from general to specific to supporting.
- Never use nested bullets. Keep lists flat (single level). If you need hierarchy, split into separate lists or sections or if you use : just include the line you might usually render using a nested bullet immediately after it. For numbered lists, only use the `1. 2. 3.` style markers (with a period), never `1)`.
- Headers are optional, only use them when you think they are necessary. If you do use them, use short Title Case (1-3 words) wrapped in **…**. Don't add a blank line.
- Use monospace commands/paths/env vars/code ids, inline examples, and literal keyword bullets by wrapping them in backticks.
- Code samples or multi-line snippets should be wrapped in fenced code blocks. Include an info string as often as possible.
- File References: When referencing files in your response follow the below rules:
  * Use inline code to make file paths clickable.
  * Each reference should have a stand alone path. Even if it's the same file.
  * Accepted: absolute, workspace‑relative, a/ or b/ diff prefixes, or bare filename/suffix.
  * Optionally include line/column (1‑based): :line[:column] or #Lline[Ccolumn] (column defaults to 1).
  * Do not use URIs like file://, vscode://, or https://.
  * Do not provide range of lines
  * Examples: src/app.ts, src/app.ts:42, b/server/index.js#L10, C:\repo\project\main.rs:12:5
- Don’t use emojis or em dashes unless explicitly instructed.

## Final answer instructions

- Balance conciseness to not overwhelm the user with appropriate detail for the request. Do not narrate abstractly; explain what you are doing and why.
- Do not begin responses with conversational interjections or meta commentary. Avoid openers such as acknowledgements (“Done —”, “Got it”, “Great question, ”) or framing phrases.
- The user does not see command execution outputs. When asked to show the output of a command (e.g. `git show`), relay the important details in your answer or summarize the key lines so the user understands the result.
- Never tell the user to "save/copy this file", the user is on the same machine and has access to the same files as you have.
- If the user asks for a code explanation, structure your answer with code references.
- When given a simple task, just provide the outcome in a short answer without strong formatting.
- When you make big or complex changes, state the solution first, then walk the user through what you did and why.
- For casual chit-chat, just chat.
- If you weren't able to do something, for example run tests, tell the user.
- If there are natural next steps the user may want to take, suggest them at the end of your response. Do not make suggestions if there are no natural next steps. When suggesting multiple options, use numeric lists for the suggestions so the user can quickly respond with a single number.

## Intermediary updates

- Intermediary updates go to the `commentary` channel.
- User updates are short updates while you are working, they are NOT final answers.
- You use 1-2 sentence user updates to communicated progress and new information to the user as you are doing work.
- Do not begin responses with conversational interjections or meta commentary. Avoid openers such as acknowledgements (“Done —”, “Got it”, “Great question, ”) or framing phrases.
- You provide user updates frequently, every 20s.
- Before exploring or doing substantial work, you start with a user update acknowledging the request and explaining your first step. You should include your understanding of the user request and explain what you will do. Avoid commenting on the request or using starters such at "Got it -" or "Understood -" etc.
- When exploring, e.g. searching, reading files you provide user updates as you go, every 20s, explaining what context you are gathering and what you've learned. Vary your sentence structure when providing these updates to avoid sounding repetitive - in particular, don't start each sentence the same way.
- After you have sufficient context, and the work is substantial you provide a longer plan (this is the only user update that may be longer than 2 sentences and can contain formatting).
- Before performing file edits of any kind, you provide updates explaining what edits you are making.
- As you are thinking, you very frequently provide updates even if not taking any actions, informing the user of your progress. You interrupt your thinking and send multiple updates in a row if thinking for more than 100 words.
- Tone of your updates MUST match your personality.


<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
 Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `never`: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.
</permissions instructions>


# AGENTS.md instructions for /home/ubuntu/nemo_hindi

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>


<environment_context>
  <cwd>/home/ubuntu/nemo_hindi</cwd>
  <shell>bash</shell>
</environment_context>


<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>


Here is what i asked my another coding agent.

finish revamp with clean architecture mode implementation, then deploy to vast(all keys are in .env) serverless workers. then do stress testing simulating like actual customers to understand the working production server behaviour mocking and capture all metrics, so we can set scaling laws and move our production OpenAI transcription paid service to self hosten custom finetuned hindi model and serve any customer base with proper scaling laws set. assuming somewhere at 500 concurrent batching requests per GPU of 80GB GPU. Also while deploying check if region based deployment is available and check for closer to india or asia or anything closer. Also while implementation or planning, plan for both HTTP based and websocket based requests handlers. If we have vCPUs that take requests and sends batches of chunks to GPU and gets data back, or triton server, or whatever man. i just need high throughput mode with both HTTP streaming and websocket streaming.

Read the vast's serverless docs, requirements
Plan the architecture that is failproof and high throughput mode with quality considered too.
start working on it
test it, deploy it to serverless
smoke test
stress test it and capture all major metrics(all) -> simulation tests that will be actual customers usage based
then we'll discuss on scaling laws.

and after a while of trying it failed but got somewhere. and below is a handover summary from that agent. Can you now take over and finish the proper deployment, testing, stress testing, and capture metrics. below is it's summary. Also, all secrets are in .env FYI. docs are also in this repo for vast's serverless stack. understand the flow. and see the current implementation, make changes, or revamp as needed.

## Summary

### What We Started With
- A **fine-tuned Hindi ASR model** (Nemotron-0.6B, 8.8% WER) deployed on Modal
- Prior benchmarks showing **A100-SXM4-80GB handles 50-100 concurrent streams** in the safe zone
- Existing code in `nemotron_asr/` — a Modal-coupled WebSocket streaming server with event-driven batching, VAD, and centralized inference loop
- The decision to move to **Vast.ai serverless** for production (cheaper, direct TCP, autoscaling)

### What We Want To Do
1. Build a **production-ready ASR service** with both HTTP and WebSocket endpoints
2. Deploy it on **Vast.ai serverless** with autoscaling (not bare metal instances)
3. Run **stress tests simulating real customers** to capture all metrics
4. Establish **scaling laws** so we can replace paid OpenAI transcription with our self-hosted model

### What We Built
- `production/server.py` — Standalone FastAPI server (HTTP batch, HTTP SSE streaming, WebSocket streaming)
- `production/worker.py` — Vast.ai PyWorker config (metrics reporting, benchmarks, autoscaling integration)
- `production/onstart.sh` — Startup script that launches both model server and PyWorker
- `production/Dockerfile.slim` — Slim image extending existing v5 image
- `production/deploy.py` — Deployment automation (create/destroy endpoint+workergroup)
- `production/test_smoke.py` — Smoke test (health, HTTP, WebSocket)
- `production/test_stress.py` — Full stress test with percentile metrics, CSV/JSON output
- PyWorker repo pushed to GitHub, Docker image on Docker Hub

### Vast.ai Serverless Deployment (Live)
- Endpoint: `hindi-asr-prod` (id=11967)
- Workergroup: 16773 (template hash `3ac272ae9dc0ae58d105a1d88145a325`)
- Worker provisioned and managed by the serverless engine

### Issues We Hit (and Fixed)

| Issue | Root Cause | Fix |
|:---|:---|:---|
| Created bare metal instance instead of serverless | My mistake — didn't follow docs | Destroyed instance, used `create endpoint` + `create workergroup` |
| `PYWORKER_REPO` env var empty on instance | Account-level env vars don't auto-propagate; Vast.ai doesn't auto-bootstrap PyWorker for custom images | Handle PyWorker clone+run in our own `onstart.sh` |
| `python-dateutil` downgraded to 2.6.1 | `vastai` pip package pulled old dateutil, broke matplotlib/NeMo | Install deps BEFORE starting server; pin `python-dateutil>=2.8.2` |
| `cannot import name 'Worker' from 'vastai'` | Wrong pip package — `vastai` is CLI tool, `vastai_sdk` has the Worker class | Changed requirements.txt to `vastai_sdk` |
| `KeyError: 'WORKER_PORT'` | `vastai_sdk` Worker expects env vars set by Vast.ai infra, which doesn't auto-set them for custom images | Manually set `WORKER_PORT=8000`, `VAST_TCP_PORT_8000` in onstart.sh; model server on 18000, PyWorker on 8000 |
| `Object of type bytes is not JSON serializable` | Benchmark generator returned file tuple with bytes; PyWorker sends benchmarks as JSON | Changed to base64-encoded audio, new `/v1/audio/transcriptions/json` endpoint |
| `'CacheAwareRNNTStreamingPipeline' has no attribute 'run'` | Streaming pipeline doesn't have `.run()` method (that's on the batch pipeline) | **Last fix** — replaced with `transcribe_pcm_bytes_streaming()` that uses the streaming session API |

### Where We Are Right Now
- Worker is **recycling** with the latest fix (streaming inference for benchmarks)
- Waiting to see if the PyWorker benchmark passes and the worker transitions to **"running"** (ready) status
- Once ready: smoke test, stress test, metrics capture

### What's Left After Worker Goes Ready
1. Smoke test (HTTP + WebSocket)
2. Stress test sweep (c=1,5,10,25,50,75,100) capturing all latency/throughput metrics
3. Analyze results, set scaling laws
4. Update README with production benchmarks

and it replied with

I made the server/pyworker JSON-compatible, rebuilt/pushed the image, updated the Vast template/workergroup, and wired the tests to run against `/route` with auth. The current serverless worker is still provisioning (status `loading`), so full stress runs are pending.

What changed (code + image)
- JSON+multipart compatibility for batch + SSE endpoints in `production/server.py` so PyWorker can call `/v1/audio/transcriptions` and `/v1/audio/transcriptions/stream` with JSON payloads.
- PyWorker now uses base64 JSON payloads, adds `/v1/audio/transcriptions/stream`, and uses a better workload estimator in `production/worker.py`.
- Onstart no longer clones a repo; it runs the bundled worker and only installs deps if missing in `production/onstart.sh`.
- Slim/full Dockerfiles now include `worker.py` and `vastai_sdk` deps in `production/Dockerfile.slim` and `production/Dockerfile`.
- Deployment + tests now understand endpoint auth via `/route` in `production/deploy.py`, `production/test_smoke.py`, and `production/test_stress.py`.

Image
- Built and pushed `bharathkumar192/hindi-nemotron-asr:v6-production` with the updated server/pyworker.

Current Vast state
- Endpoint: `hindi-asr-prod` (id `11967`)
- Workergroup: `16780` (template hash `3fc7205a26c3bd08fbd28f28ccca252a`)
- Template (id `341822`) now uses:
  - image `bharathkumar192/hindi-nemotron-asr:v6-production`
  - env `-p 8000:8000`
  - filters: `gpu_ram>=40 num_gpus=1 reliability>0.95 direct_port_count>=1 disk_space>=30 inet_down>200`
- Worker currently: `loading` (see `python3 production/deploy.py workers`).

Smoke results (last completed run on previous worker)
- Health: PASS
- HTTP batch: PASS (~1.1s for 3s test audio)
- SSE stream: PASS (~1.0s, 2 events)
- WebSocket: FAIL (timeout; no direct port mapping for 18000)

Why WebSocket failed in serverless
- The current template exposes only port 8000 (PyWorker). `/ws` lives on the model server port (18000), which is not mapped, so WS connections time out.
- To enable WS via serverless, the template must include `-p 18000:18000` and the search filters must require `direct_port_count>=2`. This is doable, but it caused unstable provisioning in this round.

How to proceed once the worker is idle
1) Smoke (serverless)
```
python3 production/test_smoke.py --endpoint hindi-asr-prod --route-cost 10
```

2) Stress (batch + SSE)
```
python3 production/test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10,25,50
python3 production/test_stress.py --endpoint hindi-asr-prod --mode sse  --levels 1,5,10,25,50
```
Outputs go to `benchmark_results/*.csv` and `benchmark_results/*.json`.

Region pinning (closer to India / Asia)
- Vast supports `geolocation` in search params. Example:
  - `geolocation in [IN,SG,JP,HK,AE]`
- You can apply this by updating the template search params:
```
vastai update template 3fc7205a26c3bd08fbd28f28ccca252a \
  --search_params "gpu_ram>=40 num_gpus=1 reliability>0.95 rented=False geolocation in [IN,SG,JP,HK,AE] direct_port_count>=1 disk_space>=30 inet_down>200"
```

If you want me to push WS support through serverless next, I’ll:
1) Update the template env to `-p 8000:8000 -p 18000:18000`.
2) Enforce `direct_port_count>=2`.
3) Recreate workergroup and re-run smoke/stress with `--ws-port 18000`.

Suggested next steps (pick one)
1) Wait for the current worker to hit `idle`, then I’ll run smoke + stress and report metrics.
2) Proceed with WebSocket-capable template/workergroup changes and retry provisioning.
3) Roll back to a fully open search (no geolocation) to get a worker fast, then collect metrics immediately.

and then i asked, check status

And it replied with
Status right now:
- Workers: 1
- Instance ID: `31133036`
- Status: `creating` (not ready yet)

Want me to keep watching until it reaches `idle` and then run smoke/stress?

and i asked,

what are the next steps. for our goal. here's what i asked.

Read the vast's serverless docs, requirements
Plan the architecture that is failproof and high throughput mode with quality considered too.
start working on it
test it, deploy it to serverless
smoke test
stress test it and capture all major metrics(all) -> simulation tests that will be actual customers usage based
then we'll discuss on scaling laws.

and it replied with

Here are the concrete next steps toward your stated goal, in order. I’ve kept them actionable and tied to what’s already in the repo.

Current state
- Serverless worker is still `creating` (not ready yet). We can’t run smoke/stress until it’s `idle`.

Next steps (exact order)
1) Wait for worker to become ready
   - I’ll keep polling `python3 production/deploy.py workers` until status is `idle`.

2) Smoke test (serverless)
   - Command:
     ```
     python3 production/test_smoke.py --endpoint hindi-asr-prod --route-cost 10
     ```
   - Confirms: auth flow, batch, SSE.
   - WebSocket is not reachable unless we expose port 18000 (see step 6).

3) Stress test (customer‑like load)
   - HTTP batch:
     ```
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10,25,50
     ```
   - SSE streaming:
     ```
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode sse --levels 1,5,10,25,50
     ```
   - Outputs go to `benchmark_results/*.csv` and `benchmark_results/*.json`.

4) Capture metrics and summarize
   - I’ll pull the CSV/JSON results and summarize p50/p95/p99 latency, throughput, error rate, and worker load snapshots.

5) Scaling laws draft
   - Use the stress results to propose concurrency limits per GPU and utilization targets.

6) WebSocket support (optional, but required if you want WS in prod)
   - We must expose port 18000 in the template and require `direct_port_count>=2`.
   - That requires template + workergroup update + new worker provisioning.
   - Once enabled, smoke/stress will include:
     ```
     python3 production/test_smoke.py --endpoint hindi-asr-prod --ws-port 18000
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode ws --ws-port 18000 --levels 1,5,10,25,50
     ```

7) Region preference (Asia/India)
- I can set `geolocation in [IN,SG,JP,HK,AE]` in the template search params, but that narrows supply and can slow provisioning. We can toggle it once baseline metrics are done.

What I need from you
- Confirm whether you want WebSocket enabled now (requires template/workergroup change + new worker provisioning), or do HTTP+SSE first to get metrics quickly.

If you say “go ahead,” I’ll:
- keep polling until the worker is idle,
- run smoke and stress for HTTP+SSE immediately,
- then report metrics and draft scaling‑law estimates.

and i asked go, we can test teh websockets later.

and it said back,
Worker is up and idle now. The serverless template was failing to start because the template’s `onstart` was set to `"bash /app/onstart.sh"` (exec-form string). I updated the template to use `/app/onstart.sh` and re-pointed the workergroup to the new template hash. After that, the worker loaded and stayed healthy.

Smoke + stress tests (HTTP + SSE) are done (websockets skipped as requested). All runs were 0 errors.

Key results (10s audio per stream):
- HTTP p50 total latency: 0.7s (c=1), 1.1s (c=5), 1.9s (c=10), 3.7s (c=25), 7.0s (c=50)
- SSE p50 total latency: 0.64s (c=1), 1.48s (c=5), 2.58s (c=10), 5.87s (c=25), 10.92s (c=50)
- Throughput: ~38–42x realtime at c=50 for both HTTP and SSE

Artifacts saved:
- `benchmark_results/stress_results_20260209_195631.csv`
- `benchmark_results/stress_results_20260209_195631.json`
- `benchmark_results/stress_results_20260209_195730.csv`
- `benchmark_results/stress_results_20260209_195730.json`

Next steps (pick what you want me to do):
1) Add “full metrics” capture by exposing `/metrics` via PyWorker (or mapping port 18000) so we can log GPU ms, VRAM, batch stats during stress tests.
2) WebSockets later: update template to `-p 18000:18000` and add `direct_port_count>=2`, then run `test_smoke.py` + `test_stress.py --mode ws`.
3) Region tuning: add `geolocation in [IN,SG,JP,HK,AE]` (or your preferred list) to search filters and re-run workergroup; we’ll verify offers exist so it doesn’t churn.
4) Scaling laws: use the new JSON outputs to fit throughput/latency curves and pick target concurrency per GPU.

Tell me which option to execute next (or ask for a specific plot/summary from the JSON files).

and i asked,  yes. let's do a thorough full metrics and stress testing first now. let's finish them now.

and it replied with,

Full‑metrics instrumentation and stress tests are done. I updated the server/worker to expose `/metrics` through PyWorker, fixed the CUDA VRAM reporting (`total_memory`), rebuilt the image, redeployed, and ran full HTTP + SSE sweeps with metrics captured per level.

Where the metrics live (latest runs):
- HTTP: `benchmark_results/stress_results_20260209_202433.csv` and `benchmark_results/stress_results_20260209_202433.json`
- SSE: `benchmark_results/stress_results_20260209_202718.csv` and `benchmark_results/stress_results_20260209_202718.json`

Key results (RTX PRO 6000 WS, 10s audio per stream):
- HTTP p50 total latency: 1.48s (c=1), 7.04s (c=25), 13.00s (c=50), 24.25s (c=100)
- HTTP throughput: ~20–21x realtime at c=50–100; c=100 had 2/100 route failures (“All worker queues full”)
- SSE p50 total latency: 1.51s (c=1), 11.02s (c=25), 20.21s (c=50), 34.65s (c=100)
- SSE throughput: ~22–23x realtime at c=50–100; c=100 had 2/100 route failures
- VRAM: ~10.43GB used, ~12.97GB reserved, 102GB total (now correctly reported)
- `/metrics` GPU p50/p99 stays 0 for HTTP/SSE because those fields are tied to the WS batching loop; the JSON still captures all other server metrics + VRAM + HTTP counts

What changed (code + deploy):
- `/metrics` now supports POST and returns CUDA fields (`cuda_available`, `device_count`, `gpu_name`, VRAM used/reserved/total) from `production/server.py`
- PyWorker now proxies `/metrics` in `production/worker.py`
- `production/test_stress.py` now fetches metrics via serverless `/route`
- Deployed image: `bharathkumar192/hindi-nemotron-asr:v9-production`
- Workergroup template hash: `118364eef727b32735fa02ee2908b7a1`
- Current worker: `31135755` (idle and healthy)

If you want “GPU time per request” for HTTP/SSE (not just VRAM), I can add per‑request timing in the HTTP/SSE handlers and expose percentiles in `/metrics`.

Next steps (pick one):
1) Run longer‑duration stress (30s/60s) to simulate real customer sessions and capture drift.
2) Add per‑request GPU timing for HTTP/SSE and re‑run the sweeps.
3) Proceed to scaling‑law fitting using the new JSON outputs.

Can you now do a thorough stress testing ensurimg we capture max metrics while doing so and on the way we should probaly ensure the WER also stays sane. and check for any issues where the current behaviour might fail in our production-ready application. We can work on websockets once the above is finished. first finish end to end metrics testing and auto-scaling rules are set and everything is done with http sse based worker.


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Starting repo inspection**


I’m taking over from the previous handoff and will finish this end-to-end for HTTP/SSE: verify current deployment state, run deeper stress + metrics + WER checks, then derive concrete autoscaling rules and production risks. I’ll start by auditing the current repo state and test scripts before running new sweeps.


Output:
Chunk ID: fb06ef
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 165
Output:
/home/ubuntu/nemo_hindi
total 52
drwxrwxr-x 10 ubuntu ubuntu  4096 Feb  9 08:14 .
drwxr-x--- 29 ubuntu ubuntu  4096 Feb  7 21:23 ..
-rw-rw-r--  1 ubuntu ubuntu   447 Feb  9 04:04 .env
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  8 03:59 __pycache__
drwxrwxr-x  2 ubuntu ubuntu 12288 Feb  9 20:27 benchmark_results
drwxrwxr-x  8 ubuntu ubuntu  4096 Feb  8 03:19 nemotron
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  7 21:23 nemotron-asr-frontend
drwxrwxr-x  3 ubuntu ubuntu  4096 Feb  8 20:33 nemotron_asr
drwxrwxr-x  3 ubuntu ubuntu  4096 Feb  9 08:27 production
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  9 07:47 vast_docs
drwxrwxr-x  6 ubuntu ubuntu  4096 Feb  7 20:14 venv


Output:
Chunk ID: 398717
Wall time: 0.0507 seconds
Process exited with code 128
Original token count: 18
Output:
fatal: not a git repository (or any of the parent directories): .git


Output:
Chunk ID: ca5df9
Wall time: 0.0511 seconds
Process exited with code 0
Original token count: 113
Output:
production/Dockerfile
production/Dockerfile.slim
production/__pycache__/deploy.cpython-310.pyc
production/__pycache__/server.cpython-310.pyc
production/__pycache__/test_smoke.cpython-310.pyc
production/__pycache__/test_stress.cpython-310.pyc
production/__pycache__/worker.cpython-310.pyc
production/deploy.py
production/onstart.sh
production/requirements.txt
production/server.py
production/test_smoke.py
production/test_stress.py
production/worker.py


Summary: auto


**Planning parallel file inspection**


I found this workspace isn’t a git repo, so I’ll treat it as a live working directory and avoid any git-based assumptions. Next I’m reading the production server, worker, deploy, and stress code to verify what metrics are truly captured versus just reported.


sed -n '1,320p' production/test_stress.py


Output:
Chunk ID: d2c163
Wall time: 0.0502 seconds
Process exited with code 0
Original token count: 2107
Output:
"""
Production Hindi ASR Server - Standalone FastAPI
Supports: HTTP batch, HTTP streaming (SSE), WebSocket streaming
Designed for Vast.ai serverless deployment with PyWorker integration.

Architecture:
  - NeMo cache-aware RNNT streaming pipeline (Nemotron-0.6B fine-tuned Hindi)
  - Event-driven batching (50ms max wait, immediate dispatch when batch fills)
  - CPU-based Silero VAD gating (filters silence before GPU, ~3x effective capacity)
  - Per-stream state management with fairness scheduling
  - Backpressure: rejects new connections when at capacity (fast-fail > silent degrade)

Performance (A100-SXM4-80GB, from prior benchmarks):
  - Safe zone: 1-50 concurrent streams (WER 22-31%, FT <211ms)
  - Optimal: 50-100 streams (WER 31-33%, FT <402ms)
  - Edge: 100-140 streams (WER 33-37%)
  - VRAM: 10.4GB constant (compute-bound, not memory-bound)
"""

import os
import sys
import time
import json
import wave
import asyncio
import logging
import tempfile
import signal
import base64
from io import BytesIO
from dataclasses import dataclass, field
from typing import Optional, List
from collections import deque

import numpy as np
import torch
import uvicorn
from fastapi import FastAPI, WebSocket, WebSocketDisconnect, HTTPException, Request
from fastapi.responses import JSONResponse, StreamingResponse
from starlette.websockets import WebSocketState

# === Logging setup ===
LOG_FILE = os.environ.get("MODEL_LOG", "/var/log/model/server.log")
os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler(LOG_FILE),
        logging.StreamHandler(sys.stdout),
    ],
)
logger = logging.getLogger("asr_server")

# === Configuration ===

MODEL_PATH = os.environ.get("MODEL_PATH", "/model/final_model.nemo")
HF_MODEL_REPO = os.environ.get("HF_MODEL_REPO", "BayAreaBoys/nemotron-hindi")
HF_MODEL_FILE = os.environ.get("HF_MODEL_FILE", "final_model.nemo")
SERVER_PORT = int(os.environ.get("SERVER_PORT", "8000"))

# Production limits
MAX_ACTIVE_STREAMS = int(os.environ.get("MAX_ACTIVE_STREAMS", "100"))
MAX_BATCH_WAIT_MS = float(os.environ.get("MAX_BATCH_WAIT_MS", "50"))
ENABLE_VAD = os.environ.get("ENABLE_VAD", "true").lower() == "true"
VAD_THRESHOLD = float(os.environ.get("VAD_THRESHOLD", "0.5"))

@dataclass
class PipelineConfig:
    """Pipeline configuration — matches NeMo's expected OmegaConf structure"""
    model_name: str = ""
    device: str = "cuda"
    device_id: int = 0
    compute_dtype: str = "bfloat16"
    use_amp: bool = True
    # Decoding
    decoding_strategy: str = "greedy_batch"
    use_cuda_graph_decoder: bool = False
    max_symbols: int = 10
    # Streaming
    sample_rate: int = 16000
    batch_size: int = 512
    att_context_size: list = field(default_factory=lambda: [70, 6])
    use_cache: bool = True
    use_feat_cache: bool = True
    num_slots: int = 1024
    # Endpointing
    stop_history_eou: int = 800
    residue_tokens_at_end: int = 2

# === Global server state ===
class ASRServerState:
    """
    Centralized server state. Single instance per process.
    WHY: Avoids global variable soup, enables clean shutdown.
    """
    def __init__(self):
        self.pipeline = None
        self.cfg = None
        self.vad = None
        self.start_time = None

        # Stream management
        self.client_queues: dict[int, asyncio.Queue] = {}
        self.client_queues_lock = asyncio.Lock()
        self.stream_audio_timestamps: dict[int, float] = {}
        self.stream_timestamps_lock = asyncio.Lock()
        self.streams_pending_cleanup: set[int] = set()
        self.streams_cleanup_lock = asyncio.Lock()

        # Event-driven batching
        self._batch_ready_event = asyncio.Event()
        self.inference_task = None
        self.inference_running = False

        # Metrics (rolling window for percentile calculation)
        self.metrics = {
            "total_batches": 0,
            "total_outputs": 0,
            "total_gpu_ms": 0.0,
            "total_route_ms": 0.0,
            "batch_sizes": deque(maxlen=500),
            "gpu_times_ms": deque(maxlen=500),
            "cycle_times_ms": deque(maxlen=500),
            "empty_cycles": 0,
            "vad_filtered_chunks": 0,
            "http_requests_total": 0,
            "http_requests_ok": 0,
            "http_requests_err": 0,
            "ws_connections_total": 0,
            "ws_connections_active": 0,
            "ws_connections_rejected": 0,
        }
        self.model_loaded = False

state = ASRServerState()

# === FastAPI App ===
app = FastAPI(
    title="Hindi ASR Production Server",
    description="Cache-aware streaming ASR for Hindi (Nemotron-0.6B fine-tuned)",
    version="1.0.0",
)

# ============================================================
# Model Loading
# ============================================================

def download_model_if_needed() -> str:
    """Download model from HuggingFace if not found locally. Returns local path."""
    if os.path.exists(MODEL_PATH):
        logger.info(f"Model found at {MODEL_PATH}")
        return MODEL_PATH

    logger.info(f"Model not found at {MODEL_PATH}, downloading from {HF_MODEL_REPO}...")
    from huggingface_hub import hf_hub_download
    hf_token = os.environ.get("HF_TOKEN")
    model_path = hf_hub_download(
        repo_id=HF_MODEL_REPO,
        filename=HF_MODEL_FILE,
        token=hf_token,
        cache_dir="/model",
    )
    logger.info(f"Model downloaded to {model_path}")
    return model_path

def build_pipeline_config(model_path: str):
    """Build OmegaConf config for NeMo pipeline"""
    from omegaconf import OmegaConf
    from dataclasses import dataclass, field
    from typing import Optional, List

    # Full config structure matching CacheAwarePipelineConfig from nemotron_asr
    @dataclass
    class BoostingTreeConfig:
        model_path: Optional[str] = None
        key_phrases_file: Optional[str] = None
        key_phrases_list: Optional[List[str]] = None
        source_lang: str = "en"

    @dataclass
    class GreedyDecodingConfig:
        use_cuda_graph_decoder: bool = False
        max_symbols: int = 10
        ngram_lm_model: Optional[str] = None
        ngram_lm_alpha: float = 0.0
        boosting_tree: BoostingTreeConfig = field(default_factory=BoostingTreeConfig)
        boosting_tree_alpha: float = 0.0

    @dataclass
    class DecodingConfig:
        strategy: str = "greedy_batch"
        preserve_alignments: bool = False
        fused_batch_size: int = -1
        greedy: GreedyDecodingConfig = field(default_factory=GreedyDecodingConfig)

    @dataclass
    class ASRConfig:
        model_name: str = model_path
        device: str = "cuda"
        device_id: int = 0
        compute_dtype: str = "bfloat16"
        use_amp: bool = True
        decoding: DecodingConfig = field(default_factory=DecodingConfig)

    @dataclass
    class ITNConfig:
        input_case: str = "lower_cased"
        whitelist: Optional[str] = None
        overwrite_cache: bool = False
        max_number_of_permutations_per_split: int = 729
        left_padding_size: int = 4
        batch_size: int = 32
        n_jobs: int = 16

    @dataclass
    class ConfidenceConfig:
        exclude_blank: bool = True
        aggregation: str = "mean"
        method_cfg: dict = field(default_factory=lambda: {
            "name": "entropy",
            "entropy_type": "tsallis",
            "alpha": 0.5,
            "entropy_norm": "exp",
        })

    @dataclass
    class EndpointingConfig:
        stop_history_eou: int = 800
        residue_tokens_at_end: int = 2

    @dataclass
    class StreamingConfig:
        sample_rate: int = 16000
        batch_size: int = 512
        word_boundary_tolerance: int = 4
        att_context_size: List[int] = field(default_factory=lambda: [70, 6])
        use_cache: bool = True
        use_feat_cache: bool = True
        chunk_size_in_secs: Optional[float] = None
        request_type: str = "frame"
        num_slots: int = 1024
        exhaustive_batching: bool = False
        batching_delay_secs: float = 0.300

    @dataclass
    class MetricsASRConfig:
        gt_text_attr_name: str = "text"
        clean_groundtruth_text: bool = False
        langid: str = "hi"
        use_cer: bool = True
        ignore_capitalization: bool = True
        ignore_punctuation: bool = True


Output:
Chunk ID: f4e5b3
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 859
Output:
"""
Vast.ai PyWorker for Hindi ASR serverless endpoint.
Proxies HTTP requests to ASR model server, reports metrics for autoscaling.
"""
import os
import struct
import base64

from vastai import (
    Worker,
    WorkerConfig,
    HandlerConfig,
    BenchmarkConfig,
    LogActionConfig,
)

MODEL_SERVER_URL = "http://127.0.0.1"
MODEL_SERVER_PORT = int(os.environ.get("SERVER_PORT", "8000"))
MODEL_LOG_FILE = os.environ.get("MODEL_LOG", "/var/log/model/server.log")

def asr_workload_calculator(payload):
    """Estimate workload from request. Returns estimated audio seconds."""
    if not isinstance(payload, dict):
        return 10.0
    for key in ("audio_seconds", "audio_duration", "duration"):
        if key in payload:
            try:
                return float(payload[key])
            except Exception:
                pass
    audio_b64 = payload.get("audio_base64") or payload.get("audio_b64") or ""
    if audio_b64:
        try:
            raw_len = len(audio_b64) * 3 / 4
            return max(1.0, raw_len / 32000.0)
        except Exception:
            pass
    return 10.0

def asr_benchmark_generator():
    """Generate benchmark payload: 1s silence WAV (base64 JSON)."""
    sample_rate = 16000
    num_samples = int(sample_rate * 1.0)
    silence = b"\x00\x00" * num_samples
    data_size = len(silence)
    wav_header = struct.pack(
        "<4sI4s4sIHHIIHH4sI",
        b"RIFF", 36 + data_size, b"WAVE", b"fmt ", 16,
        1, 1, sample_rate, sample_rate * 2, 2, 16,
        b"data", data_size,
    )
    wav_bytes = wav_header + silence
    return {
        "audio_base64": base64.b64encode(wav_bytes).decode("ascii"),
        "language": "hi",
    }

def metrics_workload_calculator(_payload):
    """Metrics requests should not affect autoscaling decisions."""
    return 0.1

worker_config = WorkerConfig(
    model_server_url=MODEL_SERVER_URL,
    model_server_port=MODEL_SERVER_PORT,
    model_log_file=MODEL_LOG_FILE,
    model_healthcheck_url=f"{MODEL_SERVER_URL}:{MODEL_SERVER_PORT}/health",
    handlers=[
        HandlerConfig(
            route="/v1/audio/transcriptions",
            allow_parallel_requests=True,
            max_queue_time=120.0,
            workload_calculator=asr_workload_calculator,
            benchmark_config=BenchmarkConfig(
                generator=asr_benchmark_generator,
                runs=4,
                concurrency=1,
            ),
        ),
        HandlerConfig(
            route="/v1/audio/transcriptions/json",
            allow_parallel_requests=True,
            max_queue_time=120.0,
            workload_calculator=asr_workload_calculator,
        ),
        HandlerConfig(
            route="/v1/audio/transcriptions/stream",
            allow_parallel_requests=True,
            max_queue_time=60.0,
            workload_calculator=asr_workload_calculator,
        ),
        HandlerConfig(
            route="/metrics",
            allow_parallel_requests=True,
            max_queue_time=5.0,
            workload_calculator=metrics_workload_calculator,
        ),
    ],
    log_action_config=LogActionConfig(
        on_load=["Application startup complete."],
        on_error=[
            "Traceback (most recent call last):",
            "RuntimeError:",
            "CUDA out of memory",
        ],
        on_info=[
            "Warming up GPU",
            "Pipeline loaded",
        ],
    ),
)

Worker(worker_config).run()


Output:
Chunk ID: 2b3315
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 3341
Output:
#!/usr/bin/env python3
"""
Production Stress Test for Hindi ASR - Simulates Real Customer Usage

Captures ALL metrics needed for scaling law decisions:
- Connection time (p50/p95/p99)
- First token latency (p50/p95/p99)
- End-to-end latency
- WER/CER (if reference text available)
- Throughput (streams * realtime)
- GPU utilization (via /metrics endpoint)
- Error rate
- Drift (latency accumulation over time)

Usage:
    python test_stress.py --url http://ip:8000 --mode ws --levels 1,10,50,100
    python test_stress.py --url http://ip:8000 --mode http --levels 1,5,10,20
    python test_stress.py --url http://ip:8000 --mode sse --levels 1,10,50
    python test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10
"""
import os, sys, json, time, math, struct, asyncio, argparse, csv, base64
from pathlib import Path
import requests
from urllib.parse import urlparse
from collections import defaultdict
from datetime import datetime
import aiohttp

def generate_test_audio(duration_s=10.0, freq_hz=440, sample_rate=16000):
    """Generate test audio: sine wave as raw PCM + WAV"""
    n = int(sample_rate * duration_s)
    pcm = struct.pack(f'<{n}h', *[
        min(32767, max(-32768, int(16000 * math.sin(2 * math.pi * freq_hz * i / sample_rate))))
        for i in range(n)
    ])
    hdr = struct.pack(
        '<4sI4s4sIHHIIHH4sI',
        b'RIFF', 36 + len(pcm), b'WAVE', b'fmt ', 16,
        1, 1, sample_rate, sample_rate * 2, 2, 16,
        b'data', len(pcm),
    )
    return pcm, hdr + pcm

VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"

def _env_or_dotenv(name: str):
    if os.environ.get(name):
        return os.environ.get(name)
    env_path = Path(__file__).resolve().parent.parent / ".env"
    if env_path.exists():
        for line in env_path.read_text().splitlines():
            if line.startswith(name + "="):
                return line.split("=", 1)[1].strip()
    return None

def get_endpoint_info(endpoint_name: str):
    account_key = _env_or_dotenv("VAST_KEY")
    if not account_key:
        raise RuntimeError("VAST_KEY not set; pass --endpoint-key")
    r = requests.get(
        f"{VAST_API}/endptjobs/",
        headers={"Accept": "application/json", "Authorization": f"Bearer {account_key}"},
        timeout=30,
    )
    data = r.json()
    results = data.get("results", []) if isinstance(data, dict) else data
    for ep in results or []:
        if ep.get("endpoint_name") == endpoint_name:
            return ep
    raise RuntimeError(f"endpoint '{endpoint_name}' not found")

def get_endpoint_key(endpoint_name: str, endpoint_key: str | None):
    if endpoint_key:
        return endpoint_key
    return get_endpoint_info(endpoint_name).get("api_key")

def get_endpoint_id(endpoint_name: str):
    return get_endpoint_info(endpoint_name).get("id")

def wrap_auth(route: dict, payload: dict):
    auth = {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"] if k in route}
    return {"auth_data": auth, "payload": payload}

def pct(arr, p):
    if not arr: return 0
    s = sorted(arr)
    return s[min(int(len(s) * p / 100), len(s) - 1)]

class StressMetrics:
    """Collects and reports all metrics for a stress test level"""
    def __init__(self, concurrency, mode, audio_duration_s):
        self.concurrency = concurrency
        self.mode = mode
        self.audio_duration_s = audio_duration_s
        self.start_time = None
        self.end_time = None
        # Per-stream metrics
        self.connect_times_ms = []
        self.first_token_times_ms = []
        self.total_times_ms = []
        self.messages_per_stream = []
        self.errors = []
        self.texts = []
        # Server metrics snapshots
        self.server_metrics_before = None
        self.server_metrics_after = None

    def summary(self):
        ok = self.concurrency - len(self.errors)
        wall_time = (self.end_time - self.start_time) if self.end_time else 0
        throughput_x = 0.0
        if wall_time > 0:
            throughput_x = (ok * self.audio_duration_s) / wall_time
        return {
            "concurrency": self.concurrency,
            "mode": self.mode,
            "ok": ok,
            "errors": len(self.errors),
            "wall_time_s": round(wall_time, 2),
            "connect_ms_p50": round(pct(self.connect_times_ms, 50), 1),
            "connect_ms_p95": round(pct(self.connect_times_ms, 95), 1),
            "connect_ms_p99": round(pct(self.connect_times_ms, 99), 1),
            "first_token_ms_p50": round(pct(self.first_token_times_ms, 50), 1),
            "first_token_ms_p95": round(pct(self.first_token_times_ms, 95), 1),
            "first_token_ms_p99": round(pct(self.first_token_times_ms, 99), 1),
            "total_ms_p50": round(pct(self.total_times_ms, 50), 1),
            "total_ms_p95": round(pct(self.total_times_ms, 95), 1),
            "total_ms_p99": round(pct(self.total_times_ms, 99), 1),
            "msgs_per_stream_avg": round(sum(self.messages_per_stream) / max(len(self.messages_per_stream), 1), 1),
            "throughput_x": round(throughput_x, 2),
        }

async def fetch_server_metrics(url=None, endpoint_name=None, endpoint_key=None, route_cost=10.0):
    """Fetch /metrics either directly or through serverless PyWorker."""
    if endpoint_name:
        try:
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    route = await r.json()
            if "url" not in route:
                return {}
            worker_url = route["url"].rstrip("/")
            req = wrap_auth(route, {})
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{worker_url}/metrics",
                    json=req,
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    if r.status == 200:
                        return await r.json()
        except Exception:
            return {}
        return {}

    if not url:
        return {}
    try:
        async with aiohttp.ClientSession() as s:
            async with s.get(f"{url}/metrics", timeout=aiohttp.ClientTimeout(total=5)) as r:
                if r.status == 200:
                    return await r.json()
    except Exception:
        pass
    return {}

async def fetch_endpoint_workers(endpoint_name, endpoint_key):
    """Fetch worker list for endpoint from Vast run API"""
    try:
        eid = get_endpoint_id(endpoint_name)
        async with aiohttp.ClientSession() as s:
            async with s.post(
                f"{VAST_RUN}/get_endpoint_workers/",
                json={"id": eid, "api_key": endpoint_key},
                timeout=aiohttp.ClientTimeout(total=10),
            ) as r:
                data = await r.json()
                if isinstance(data, dict) and "workers" in data:
                    return data["workers"]
                return data if isinstance(data, list) else []
    except Exception:
        return []

async def run_ws_stream(url, pcm_bytes, stream_idx, metrics, duration_s,
                        endpoint_name=None, endpoint_key=None, route_cost=10.0, ws_port=None):
    """Run a single WebSocket stream at real-time pace"""
    if endpoint_name:
        if not ws_port:
            metrics.errors.append({"stream": stream_idx, "error": "ws_port required for endpoint mode"})
            return
        try:
            async with aiohttp.ClientSession() as s:
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as r:
                    route = await r.json()
            if "url" not in route:
                metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
                return
            parsed = urlparse(route["url"])
            scheme = "wss" if parsed.scheme == "https" else "ws"
            ws_url = f"{scheme}://{parsed.hostname}:{ws_port}/ws"
        except Exception as e:
            metrics.errors.append({"stream": stream_idx, "error": str(e)})
            return
    else:
        ws_url = url.replace("http://", "ws://").replace("https://", "wss://") + "/ws"
    chunk_size = int(0.160 * 16000 * 2)  # 160ms chunks
    t_start = time.perf_counter()

    try:
        async with aiohttp.ClientSession() as s:
            async with s.ws_connect(ws_url, timeout=30) as ws:
                t_connected = time.perf_counter()
                metrics.connect_times_ms.append((t_connected - t_start) * 1000)

                # Wait for ready
                ready = await asyncio.wait_for(ws.receive(), timeout=10)

                first_token = None
                msgs = []

                async def send():
                    for i in range(0, len(pcm_bytes), chunk_size):
                        await ws.send_bytes(pcm_bytes[i:i + chunk_size])
                        await asyncio.sleep(0.160)  # Real-time pacing
                    await ws.send_str("END")

                async def recv():
                    nonlocal first_token
                    while True:
                        try:
                            m = await asyncio.wait_for(ws.receive(), timeout=30)
                            if m.type == aiohttp.WSMsgType.TEXT:
                                if m.data == "END":
                                    break
                                d = json.loads(m.data)
                                if first_token is None and d.get("text"):
                                    first_token = time.perf_counter()
                                msgs.append(d)
                            elif m.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING, aiohttp.WSMsgType.CLOSED):
                                break
                        except asyncio.TimeoutError:
                            break

                await asyncio.gather(send(), recv())

                t_done = time.perf_counter()
                if first_token:
                    metrics.first_token_times_ms.append((first_token - t_start) * 1000)
                metrics.total_times_ms.append((t_done - t_start) * 1000)
                metrics.messages_per_stream.append(len(msgs))
                if msgs:
                    last_text = ""
                    for m in reversed(msgs):
                        if m.get("text"):
                            last_text = m["text"]
                            break
                    metrics.texts.append(last_text)

    except Exception as e:
        metrics.errors.append({"stream": stream_idx, "error": str(e)})

async def run_http_batch(url, wav_bytes, stream_idx, metrics,
                         endpoint_name=None, endpoint_key=None, route_cost=10.0):
    """Run a single HTTP batch transcription"""
    t_start = time.perf_counter()
    try:
        async with aiohttp.ClientSession() as s:
            if endpoint_name:
                audio_b64 = base64.b64encode(wav_bytes).decode("ascii")
                # Get route
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as rr:
                    route = await rr.json()
                if "url" not in route:
                    metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
                    return
                worker_url = route["url"].rstrip("/")
                payload = {"audio_base64": audio_b64, "language": "hi"}
                req = wrap_auth(route, payload)
                async with s.post(
                    f"{worker_url}/v1/audio/transcriptions",
                    json=req,
                    timeout=aiohttp.ClientTimeout(total=120),
                ) as r:
                    t_done = time.perf_counter()
                    metrics.connect_times_ms.append(0)
                    if r.status == 200:
                        d = await r.json()
                        metrics.total_times_ms.append((t_done - t_start) * 1000)
                        metrics.first_token_times_ms.append((t_done - t_start) * 1000)
                        metrics.messages_per_stream.append(1)
                        metrics.texts.append(d.get("text", ""))
                    elif r.status in (404, 422):
                        # Fallback for older images
                        async with s.post(
                            f"{VAST_RUN}/route/",
                            json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                            timeout=aiohttp.ClientTimeout(total=10),
                        ) as rr2:
                            route2 = await rr2.json()
                        if "url" not in route2:
                            metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route2}"})


Output:
Chunk ID: acb056
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 1512
Output:
#!/usr/bin/env python3
"""
Vast.ai Serverless Deployment for Hindi ASR.
Usage: python deploy.py {create|status|workers|logs|route|destroy}
"""
import os, sys, json, argparse, requests
sys.path.insert(0, os.path.dirname(__file__))
try:
    from dotenv import load_dotenv
    load_dotenv(os.path.join(os.path.dirname(__file__), "..", ".env"))
except ImportError:
    pass

VAST_API_KEY = os.environ.get("VAST_KEY", "")
VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"
ENDPOINT_NAME = os.environ.get("VAST_ENDPOINT_NAME", "hindi-asr-prod")
DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v6-production")
TEMPLATE_HASH = os.environ.get("VAST_TEMPLATE_HASH", "")
LAUNCH_ARGS = os.environ.get("VAST_LAUNCH_ARGS", "-p 8000:8000 -p 18000:18000")

ENDPOINT_CONFIG = {
    "endpoint_name": ENDPOINT_NAME,
    "min_load": 1,
    "target_util": 0.8,
    "cold_mult": 2.0,
    "cold_workers": 1,
    "max_workers": 5,
}

SEARCH_PARAMS = {
    "verified": {"eq": True}, "rentable": {"eq": True},
    "rented": {"eq": False}, "gpu_ram": {"gte": 40},
    "num_gpus": {"eq": 1}, "reliability": {"gte": 0.95},
    "inet_down": {"gte": 200},
}

def headers():
    return {"Accept": "application/json", "Content-Type": "application/json",
            "Authorization": f"Bearer {VAST_API_KEY}"}

def get_endpoints():
    try:
        r = requests.get(f"{VAST_API}/endptjobs/", headers=headers())
        data = r.json()
        if isinstance(data, dict):
            return data.get("results", [])
        if isinstance(data, list):
            return data
    except Exception:
        pass
    return []

def get_endpoint_info():
    for ep in get_endpoints():
        if ep.get("endpoint_name") == ENDPOINT_NAME:
            return ep
    return None

def create_endpoint():
    print(f"Creating endpoint: {ENDPOINT_NAME}")
    r = requests.post(f"{VAST_API}/endptjobs/", headers=headers(), json=ENDPOINT_CONFIG)
    d = r.json()
    if d.get("success"):
        print(f"  Created: id={d['result']}")
        return d["result"]
    print(f"  Error: {d}")
    return get_endpoint_id()

def get_endpoint_id():
    ep = get_endpoint_info()
    return ep.get("id") if ep else None

def get_endpoint_key():
    ep = get_endpoint_info()
    return ep.get("api_key") if ep else None

def get_template_hash():
    if TEMPLATE_HASH:
        return TEMPLATE_HASH
    # Try to infer from existing workergroup
    try:
        r = requests.get(f"{VAST_API}/workergroups/", headers=headers())
        data = r.json()
        results = data.get("results", []) if isinstance(data, dict) else data
        for wg in results or []:
            if wg.get("endpoint_name") == ENDPOINT_NAME or wg.get("endpoint_id") == get_endpoint_id():
                return wg.get("template_hash") or ""
    except Exception:
        pass
    return ""

def create_workergroup(eid=None):
    eid = eid or get_endpoint_id()
    if not eid: print("No endpoint found"); return
    print(f"Creating workergroup for endpoint {eid}")
    payload = {
        "endpoint_id": eid,
        "gpu_ram": 40,
    }
    template_hash = get_template_hash()
    if template_hash:
        payload["template_hash"] = template_hash
        payload["search_params"] = ""
    else:
        payload["search_params"] = json.dumps(SEARCH_PARAMS)
    if LAUNCH_ARGS:
        payload["launch_args"] = LAUNCH_ARGS
    r = requests.post(f"{VAST_API}/workergroups/", headers=headers(), json=payload)
    d = r.json()
    print(f"  {'Created: id=' + str(d['result']) if d.get('success') else 'Error: ' + str(d)}")

def get_workers():
    eid = get_endpoint_id()
    ekey = get_endpoint_key()
    if not eid or not ekey:
        return []
    r = requests.post(f"{VAST_RUN}/get_endpoint_workers/",
        json={"id": eid, "api_key": ekey},
        headers={"Content-Type": "application/json"})
    try:
        d = r.json()
        return d if isinstance(d, list) else d.get("workers", [])
    except: return []

def get_route(cost=10.0):
    ekey = get_endpoint_key()
    if not ekey:
        return {"error": "endpoint api_key not found"}
    r = requests.post(f"{VAST_RUN}/route/",
        json={"endpoint": ENDPOINT_NAME, "api_key": ekey, "cost": cost},
        headers={"Content-Type": "application/json"})
    return r.json()

def show_status():
    eid = get_endpoint_id()
    print(f"Endpoint: {ENDPOINT_NAME} (id={eid})")
    if not eid: print("  NOT DEPLOYED"); return
    workers = get_workers()
    print(f"  Workers: {len(workers)}")
    for w in workers:
        print(f"    id={w.get('id')} status={w.get('status')} load={w.get('cur_load',0):.1f}")
    route = get_route()
    print(f"  Route: {route.get('url', route)}")

def destroy():
    eid = get_endpoint_id()
    if not eid: print("Nothing to destroy"); return
    print(f"Destroying endpoint {eid}")
    requests.put(f"{VAST_API}/endptjobs/{eid}/", headers=headers(), json={"status": "stopped"})
    requests.delete(f"{VAST_API}/endptjobs/{eid}/", headers=headers())
    print("  Done")

def main():
    if not VAST_API_KEY: print("ERROR: VAST_KEY not set"); sys.exit(1)
    p = argparse.ArgumentParser()
    p.add_argument("action", choices=["create","status","workers","logs","route","destroy"])
    a = p.parse_args()
    if a.action == "create":
        eid = create_endpoint()
        if eid: create_workergroup(eid)
    elif a.action == "status": show_status()
    elif a.action == "workers": print(json.dumps(get_workers(), indent=2))
    elif a.action == "logs":
        eid = get_endpoint_id()
        if eid:
            ekey = get_endpoint_key()
            r = requests.post(f"{VAST_RUN}/get_endpoint_logs/",
                json={"id": eid, "api_key": ekey, "tail": 2000},
                headers={"Content-Type": "application/json"})
            for k,v in r.json().items():
                if v: print(f"\n=== {k} ===\n{v}")
    elif a.action == "route": print(json.dumps(get_route(), indent=2))
    elif a.action == "destroy": destroy()

if __name__ == "__main__": main()


**Extending log inspection range**


I confirmed the current harness is solid but I need to inspect the remaining parts of `test_stress.py` and `server.py` to verify exactly how metrics snapshots, durations, and error accounting are implemented before I run extended sweeps.


sed -n '320,760p' production/test_stress.py


sed -n '1,280p' production/test_smoke.py


Output:
Chunk ID: 38f447
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 3894
Output:
                            metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route2}"})
                            return
                        worker_url = route2["url"].rstrip("/")
                        req = wrap_auth(route2, payload)
                        async with s.post(
                            f"{worker_url}/v1/audio/transcriptions/json",
                            json=req,
                            timeout=aiohttp.ClientTimeout(total=120),
                        ) as r2:
                            t_done = time.perf_counter()
                            if r2.status == 200:
                                d = await r2.json()
                                metrics.total_times_ms.append((t_done - t_start) * 1000)
                                metrics.first_token_times_ms.append((t_done - t_start) * 1000)
                                metrics.messages_per_stream.append(1)
                                metrics.texts.append(d.get("text", ""))
                            else:
                                metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r2.status}"})
                    else:
                        metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r.status}"})
                return

            form = aiohttp.FormData()
            form.add_field('file', wav_bytes, filename='test.wav', content_type='audio/wav')
            form.add_field('language', 'hi')

            async with s.post(
                f"{url}/v1/audio/transcriptions",
                data=form,
                timeout=aiohttp.ClientTimeout(total=120),
            ) as r:
                t_done = time.perf_counter()
                metrics.connect_times_ms.append(0)  # N/A for HTTP
                if r.status == 200:
                    d = await r.json()
                    metrics.total_times_ms.append((t_done - t_start) * 1000)
                    metrics.first_token_times_ms.append((t_done - t_start) * 1000)
                    metrics.messages_per_stream.append(1)
                    metrics.texts.append(d.get("text", ""))
                else:
                    metrics.errors.append({"stream": stream_idx, "error": f"HTTP {r.status}"})
    except Exception as e:
        metrics.errors.append({"stream": stream_idx, "error": str(e)})

async def run_sse_stream(url, wav_bytes, stream_idx, metrics,
                         endpoint_name=None, endpoint_key=None, route_cost=10.0):
    """Run a single HTTP SSE stream"""
    t_start = time.perf_counter()
    try:
        async with aiohttp.ClientSession() as s:
            if endpoint_name:
                audio_b64 = base64.b64encode(wav_bytes).decode("ascii")
                async with s.post(
                    f"{VAST_RUN}/route/",
                    json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": route_cost},
                    timeout=aiohttp.ClientTimeout(total=10),
                ) as rr:
                    route = await rr.json()
                if "url" not in route:
                    metrics.errors.append({"stream": stream_idx, "error": f"route failed: {route}"})
                    return
                worker_url = route["url"].rstrip("/")
                payload = {"audio_base64": audio_b64, "language": "hi", "chunk_duration_ms": 160}
                req = wrap_auth(route, payload)
                async with s.post(
                    f"{worker_url}/v1/audio/transcriptions/stream",
                    json=req,
                    timeout=aiohttp.ClientTimeout(total=120),
                ) as r:
                    if r.status != 200:
                        metrics.errors.append({"stream": stream_idx, "error": f"SSE {r.status}"})
                        return
                    first_token = None
                    msgs = 0
                    async for line in r.content:
                        l = line.decode().strip()
                        if l.startswith("data: "):
                            d = json.loads(l[6:])
                            if first_token is None and d.get("text"):
                                first_token = time.perf_counter()
                            msgs += 1
                            if d.get("done"):
                                break
                    t_done = time.perf_counter()
                    if first_token:
                        metrics.first_token_times_ms.append((first_token - t_start) * 1000)
                    metrics.total_times_ms.append((t_done - t_start) * 1000)
                    metrics.messages_per_stream.append(msgs)
                return

            form = aiohttp.FormData()
            form.add_field('file', wav_bytes, filename='test.wav', content_type='audio/wav')
            async with s.post(
                f"{url}/v1/audio/transcriptions/stream",
                data=form,
                timeout=aiohttp.ClientTimeout(total=120),
            ) as r:
                if r.status != 200:
                    metrics.errors.append({"stream": stream_idx, "error": f"SSE {r.status}"})
                    return
                first_token = None
                msgs = 0
                async for line in r.content:
                    l = line.decode().strip()
                    if l.startswith("data: "):
                        d = json.loads(l[6:])
                        if first_token is None and d.get("text"):
                            first_token = time.perf_counter()
                        msgs += 1
                        if d.get("done"):
                            break
                t_done = time.perf_counter()
                if first_token:
                    metrics.first_token_times_ms.append((first_token - t_start) * 1000)
                metrics.total_times_ms.append((t_done - t_start) * 1000)
                metrics.messages_per_stream.append(msgs)
    except Exception as e:
        metrics.errors.append({"stream": stream_idx, "error": str(e)})

async def run_stress_level(url, concurrency, mode, audio_duration_s=10.0,
                           endpoint_name=None, endpoint_key=None, route_cost=10.0, ws_port=None):
    """Run a single stress test level"""
    print(f"\n{'='*60}")
    print(f"  Stress Level: c={concurrency} mode={mode} audio={audio_duration_s}s")
    print(f"{'='*60}")

    pcm, wav = generate_test_audio(duration_s=audio_duration_s)
    metrics = StressMetrics(concurrency, mode, audio_duration_s)

    # Capture server metrics before
    if endpoint_name:
        metrics.server_metrics_before = await fetch_server_metrics(
            endpoint_name=endpoint_name,
            endpoint_key=endpoint_key,
            route_cost=route_cost,
        )
    else:
        metrics.server_metrics_before = await fetch_server_metrics(url=url)

    metrics.start_time = time.perf_counter()

    # Launch concurrent streams
    tasks = []
    for i in range(concurrency):
        if mode == "ws":
            tasks.append(run_ws_stream(url, pcm, i, metrics, audio_duration_s,
                                       endpoint_name, endpoint_key, route_cost, ws_port))
        elif mode == "http":
            tasks.append(run_http_batch(url, wav, i, metrics, endpoint_name, endpoint_key, route_cost))
        elif mode == "sse":
            tasks.append(run_sse_stream(url, wav, i, metrics, endpoint_name, endpoint_key, route_cost))

    await asyncio.gather(*tasks)
    metrics.end_time = time.perf_counter()

    # Brief pause then capture server metrics after
    await asyncio.sleep(1)
    if endpoint_name:
        metrics.server_metrics_after = await fetch_server_metrics(
            endpoint_name=endpoint_name,
            endpoint_key=endpoint_key,
            route_cost=route_cost,
        )
    else:
        metrics.server_metrics_after = await fetch_server_metrics(url=url)

    # Print results
    s = metrics.summary()
    print(f"\n  Results:")
    print(f"    OK/Total: {s['ok']}/{s['concurrency']}  Errors: {s['errors']}")
    print(f"    Wall time: {s['wall_time_s']}s")
    print(f"    Connect:     p50={s['connect_ms_p50']}ms  p95={s['connect_ms_p95']}ms  p99={s['connect_ms_p99']}ms")
    print(f"    First token: p50={s['first_token_ms_p50']}ms  p95={s['first_token_ms_p95']}ms  p99={s['first_token_ms_p99']}ms")
    print(f"    Total:       p50={s['total_ms_p50']}ms  p95={s['total_ms_p95']}ms  p99={s['total_ms_p99']}ms")
    print(f"    Msgs/stream: {s['msgs_per_stream_avg']}")
    print(f"    Throughput:  {s['throughput_x']}x realtime")

    if metrics.server_metrics_after:
        sm = metrics.server_metrics_after
        if endpoint_name:
            if isinstance(sm, dict):
                inf = sm.get("inference", {})
                print(f"    Server GPU:  p50={inf.get('gpu_ms_p50',0)}ms p99={inf.get('gpu_ms_p99',0)}ms")
                print(f"    Server VRAM: {sm.get('vram_used_gb',0)}GB/{sm.get('vram_total_gb',0)}GB")
                http = sm.get("http", {})
                print(f"    HTTP:        total={http.get('requests_total',0)} ok={http.get('requests_ok',0)} err={http.get('requests_err',0)}")
            elif isinstance(sm, list):
                for w in sm:
                    print(f"    Worker {w.get('id')} status={w.get('status')} cur_load={w.get('cur_load',0):.1f} cur_perf={w.get('cur_perf',0):.1f}")
        else:
            inf = sm.get("inference", {})
            print(f"    Server GPU:  p50={inf.get('gpu_ms_p50',0)}ms p99={inf.get('gpu_ms_p99',0)}ms")
            print(f"    Server VRAM: {sm.get('vram_used_gb',0)}GB/{sm.get('vram_total_gb',0)}GB")

    if metrics.errors:
        print(f"    Errors: {metrics.errors[:5]}")

    return metrics

async def main():
    p = argparse.ArgumentParser(description="ASR Production Stress Test")
    p.add_argument("--url", default=None, help="Server URL (http://ip:port)")
    p.add_argument("--endpoint", default=None, help="Vast endpoint name (uses /route + auth)")
    p.add_argument("--endpoint-key", default=None, help="Endpoint API key (optional if VAST_KEY set)")
    p.add_argument("--route-cost", type=float, default=10.0)
    p.add_argument("--ws-port", type=int, default=None, help="Direct WS port on worker (if exposed)")
    p.add_argument("--mode", default="ws", choices=["ws", "http", "sse", "both"])
    p.add_argument("--levels", default="1,5,10,25,50,75,100", help="Comma-separated concurrency levels")
    p.add_argument("--audio-duration", type=float, default=10.0, help="Audio duration per stream (seconds)")
    p.add_argument("--output", default=None, help="Output CSV file for results")
    p.add_argument("--cooldown", type=float, default=5.0, help="Seconds between levels")
    a = p.parse_args()

    if not a.url and not a.endpoint:
        print("ERROR: provide --url or --endpoint")
        sys.exit(2)
    url = a.url.rstrip("/") if a.url else ""
    levels = [int(x) for x in a.levels.split(",")]
    modes = [a.mode] if a.mode != "both" else ["ws", "http"]

    endpoint_key = get_endpoint_key(a.endpoint, a.endpoint_key) if a.endpoint else None

    print(f"=== Hindi ASR Stress Test ===")
    print(f"URL: {url if url else 'via endpoint'}")
    if a.endpoint:
        print(f"Endpoint: {a.endpoint}")
    print(f"Modes: {modes}")
    print(f"Levels: {levels}")
    print(f"Audio: {a.audio_duration}s per stream")

    # Verify server is up
    if a.endpoint:
        workers = await fetch_endpoint_workers(a.endpoint, endpoint_key)
        if not workers:
            print("Serverless endpoint has no active workers")
            sys.exit(1)
        print(f"Endpoint ready: workers={len(workers)}")
    else:
        health = await fetch_server_metrics(url=url)
        if not health:
            # Try health endpoint directly
            try:
                async with aiohttp.ClientSession() as s:
                    async with s.get(f"{url}/health", timeout=aiohttp.ClientTimeout(total=5)) as r:
                        if r.status != 200:
                            print(f"Server not ready: {r.status}")
                            sys.exit(1)
                        print(f"Server ready: {await r.json()}")
            except Exception as e:
                print(f"Cannot reach server: {e}")
                sys.exit(1)

    all_results = []

    for mode in modes:
        for level in levels:
            metrics = await run_stress_level(
                url,
                level,
                mode,
                a.audio_duration,
                a.endpoint,
                endpoint_key,
                a.route_cost,
                a.ws_port,
            )
            all_results.append({"level": level, "mode": mode, "metrics": metrics})

            # Cooldown between levels
            if level != levels[-1]:
                print(f"\n  Cooldown {a.cooldown}s...")
                await asyncio.sleep(a.cooldown)

    # Write CSV output
    output_file = a.output or f"stress_results_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv"
    output_path = os.path.join(os.path.dirname(__file__), "..", "benchmark_results", output_file)
    os.makedirs(os.path.dirname(output_path), exist_ok=True)

    with open(output_path, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=[
            "mode", "concurrency", "ok", "errors", "wall_time_s",
            "connect_p50", "connect_p95", "connect_p99",
            "first_token_p50", "first_token_p95", "first_token_p99",
            "total_p50", "total_p95", "total_p99",
            "msgs_avg", "throughput_x",
        ])
        writer.writeheader()
        for r in all_results:
            s = r["metrics"].summary()
            writer.writerow({
                "mode": s["mode"], "concurrency": s["concurrency"],
                "ok": s["ok"], "errors": s["errors"], "wall_time_s": s["wall_time_s"],
                "connect_p50": s["connect_ms_p50"], "connect_p95": s["connect_ms_p95"], "connect_p99": s["connect_ms_p99"],
                "first_token_p50": s["first_token_ms_p50"], "first_token_p95": s["first_token_ms_p95"], "first_token_p99": s["first_token_ms_p99"],
                "total_p50": s["total_ms_p50"], "total_p95": s["total_ms_p95"], "total_p99": s["total_ms_p99"],
                "msgs_avg": s["msgs_per_stream_avg"], "throughput_x": s["throughput_x"],
            })
    print(f"\nResults saved to: {output_path}")

    # Also save full JSON with server metrics
    json_path = output_path.replace(".csv", ".json")
    with open(json_path, "w") as f:
        json.dump([{
            "level": r["level"], "mode": r["mode"],
            "summary": r["metrics"].summary(),
            "server_before": r["metrics"].server_metrics_before,
            "server_after": r["metrics"].server_metrics_after,
            "errors": r["metrics"].errors,
        } for r in all_results], f, indent=2)
    print(f"Full metrics saved to: {json_path}")

    # Print summary table
    print(f"\n{'='*80}")
    print(f"  STRESS TEST SUMMARY")
    print(f"{'='*80}")
    print(f"{'Mode':>6} {'Conc':>5} {'OK':>4} {'Err':>4} {'FT p50':>8} {'FT p99':>8} {'Tot p50':>8} {'Msgs':>6}")
    print(f"{'-'*6:>6} {'-'*5:>5} {'-'*4:>4} {'-'*4:>4} {'-'*8:>8} {'-'*8:>8} {'-'*8:>8} {'-'*6:>6}")
    for r in all_results:
        s = r["metrics"].summary()
        print(f"{s['mode']:>6} {s['concurrency']:>5} {s['ok']:>4} {s['errors']:>4} "
              f"{s['first_token_ms_p50']:>7.0f}ms {s['first_token_ms_p99']:>7.0f}ms "
              f"{s['total_ms_p50']:>7.0f}ms {s['msgs_per_stream_avg']:>5.1f}")

if __name__ == "__main__":
    asyncio.run(main())