Output:
Chunk ID: a9a31e
Wall time: 1.0017 seconds
Process running with session ID 37278
Original token count: 321
Output:
The push refers to repository [docker.io/bharathkumar192/hindi-nemotron-asr]
8cd11509043e: Preparing
9cd2f3720187: Preparing
9cd2f3720187: Preparing
91aefb6018d2: Preparing
c31f7f4d2988: Preparing
3c45e67d2f27: Preparing
8c394962695c: Preparing
d3e85491259d: Preparing
891287ea98fb: Preparing
0e5872be6ca2: Preparing
145b1502b83c: Preparing
a0200b718862: Preparing
2c753ce5f8e2: Preparing
027c387a337f: Preparing
d995e7230cef: Preparing
84d22ebba73e: Preparing
96375e867c78: Preparing
18c3fae4e9c4: Preparing
f72f00af3d11: Preparing
cc9ee102691b: Preparing
f0caebf6d886: Preparing
782979f338e9: Preparing
2573e0d81582: Preparing
8c394962695c: Waiting
d3e85491259d: Waiting
891287ea98fb: Waiting
2c753ce5f8e2: Waiting
0e5872be6ca2: Waiting
d995e7230cef: Waiting
027c387a337f: Waiting
84d22ebba73e: Waiting
96375e867c78: Waiting
18c3fae4e9c4: Waiting
145b1502b83c: Waiting
a0200b718862: Waiting
f0caebf6d886: Waiting
782979f338e9: Waiting
cc9ee102691b: Waiting
f72f00af3d11: Waiting
2573e0d81582: Waiting
3c45e67d2f27: Layer already exists
8c394962695c: Layer already exists
d3e85491259d: Layer already exists
891287ea98fb: Layer already exists
0e5872be6ca2: Layer already exists
145b1502b83c: Layer already exists
a0200b718862: Layer already exists
2c753ce5f8e2: Layer already exists


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Output:
Chunk ID: 93ac97
Wall time: 1.5709 seconds
Process exited with code 0
Original token count: 135
Output:
027c387a337f: Layer already exists
d995e7230cef: Layer already exists
84d22ebba73e: Layer already exists
91aefb6018d2: Pushed
8cd11509043e: Pushed
96375e867c78: Layer already exists
9cd2f3720187: Pushed
18c3fae4e9c4: Layer already exists
f72f00af3d11: Layer already exists
cc9ee102691b: Layer already exists
f0caebf6d886: Layer already exists
c31f7f4d2988: Pushed
782979f338e9: Layer already exists
2573e0d81582: Layer already exists
v8-production: digest: sha256:278ed40975b099fd96b66e10a1d7e377e7e3edb79a069e6ac00223ff16e67ee6 size: 5142


Summary: auto


**Updating template to v8 image**


bash -lc 'set -a; source .env; vastai update template 861cc20c3590d38c81c0cc8ce286d431 --image "bharathkumar192/hindi-nemotron-asr:v8-production" --env "-p 8000:8000" --onstart-cmd "/app/onstart.sh" --search_params "verified==True rentable==True rented==False external==False gpu_ram>=40 num_gpus==1 reliability>0.95 direct_port_count>=1 disk_space>=30 inet_down>200" --api-key "$VAST_KEY"'


Output:
Chunk ID: 0c0901
Wall time: 0.3307 seconds
Process exited with code 0
Original token count: 222
Output:
updated template: {
 "hash_id": "4f9691e7d1c7f9c0332766afff14709b",
 "name": null,
 "image": "bharathkumar192/hindi-nemotron-asr:v8-production",
 "tag": null,
 "href": null,
 "repo": null,
 "env": "-p 8000:8000",
 "onstart": "/app/onstart.sh",
 "jup_direct": false,
 "ssh_direct": false,
 "use_jupyter_lab": false,
 "runtype": "args",
 "use_ssh": false,
 "jupyter_dir": null,
 "docker_login_repo": null,
 "extra_filters": "{\"verified\": {\"eq\": true}, \"external\": {\"eq\": false}, \"rentable\": {\"eq\": true}, \"rented\": {\"eq\": false}, \"gpu_ram\": {\"gte\": 40000.0}, \"num_gpus\": {\"eq\": \"1\"}, \"reliability\": {\"gt\": \"0.95\"}, \"direct_port_count\": {\"gte\": \"1\"}, \"disk_space\": {\"gte\": \"30\"}, \"inet_down\": {\"gt\": \"200\"}}",
 "recommended_disk_space": null,
 "readme_visible": true,
 "desc": null,
 "private": true,
 "readme_hash": null,
 "id": 341822
}


**Updating workergroup template hash**


bash -lc 'set -a; source .env; vastai update workergroup 16780 --template_hash 4f9691e7d1c7f9c0332766afff14709b --endpoint_id 11967 --api-key "$VAST_KEY"'


Output:
Chunk ID: 165e6f
Wall time: 0.3447 seconds
Process exited with code 0
Original token count: 10
Output:
workergroup update {'success': True}


**Recycling worker for new image**


bash -lc 'set -a; source .env; vastai destroy instance 31135468 --api-key "$VAST_KEY"'


Output:
Chunk ID: 1a653f
Wall time: 1.0006 seconds
Process running with session ID 70399
Original token count: 8
Output:
destroying instance 31135468.


**Waiting for worker idle state**


bash -lc 'set -a; source .env; for i in {1..20}; do python3 production/deploy.py workers; sleep 10; done'


Output:
Chunk ID: 3b0592
Wall time: 1.0016 seconds
Process running with session ID 41817
Original token count: 122
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0005405520398296715,
    "cur_perf": 0.030319856360513254,
    "disk_usage": 4.7655792236328125,
    "dlperf": 283.32661709420836,
    "id": 31135468,
    "loaded_at": 1770667933.9004111,
    "measured_perf": 61.754168987164974,
    "new_load": 0.0,
    "outdated": false,
    "perf": 61.75416946411133,
    "reliability": 0.9999999990281263,
    "reqs_working": 0,
    "started_at": 1770667868.1913676,
    "status": "idle"
  }
]


**Waiting for worker cleanup**

The old worker might not have fully destroyed yet, so I'll keep polling to see updated session output before proceeding.


Output:
Chunk ID: a78c1b
Wall time: 5.0011 seconds
Process running with session ID 41817
Original token count: 0
Output:


Output:
Chunk ID: af9536
Wall time: 5.0006 seconds
Process running with session ID 41817
Original token count: 103
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 0.0,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": true,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "creating"
  }
]


**Tracking new worker initialization**


Output:
Chunk ID: 5dedf9
Wall time: 5.0011 seconds
Process running with session ID 41817
Original token count: 103
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 0.0,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": true,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "creating"
  }
]


**Waiting for model readiness**


Output:
Chunk ID: b20741
Wall time: 5.0016 seconds
Process running with session ID 41817
Original token count: 0
Output:


Output:
Chunk ID: 561af7
Wall time: 5.0012 seconds
Process running with session ID 41817
Original token count: 103
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 0.0,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": true,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "loading"
  }
]


**Waiting on loading status**


Output:
Chunk ID: eb4f9f
Wall time: 5.0008 seconds
Process running with session ID 41817
Original token count: 107
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 6.644020080566406,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "loading"
  }
]


**Waiting for model loading idle**


Output:
Chunk ID: 5b9bec
Wall time: 5.0003 seconds
Process running with session ID 41817
Original token count: 108
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644580841064453,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "model_loading"
  }
]


**Waiting for idle state**


Output:
Chunk ID: 5bf5d1
Wall time: 5.0006 seconds
Process running with session ID 41817
Original token count: 0
Output:


Output:
Chunk ID: 2cd0c2
Wall time: 5.0005 seconds
Process running with session ID 41817
Original token count: 217
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644573211669922,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 0.0,
    "measured_perf": 62.906097412109375,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.906097412109375,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "model_loading"
  }
]
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.6448211669921875,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 1770668354.6391602,
    "measured_perf": 31.57409409836805,
    "new_load": 0.0,
    "outdated": false,
    "perf": 31.574094772338867,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "idle"
  }
]


**Checking metrics endpoint**

I'm inspecting the /metrics route to verify it returns CUDA-related fields and possibly VRAM details.


bash -lc 'set -a; source .env; python3 - << "PY"
import os, requests
VAST_API = "https://console.vast.ai/api/v0"
VAST_RUN = "https://run.vast.ai"
endpoint_name = "hindi-asr-prod"
api_key = os.environ["VAST_KEY"]
rows = requests.get(f"{VAST_API}/endptjobs/", headers={"Accept":"application/json","Authorization":f"Bearer {api_key}"}).json()
if isinstance(rows, dict):
    rows = rows.get("results", [])
end = next(ep for ep in rows if ep.get("endpoint_name") == endpoint_name)
endpoint_key = end["api_key"]
route = requests.post(f"{VAST_RUN}/route/", json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": 1}).json()
worker_url = route["url"].rstrip("/")
req = {"auth_data": {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"]}, "payload": {}}
resp = requests.post(f"{worker_url}/metrics", json=req, timeout=20)
print(resp.status_code)
print(resp.text[:800])
PY'


Output:
Chunk ID: 2e4521
Wall time: 0.7150 seconds
Process exited with code 0
Original token count: 134
Output:
200
{"active_streams":0,"max_streams":100,"inference":{"total_batches":0,"total_outputs":0,"empty_cycles":0,"vad_filtered_chunks":0,"gpu_ms_p50":0,"gpu_ms_p99":0,"cycle_ms_p50":0,"cycle_ms_p99":0,"batch_size_avg":0.0},"http":{"requests_total":5,"requests_ok":5,"requests_err":0},"ws":{"connections_total":0,"connections_active":0,"connections_rejected":0},"cuda_available":true,"cuda_device_count":1,"gpu_name":"NVIDIA RTX PRO 6000 Blackwell Workstation Edition","vram_used_gb":0,"vram_reserved_gb":0,"vram_total_gb":0,"uptime_s":27.0}


Summary: auto


**Identifying attribute typo causing VRAM zero**

The code incorrectly uses `.total_mem` instead of `.total_memory` from `torch.cuda.get_device_properties`, causing the VRAM total to show zero due to fallback after an AttributeError. Both health and metrics code should be fixed to use `.total_memory`.


**Planning patch for VRAM attribute typo**

I'm searching all occurrences of `total_mem` to replace with `total_memory` in health and metrics code for accurate VRAM reporting.


rg -n "total_mem" -S production/server.py


Output:
Chunk ID: b29dca
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 59
Output:
503:        vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9
549:            vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9
1177:        vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9


**Planning memory field update**


Input:
*** Begin Patch
*** Update File: production/server.py
@@
-        vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9
+        vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
@@
-            vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9
+            vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
@@
-        vram_total = torch.cuda.get_device_properties(0).total_mem / 1e9
+        vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
*** End Patch


Output: {"output":"Success. Updated the following files:\nM production/server.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Rebuilding image for correct metrics**


docker build -f production/Dockerfile.slim -t bharathkumar192/hindi-nemotron-asr:v9-production .


Output:
Chunk ID: 02776c
Wall time: 0.5069 seconds
Process exited with code 0
Original token count: 345
Output:
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile.slim
#1 transferring dockerfile: 520B done
#1 WARN: UndefinedVar: Usage of undefined variable '$PYTHONPATH' (line 7)
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/bharathkumar192/hindi-nemotron-asr:v5-vram
#2 DONE 0.0s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/7] FROM docker.io/bharathkumar192/hindi-nemotron-asr:v5-vram
#4 DONE 0.0s

#5 [internal] load build context
#5 transferring context: 45.83kB done
#5 DONE 0.0s

#6 [2/7] RUN pip install --no-cache-dir soundfile resampy vastai_sdk "python-dateutil>=2.8.2" 2>/dev/null || true
#6 CACHED

#7 [3/7] COPY production/server.py /app/server.py
#7 DONE 0.0s

#8 [4/7] COPY production/worker.py /app/worker.py
#8 DONE 0.0s

#9 [5/7] COPY production/onstart.sh /app/onstart.sh
#9 DONE 0.0s

#10 [6/7] RUN chmod +x /app/onstart.sh
#10 DONE 0.2s

#11 [7/7] RUN mkdir -p /var/log/model
#11 DONE 0.2s

#12 exporting to image
#12 exporting layers 0.0s done
#12 writing image sha256:901957c0f15747545fe11317cf451600e31290a2320897cc6c3bf3f8ef5c235e done
#12 naming to docker.io/bharathkumar192/hindi-nemotron-asr:v9-production done
#12 DONE 0.0s

 1 warning found (use docker --debug to expand):
 - UndefinedVar: Usage of undefined variable '$PYTHONPATH' (line 7)


**Pushing new version**


Output:
Chunk ID: d094af
Wall time: 1.0010 seconds
Process running with session ID 49384
Original token count: 321
Output:
The push refers to repository [docker.io/bharathkumar192/hindi-nemotron-asr]
d0e820a743f9: Preparing
146e3af8b310: Preparing
146e3af8b310: Preparing
f3eddf1ef0f9: Preparing
1897f030d037: Preparing
3c45e67d2f27: Preparing
8c394962695c: Preparing
d3e85491259d: Preparing
891287ea98fb: Preparing
0e5872be6ca2: Preparing
145b1502b83c: Preparing
a0200b718862: Preparing
2c753ce5f8e2: Preparing
027c387a337f: Preparing
d995e7230cef: Preparing
84d22ebba73e: Preparing
96375e867c78: Preparing
18c3fae4e9c4: Preparing
f72f00af3d11: Preparing
891287ea98fb: Waiting
cc9ee102691b: Preparing
145b1502b83c: Waiting
f0caebf6d886: Preparing
782979f338e9: Preparing
2573e0d81582: Preparing
0e5872be6ca2: Waiting
8c394962695c: Waiting
d3e85491259d: Waiting
a0200b718862: Waiting
18c3fae4e9c4: Waiting
2c753ce5f8e2: Waiting
f0caebf6d886: Waiting
84d22ebba73e: Waiting
f72f00af3d11: Waiting
027c387a337f: Waiting
96375e867c78: Waiting
cc9ee102691b: Waiting
782979f338e9: Waiting
d995e7230cef: Waiting
2573e0d81582: Waiting
3c45e67d2f27: Layer already exists
8c394962695c: Layer already exists
d3e85491259d: Layer already exists
891287ea98fb: Layer already exists
0e5872be6ca2: Layer already exists
145b1502b83c: Layer already exists
a0200b718862: Layer already exists
2c753ce5f8e2: Layer already exists


Output:
Chunk ID: 32bb30
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 135
Output:
027c387a337f: Layer already exists
d995e7230cef: Layer already exists
84d22ebba73e: Layer already exists
146e3af8b310: Pushed
d0e820a743f9: Pushed
96375e867c78: Layer already exists
1897f030d037: Pushed
f3eddf1ef0f9: Pushed
18c3fae4e9c4: Layer already exists
cc9ee102691b: Layer already exists
f72f00af3d11: Layer already exists
f0caebf6d886: Layer already exists
782979f338e9: Layer already exists
2573e0d81582: Layer already exists
v9-production: digest: sha256:521e2b306c07a7685a05eb00866763e33efa547cc632dbc63f13d263bc2e3146 size: 5142


**Updating template to v9**


bash -lc 'set -a; source .env; vastai update template 4f9691e7d1c7f9c0332766afff14709b --image "bharathkumar192/hindi-nemotron-asr:v9-production" --env "-p 8000:8000" --onstart-cmd "/app/onstart.sh" --search_params "verified==True rentable==True rented==False external==False gpu_ram>=40 num_gpus==1 reliability>0.95 direct_port_count>=1 disk_space>=30 inet_down>200" --api-key "$VAST_KEY"'


Output:
Chunk ID: b9f94c
Wall time: 0.5001 seconds
Process exited with code 0
Original token count: 222
Output:
updated template: {
 "hash_id": "118364eef727b32735fa02ee2908b7a1",
 "name": null,
 "image": "bharathkumar192/hindi-nemotron-asr:v9-production",
 "tag": null,
 "href": null,
 "repo": null,
 "env": "-p 8000:8000",
 "onstart": "/app/onstart.sh",
 "jup_direct": false,
 "ssh_direct": false,
 "use_jupyter_lab": false,
 "runtype": "args",
 "use_ssh": false,
 "jupyter_dir": null,
 "docker_login_repo": null,
 "extra_filters": "{\"verified\": {\"eq\": true}, \"external\": {\"eq\": false}, \"rentable\": {\"eq\": true}, \"rented\": {\"eq\": false}, \"gpu_ram\": {\"gte\": 40000.0}, \"num_gpus\": {\"eq\": \"1\"}, \"reliability\": {\"gt\": \"0.95\"}, \"direct_port_count\": {\"gte\": \"1\"}, \"disk_space\": {\"gte\": \"30\"}, \"inet_down\": {\"gt\": \"200\"}}",
 "recommended_disk_space": null,
 "readme_visible": true,
 "desc": null,
 "private": true,
 "readme_hash": null,
 "id": 341822
}


**Updating workergroup hash**


bash -lc 'set -a; source .env; vastai update workergroup 16780 --template_hash 118364eef727b32735fa02ee2908b7a1 --endpoint_id 11967 --api-key "$VAST_KEY"'


Output:
Chunk ID: ed89c6
Wall time: 0.3040 seconds
Process exited with code 0
Original token count: 10
Output:
workergroup update {'success': True}


**Destroying current worker instance**


bash -lc 'set -a; source .env; vastai destroy instance 31135685 --api-key "$VAST_KEY"'


Output:
Chunk ID: 419cc6
Wall time: 0.5424 seconds
Process exited with code 0
Original token count: 8
Output:
destroying instance 31135685.


**Waiting for new worker idle**


Output:
Chunk ID: 292dc1
Wall time: 1.0017 seconds
Process running with session ID 38705
Original token count: 122
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.006763797713952795,
    "cur_perf": 0.013416005849389605,
    "disk_usage": 4.6448211669921875,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 1770668354.6391602,
    "measured_perf": 31.57409409836805,
    "new_load": 0.0,
    "outdated": false,
    "perf": 31.519132614135742,
    "reliability": 0.9995645438151909,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "idle"
  }
]


**Waiting for worker update**


Output:
Chunk ID: 1e2521
Wall time: 5.0007 seconds
Process running with session ID 38705
Original token count: 0
Output:


Output:
Chunk ID: f78ea3
Wall time: 5.0011 seconds
Process running with session ID 38705
Original token count: 122
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0038472487638356555,
    "cur_perf": 0.013416005849389605,
    "disk_usage": 4.6448211669921875,
    "dlperf": 282.5863918716072,
    "id": 31135685,
    "loaded_at": 1770668354.6391602,
    "measured_perf": 31.57409409836805,
    "new_load": 0.0,
    "outdated": false,
    "perf": 31.556838989257812,
    "reliability": 0.9998633492350738,
    "reqs_working": 0,
    "started_at": 1770668288.1913676,
    "status": "idle"
  }
]


**Noticing instance persistence**

The destroy method may not have immediately removed the instance, as it still appears present; this might indicate a delay or failure in destruction. I'll keep monitoring to see if the instance transitions to a new ID.


Output:
Chunk ID: 7d881c
Wall time: 5.0010 seconds
Process running with session ID 38705
Original token count: 103
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 0.0,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 0.0,
    "measured_perf": 62.24946212768555,
    "new_load": 0.0,
    "outdated": true,
    "perf": 62.24946212768555,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "creating"
  }
]


Summary: auto


**Waiting for idle state**


Output:
Chunk ID: 22f9b9
Wall time: 5.0005 seconds
Process running with session ID 38705
Original token count: 108
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 2.6702880859375e-05,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 0.0,
    "measured_perf": 62.24946212768555,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.24946212768555,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "model_loading"
  }
]


Output:
Chunk ID: fd31d5
Wall time: 5.0006 seconds
Process running with session ID 38705
Original token count: 108
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.326412200927734,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 0.0,
    "measured_perf": 62.24946212768555,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.24946212768555,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "model_loading"
  }
]


**Waiting for idle**


Output:
Chunk ID: 1a2576
Wall time: 5.0007 seconds
Process running with session ID 38705
Original token count: 108
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644615173339844,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 0.0,
    "measured_perf": 62.24946212768555,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.24946212768555,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "model_loading"
  }
]


Output:
Chunk ID: 68b931
Wall time: 5.0012 seconds
Process running with session ID 38705
Original token count: 0
Output:


Output:
Chunk ID: e2daa4
Wall time: 5.0011 seconds
Process running with session ID 38705
Original token count: 108
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644615173339844,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 0.0,
    "measured_perf": 62.24946212768555,
    "new_load": 0.0,
    "outdated": false,
    "perf": 62.24946212768555,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "model_loading"
  }
]


**Waiting for process completion**


Output:
Chunk ID: 08d53c
Wall time: 5.0012 seconds
Process running with session ID 38705
Original token count: 109
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644863128662109,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 1770668480.1470323,
    "measured_perf": 32.70523485167017,
    "new_load": 0.0,
    "outdated": false,
    "perf": 32.70523452758789,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770668425.1913676,
    "status": "idle"
  }
]


**Checking VRAM total metric**


Output:
Chunk ID: 6ee177
Wall time: 0.7695 seconds
Process exited with code 0
Original token count: 137
Output:
200
{"active_streams":0,"max_streams":100,"inference":{"total_batches":0,"total_outputs":0,"empty_cycles":0,"vad_filtered_chunks":0,"gpu_ms_p50":0,"gpu_ms_p99":0,"cycle_ms_p50":0,"cycle_ms_p99":0,"batch_size_avg":0.0},"http":{"requests_total":5,"requests_ok":5,"requests_err":0},"ws":{"connections_total":0,"connections_active":0,"connections_rejected":0},"cuda_available":true,"cuda_device_count":1,"gpu_name":"NVIDIA RTX PRO 6000 Blackwell Workstation Edition","vram_used_gb":10.43,"vram_reserved_gb":12.97,"vram_total_gb":102.0,"uptime_s":26.0}


**Running stress tests with full metrics**


bash -lc 'set -a; source .env; python3 production/test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10,25,50,75,100 --audio-duration 10 --cooldown 5'


Output:
Chunk ID: 65b4c0
Wall time: 1.0012 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 7a7e67
Wall time: 5.0011 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 584aed
Wall time: 5.0014 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 89bff4
Wall time: 5.0009 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 2ee34d
Wall time: 5.0009 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: c0173c
Wall time: 5.0017 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: fe1501
Wall time: 5.0014 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 8be80f
Wall time: 5.0015 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 731ae7
Wall time: 5.0013 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 65e9c0
Wall time: 5.0011 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 3ed621
Wall time: 5.0007 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Checking worker status for tests**


bash -lc 'set -a; source .env; python3 production/deploy.py workers'


Output:
Chunk ID: 27b769
Wall time: 0.6501 seconds
Process exited with code 0
Original token count: 122
Output:
[
  {
    "cur_load": 760.0,
    "cur_load_rolling_avg": 7.800704450392651,
    "cur_perf": 0.01822794377664921,
    "disk_usage": 4.707130432128906,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 1770668480.1470323,
    "measured_perf": 32.70523485167017,
    "new_load": 750.0,
    "outdated": false,
    "perf": 31.922658920288086,
    "reliability": 0.9939635166397678,
    "reqs_working": 76,
    "started_at": 1770668425.1913676,
    "status": "idle"
  }
]


**Waiting for stress test results**


Output:
Chunk ID: fd7a15
Wall time: 5.0016 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 557541
Wall time: 5.0011 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: fb8eac
Wall time: 5.0016 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Monitoring ongoing process**


Output:
Chunk ID: dc1859
Wall time: 5.0017 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: c1c310
Wall time: 5.0011 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Noting output timing**


Output:
Chunk ID: 5a1231
Wall time: 5.0016 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 446dc2
Wall time: 5.0008 seconds
Process running with session ID 51307
Original token count: 0
Output:


Output:
Chunk ID: 18004c
Wall time: 5.0013 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Checking worker concurrency levels**


Output:
Chunk ID: 75e88e
Wall time: 0.7295 seconds
Process exited with code 0
Original token count: 121
Output:
[
  {
    "cur_load": 470.0,
    "cur_load_rolling_avg": 16.081699956005117,
    "cur_perf": 13.570397086715628,
    "disk_usage": 4.769649505615234,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 1770668480.1470323,
    "measured_perf": 32.70523485167017,
    "new_load": 0.0,
    "outdated": false,
    "perf": 31.791423797607422,
    "reliability": 0.992940393129173,
    "reqs_working": 47,
    "started_at": 1770668425.1913676,
    "status": "idle"
  }
]


**Monitoring request completion**


Output:
Chunk ID: 333380
Wall time: 5.0011 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Waiting for output**


Output:
Chunk ID: 4e86cc
Wall time: 5.0008 seconds
Process running with session ID 51307
Original token count: 0
Output:


**Waiting for completion**


Output:
Chunk ID: 5c799d
Wall time: 1.2846 seconds
Process exited with code 0
Original token count: 1367
Output:
=== Hindi ASR Stress Test ===
URL: via endpoint
Endpoint: hindi-asr-prod
Modes: ['http']
Levels: [1, 5, 10, 25, 50, 75, 100]
Audio: 10.0s per stream
Endpoint ready: workers=1

============================================================
  Stress Level: c=1 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 1/1  Errors: 0
    Wall time: 1.48s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=1482.8ms  p95=1482.8ms  p99=1482.8ms
    Total:       p50=1482.8ms  p95=1482.8ms  p99=1482.8ms
    Msgs/stream: 1.0
    Throughput:  6.74x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=6 ok=6 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=5 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 5/5  Errors: 0
    Wall time: 3.32s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=2395.0ms  p95=3319.5ms  p99=3319.5ms
    Total:       p50=2395.0ms  p95=3319.5ms  p99=3319.5ms
    Msgs/stream: 1.0
    Throughput:  15.04x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=11 ok=11 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=10 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 10/10  Errors: 0
    Wall time: 5.61s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=3757.6ms  p95=5607.1ms  p99=5607.1ms
    Total:       p50=3757.6ms  p95=5607.1ms  p99=5607.1ms
    Msgs/stream: 1.0
    Throughput:  17.82x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=21 ok=21 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=25 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 25/25  Errors: 0
    Wall time: 12.62s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=7036.0ms  p95=12146.2ms  p99=12600.3ms
    Total:       p50=7036.0ms  p95=12146.2ms  p99=12600.3ms
    Msgs/stream: 1.0
    Throughput:  19.8x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=46 ok=46 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=50 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 50/50  Errors: 0
    Wall time: 24.09s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=13002.7ms  p95=23129.7ms  p99=24067.7ms
    Total:       p50=13002.7ms  p95=23129.7ms  p99=24067.7ms
    Msgs/stream: 1.0
    Throughput:  20.75x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=96 ok=96 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=75 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 75/75  Errors: 0
    Wall time: 35.68s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=18600.7ms  p95=34238.2ms  p99=35663.1ms
    Total:       p50=18600.7ms  p95=34238.2ms  p99=35663.1ms
    Msgs/stream: 1.0
    Throughput:  21.02x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=171 ok=171 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=100 mode=http audio=10.0s
============================================================

  Results:
    OK/Total: 98/100  Errors: 2
    Wall time: 46.36s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=24246.5ms  p95=44499.0ms  p99=46314.1ms
    Total:       p50=24246.5ms  p95=44499.0ms  p99=46314.1ms
    Msgs/stream: 1.0
    Throughput:  21.14x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=269 ok=269 err=0
    Errors: [{'stream': 53, 'error': "route failed: {'__request_id': 'fffcebd4-c53f-44c6-a3ee-c5a1f01d1a77', 'endpoint': 'hindi-asr-prod', 'request_idx': 767, 'status': 'All worker queues full!'}"}, {'stream': 6, 'error': "route failed: {'__request_id': '17e00bfe-7ef9-4675-821b-2dcb792f5e57', 'endpoint': 'hindi-asr-prod', 'request_idx': 766, 'status': 'All worker queues full!'}"}]

Results saved to: /home/ubuntu/nemo_hindi/production/../benchmark_results/stress_results_20260209_202433.csv
Full metrics saved to: /home/ubuntu/nemo_hindi/production/../benchmark_results/stress_results_20260209_202433.json

================================================================================
  STRESS TEST SUMMARY
================================================================================
  Mode  Conc   OK  Err   FT p50   FT p99  Tot p50   Msgs
------ ----- ---- ---- -------- -------- -------- ------
  http     1    1    0    1483ms    1483ms    1483ms   1.0
  http     5    5    0    2395ms    3320ms    2395ms   1.0
  http    10   10    0    3758ms    5607ms    3758ms   1.0
  http    25   25    0    7036ms   12600ms    7036ms   1.0
  http    50   50    0   13003ms   24068ms   13003ms   1.0
  http    75   75    0   18601ms   35663ms   18601ms   1.0
  http   100   98    2   24246ms   46314ms   24246ms   1.0


**Preparing SSE stress test**

I'm setting up to run SSE stress tests at the same levels as before.


bash -lc 'set -a; source .env; python3 production/test_stress.py --endpoint hindi-asr-prod --mode sse --levels 1,5,10,25,50,75,100 --audio-duration 10 --cooldown 5'


Output:
Chunk ID: 8c3b8d
Wall time: 1.0006 seconds
Process running with session ID 48611
Original token count: 0
Output:


Summary: auto


Output:
Chunk ID: 2cd071
Wall time: 5.0009 seconds
Process running with session ID 48611
Original token count: 0
Output:


**Waiting for results flush**


Output:
Chunk ID: 1aec31
Wall time: 5.0007 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: c13555
Wall time: 5.0012 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 6c3e68
Wall time: 5.0004 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 37ff39
Wall time: 5.0010 seconds
Process running with session ID 48611
Original token count: 0
Output:


**Extending wait time**


Output:
Chunk ID: 248872
Wall time: 5.0017 seconds
Process running with session ID 48611
Original token count: 0
Output:


**Verifying worker status**


Output:
Chunk ID: 28301b
Wall time: 0.6471 seconds
Process exited with code 0
Original token count: 121
Output:
[
  {
    "cur_load": 500.0,
    "cur_load_rolling_avg": 31.62044463700777,
    "cur_perf": 0.01834989363389278,
    "disk_usage": 4.769657135009766,
    "dlperf": 282.5863918716072,
    "id": 31135755,
    "loaded_at": 1770668480.1470323,
    "measured_perf": 32.70523485167017,
    "new_load": 0.0,
    "outdated": false,
    "perf": 26.99582862854004,
    "reliability": 0.9531688723781757,
    "reqs_working": 50,
    "started_at": 1770668425.1913676,
    "status": "idle"
  }
]


Output:
Chunk ID: fa49b5
Wall time: 5.0015 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 29f6c3
Wall time: 5.0011 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 0ef4f9
Wall time: 5.0010 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 65bb46
Wall time: 5.0009 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: d0d544
Wall time: 5.0014 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: d95595
Wall time: 5.0011 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 7dfaf7
Wall time: 5.0007 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 28e83a
Wall time: 5.0008 seconds
Process running with session ID 48611
Original token count: 0
Output:


**Observing longer SSE timing**


Output:
Chunk ID: f52c28
Wall time: 5.0010 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: f56b9e
Wall time: 5.0010 seconds
Process running with session ID 48611
Original token count: 0
Output:


**Checking current status**


Output:
Chunk ID: 958be3
Wall time: 5.0004 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: a935e9
Wall time: 5.0007 seconds
Process running with session ID 48611
Original token count: 0
Output:


Output:
Chunk ID: 4c3c97
Wall time: 4.3302 seconds
Process exited with code 0
Original token count: 1368
Output:
=== Hindi ASR Stress Test ===
URL: via endpoint
Endpoint: hindi-asr-prod
Modes: ['sse']
Levels: [1, 5, 10, 25, 50, 75, 100]
Audio: 10.0s per stream
Endpoint ready: workers=1

============================================================
  Stress Level: c=1 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 1/1  Errors: 0
    Wall time: 1.51s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=1062.0ms  p95=1062.0ms  p99=1062.0ms
    Total:       p50=1510.8ms  p95=1510.8ms  p99=1510.8ms
    Msgs/stream: 2.0
    Throughput:  6.62x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=270 ok=270 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=5 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 5/5  Errors: 0
    Wall time: 3.14s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=1162.4ms  p95=1765.8ms  p99=1765.8ms
    Total:       p50=2892.3ms  p95=3140.4ms  p99=3140.4ms
    Msgs/stream: 2.0
    Throughput:  15.9x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=275 ok=275 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=10 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 10/10  Errors: 0
    Wall time: 5.23s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=1282.2ms  p95=3245.2ms  p99=3245.2ms
    Total:       p50=4929.7ms  p95=5231.8ms  p99=5231.8ms
    Msgs/stream: 2.0
    Throughput:  19.1x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=285 ok=285 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=25 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 25/25  Errors: 0
    Wall time: 11.47s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=1817.5ms  p95=3437.2ms  p99=3562.2ms
    Total:       p50=11017.5ms  p95=11454.7ms  p99=11458.6ms
    Msgs/stream: 2.0
    Throughput:  21.81x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=310 ok=310 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=50 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 50/50  Errors: 0
    Wall time: 21.99s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=3615.4ms  p95=9733.4ms  p99=10284.3ms
    Total:       p50=20213.8ms  p95=21978.3ms  p99=21985.9ms
    Msgs/stream: 2.0
    Throughput:  22.74x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=360 ok=360 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=75 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 75/75  Errors: 0
    Wall time: 32.56s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=6601.6ms  p95=19486.8ms  p99=20753.6ms
    Total:       p50=28168.3ms  p95=32510.5ms  p99=32553.4ms
    Msgs/stream: 2.0
    Throughput:  23.03x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=435 ok=435 err=0

  Cooldown 5.0s...

============================================================
  Stress Level: c=100 mode=sse audio=10.0s
============================================================

  Results:
    OK/Total: 98/100  Errors: 2
    Wall time: 42.14s
    Connect:     p50=0ms  p95=0ms  p99=0ms
    First token: p50=10511.4ms  p95=28720.9ms  p99=30372.1ms
    Total:       p50=34644.6ms  p95=42111.0ms  p99=42129.9ms
    Msgs/stream: 2.0
    Throughput:  23.26x realtime
    Server GPU:  p50=0ms p99=0ms
    Server VRAM: 10.43GB/102.0GB
    HTTP:        total=533 ok=533 err=0
    Errors: [{'stream': 91, 'error': "route failed: {'__request_id': '89acdddd-3a20-42ab-b01d-311003597b3f', 'endpoint': 'hindi-asr-prod', 'request_idx': 1046, 'status': 'All worker queues full!'}"}, {'stream': 45, 'error': "route failed: {'__request_id': 'a4082e48-4255-4375-97d1-6e04f022467b', 'endpoint': 'hindi-asr-prod', 'request_idx': 1047, 'status': 'All worker queues full!'}"}]

Results saved to: /home/ubuntu/nemo_hindi/production/../benchmark_results/stress_results_20260209_202718.csv
Full metrics saved to: /home/ubuntu/nemo_hindi/production/../benchmark_results/stress_results_20260209_202718.json

================================================================================
  STRESS TEST SUMMARY
================================================================================
  Mode  Conc   OK  Err   FT p50   FT p99  Tot p50   Msgs
------ ----- ---- ---- -------- -------- -------- ------
   sse     1    1    0    1062ms    1062ms    1511ms   2.0
   sse     5    5    0    1162ms    1766ms    2892ms   2.0
   sse    10   10    0    1282ms    3245ms    4930ms   2.0
   sse    25   25    0    1818ms    3562ms   11018ms   2.0
   sse    50   50    0    3615ms   10284ms   20214ms   2.0
   sse    75   75    0    6602ms   20754ms   28168ms   2.0
   sse   100   98    2   10511ms   30372ms   34645ms   2.0


**Noticing incomplete GPU timing metrics**

Current GPU timing metrics come only from WebSocket batching, which HTTP/SSE paths lack, so GPU p50/p99 show zero; adding GPU timing to HTTP inference would require deeper instrumentation but the existing processing_time and VRAM metrics partially cover performance.


**Considering adding HTTP latency metrics**

The stress test logs VRAM and HTTP counts but no GPU ms, which may be acceptable; adding detailed HTTP latency percentiles to /metrics aligns with the user's request for thorough metrics and complements existing VRAM and server stats, so the plan status updates reflect this progress.


Add metrics exposure for serverless (POST /metrics in server, /metrics handler in worker) and extend stress test to fetch metrics via endpoint route

Rebuild/push new Docker image, update Vast template/workergroup, and wait for worker to reach idle

Run full-metrics stress tests (HTTP + SSE) and collect results


Output: Plan updated


**Summarizing benchmark results and changes**

I'll report key benchmark results, note GPU per-batch metrics stay zero due to WS loop design, and highlight errors at high concurrency from queue limits affecting scaling. I'll also summarize code updates across server.py, worker.py, tests, and deployment artifacts related to VRAM and metrics. Next steps and doc updates will be noted but deferred.


Output:
Chunk ID: 5fc746
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 189
Output:
total 4576
-rw-rw-r-- 1 ubuntu ubuntu  15995 Feb  9 20:27 stress_results_20260209_202718.json
-rw-rw-r-- 1 ubuntu ubuntu    724 Feb  9 20:27 stress_results_20260209_202718.csv
-rw-rw-r-- 1 ubuntu ubuntu  15974 Feb  9 20:24 stress_results_20260209_202433.json
-rw-rw-r-- 1 ubuntu ubuntu    736 Feb  9 20:24 stress_results_20260209_202433.csv
-rw-rw-r-- 1 ubuntu ubuntu  13119 Feb  9 20:14 stress_results_20260209_201434.json
-rw-rw-r-- 1 ubuntu ubuntu    725 Feb  9 20:14 stress_results_20260209_201434.csv
-rw-rw-r-- 1 ubuntu ubuntu   8403 Feb  9 19:57 stress_results_20260209_195730.json
-rw-rw-r-- 1 ubuntu ubuntu    543 Feb  9 19:57 stress_results_20260209_195730.csv
-rw-rw-r-- 1 ubuntu ubuntu   8418 Feb  9 19:56 stress_results_20260209_195631.json


**Summarizing final stress test metrics**

The latest full metrics are from timestamps 20260209_202433 (HTTP) and 20260209_202718 (SSE), with prior data to ignore. The current worker ID is 31135755 running image v9-production, with template hash 118364... and image version v9. HTTP metric counts are cumulative and show increasing test loads; throughput (21x realtime) is lower than earlier due to switching from A100 to RTX PRO 6000 GPUs and possible pipeline changes, but this is acceptable. Errors at concurrency 100 from "All worker queues full" suggest increasing worker capacity or adjusting queues/timing. Suggested next steps include testing audio duration around 30s to better simulate real usage and exploring varied workloads. Recent code changes affecting metrics were made in server.py, worker.py, and the stress test itself. The final summary will detail these results, metric locations, and recommended next actions.


**Finalizing stress test summary and metrics details**

The response will start with a brief overview of code changes—worker updated to v9-production with /metrics now exposing CUDA info—and summarize completed full-metrics stress tests. I'll include concise throughput and latency stats at key concurrencies for HTTP and SSE, highlight errors at concurrency 100 from queue limits, and note GPU p50/p99 zeros are expected since those relate only to WebSocket batching. VRAM usage and memory capacity indicate no memory bottleneck. The JSON metrics files include server_before/after fields per-level, confirming comprehensive data capture. Next steps will propose longer audio tests, WebSocket tests, and scaling analysis, along with a caution about hardware differences since switching to RTX PRO 6000. Uncommitted code changes will be mentioned too.


python3 - <<'PY'
import json
from pathlib import Path
p=Path('benchmark_results/stress_results_20260209_202433.json')
D=json.loads(p.read_text())
print(D[0].keys())
print(D[0]['server_after'].keys())
PY


Output:
Chunk ID: 6ab9b1
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 68
Output:
dict_keys(['level', 'mode', 'summary', 'server_before', 'server_after', 'errors'])
dict_keys(['active_streams', 'max_streams', 'inference', 'http', 'ws', 'cuda_available', 'cuda_device_count', 'gpu_name', 'vram_used_gb', 'vram_reserved_gb', 'vram_total_gb', 'uptime_s'])


**Summarizing test and instrumentation results**

Completed full metrics instrumentation and stress tests with files changed in production/server.py, worker.py, and test_stress.py. Deployed new image and updated template hash; stress test showed concurrency limit errors at 100 workers, indicating queue saturation. GPU latency metrics are zero due to no WebSocket batching yet. Next steps include longer audio tests, adding WS metrics, scaling law fitting, and region filtering, with a note on possible per-request GPU metrics instrumentation if needed.


Full‑metrics instrumentation and stress tests are done. I updated the server/worker to expose `/metrics` through PyWorker, fixed the CUDA VRAM reporting (`total_memory`), rebuilt the image, redeployed, and ran full HTTP + SSE sweeps with metrics captured per level.

Where the metrics live (latest runs):
- HTTP: `benchmark_results/stress_results_20260209_202433.csv` and `benchmark_results/stress_results_20260209_202433.json`
- SSE: `benchmark_results/stress_results_20260209_202718.csv` and `benchmark_results/stress_results_20260209_202718.json`

Key results (RTX PRO 6000 WS, 10s audio per stream):
- HTTP p50 total latency: 1.48s (c=1), 7.04s (c=25), 13.00s (c=50), 24.25s (c=100)
- HTTP throughput: ~20–21x realtime at c=50–100; c=100 had 2/100 route failures (“All worker queues full”)
- SSE p50 total latency: 1.51s (c=1), 11.02s (c=25), 20.21s (c=50), 34.65s (c=100)
- SSE throughput: ~22–23x realtime at c=50–100; c=100 had 2/100 route failures
- VRAM: ~10.43GB used, ~12.97GB reserved, 102GB total (now correctly reported)
- `/metrics` GPU p50/p99 stays 0 for HTTP/SSE because those fields are tied to the WS batching loop; the JSON still captures all other server metrics + VRAM + HTTP counts

What changed (code + deploy):
- `/metrics` now supports POST and returns CUDA fields (`cuda_available`, `device_count`, `gpu_name`, VRAM used/reserved/total) from `production/server.py`
- PyWorker now proxies `/metrics` in `production/worker.py`
- `production/test_stress.py` now fetches metrics via serverless `/route`
- Deployed image: `bharathkumar192/hindi-nemotron-asr:v9-production`
- Workergroup template hash: `118364eef727b32735fa02ee2908b7a1`
- Current worker: `31135755` (idle and healthy)

If you want “GPU time per request” for HTTP/SSE (not just VRAM), I can add per‑request timing in the HTTP/SSE handlers and expose percentiles in `/metrics`.

Next steps (pick one):
1) Run longer‑duration stress (30s/60s) to simulate real customer sessions and capture drift.
2) Add per‑request GPU timing for HTTP/SSE and re‑run the sweeps.
3) Proceed to scaling‑law fitting using the new JSON outputs.