Output:
Chunk ID: 5abc3d
Wall time: 0.6352 seconds
Process exited with code 0
Original token count: 1512
Output:

=== debug ===
nch id 11998 updating metrics
workers hindi-asr-h100-bench: 31145035,38.883190,protected:0  1
1770691703.769 [asm_ratio_manager.cpp:791 log_endpoint_workers] hindi-asr-h100-bench 0x7f02c808c600 workers_lut_: 1770691685.519309 workers: 1  rdy_workers: 0
1770691703.769 [asm_ratio_manager.cpp:794 manage_workers] workers hindi-asr-h100-bench:
[       ID]        Perf      MeasPerf   Reli    DLPerf   CurPerf    NewLoad, CurLoad CurTasks ReqsWork          LoadedAt         CreatedAt         StartedAt       Disk Status          IntendedStatus  LastDbStatus    PerfPerDollar
[ 31145035]      38.883         0.000  1.000   344.892     0.000       0.00,    0.00        0        0             0.000    1770691529.769    1770691529.769      0.000 loading         running         loading              23.145
1770691703.769 [asm_ratio_manager.cpp:806 manage_workers] endpoint: 11998 hindi-asr-h100-bench has agroups: ID:16830 autogroup size: 1
[manage_workers][update_strategy] cur_loadps: 0.00 1: (0.00,14.93) 2: (0.01,58.21) 3: (14.81,945.87) 4: (15.54,1251.64) 5: (15.81,1516.73) E: (0.00,14.93)
[manage_workers][update_strategy] etime: 1.0000 grp_new_load: 0.000000 extra_load: 0.000 extra_tasks: 0 min_load: 1.0 predload_1s: 1.0 10s: 1.0 30s: 1.0 3600s: 15.5
[manage_workers][update_strategy] avg_perf: 38.883190, avg_perf_rolling: 70.708242, avg_idle_perf: 0.000000, targperf_3600s: 33.299957, predperf_3600s: 38.883190 perf_gap: -5.583234
[manage_workers][update_strategy] pred cap/targ_1s: 0.0/1.4 10s: 38.9/1.4 30s: 38.9/1.4 3600s: 38.9/33.3 24hr: 38.9/33.9  create_score: -0.08/-0.08 rolling: 0.50, delete_score: 0.07 rolling: 0.07
[manage_workers][update_strategy] score/thresh: start: -3745.461873/0.200000 ... stop: -142.857143/0.800000 ... create: -0.078962/0.500000 ... delete: 0.070766/1.000000
1770691703.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691703.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers

=== info1 ===
sm_ratio_manager.cpp:1025 manage_workers] |eg.autogroups|:1 (0.588269 > 0.500000)
1770690972.769 [asm_ratio_manager.cpp:1047 manage_workers] (0.588269 > 0.500000) |best|:1 |offers|:20 create_perf_gap_:51.801048
1770690972.769[asm_ratio_manager.cpp:1057 manage_workers] creating 1 perf_gap: 51.801048, create_score: 0.588269, avg_perf: 110.125294
1770690972.769493 [reconcile_workers] Creating worker from offer 27615974
1770690972.769493 [be_common.cpp:526 create_instances] |X|: 1:   (27615974,34267,213.261328,347.695801)
1770690972.769493 [be_common.cpp:556 create_instances]  {"MASTE...
1770690972.769493 [be_common.cpp:596 create_instances]  31144824 done setting callback
1770691060.769 [asm_ratio_manager.cpp:1142 manage_workers] errored_workers (1/4)  1: 31144514
1770691060.769493 [reconcile_workers] Restarting worker 31144514: error -> RESTARTED
1770691060.769493 [be_common.cpp:397 _restart_instances] 1:  31144514,error,15.751575,29.657583
1770691060.769493 [be_common.cpp:401 _restart_instances]   resetting started_at
1770691107.769 [asm_ratio_manager.cpp:1025 manage_workers] |eg.autogroups|:1 (0.547936 > 0.500000)
1770691107.769 [asm_ratio_manager.cpp:1047 manage_workers] (0.547936 > 0.500000) |best|:1 |offers|:21 create_perf_gap_:15.377588
1770691107.769[asm_ratio_manager.cpp:1057 manage_workers] creating 1 perf_gap: 15.377588, create_score: 0.547936, avg_perf: 109.878079
1770691107.769 [asm_ratio_manager.cpp:1142 manage_workers] errored_workers (0/4)  1: 31144514
1770691107.769493 [reconcile_workers] Destroying worker 31144514: error -> DESTROYED
1770691107.769493 [reconcile_workers] Creating worker from offer 27615974
1770691107.769493 [be_common.cpp:526 create_instances] |X|: 1:   (27615974,34267,19.226525,31.167986)
1770691107.769493 [be_common.cpp:556 create_instances]  {"MASTE...
1770691107.769493 [be_common.cpp:728 _destroy_instances] 1:  31144514,49188, error,0.123450,29.930974
1770691107.769493 [be_common.cpp:566 _create_instances] create 0 failed,

=== trace ===
621 update_workers] done with 1 endpoint workers
1770691676.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691676.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691677.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691677.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691678.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691678.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691679.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691679.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691680.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691680.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691681.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691681.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691682.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691682.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691683.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691683.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
1770691684.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint workers
1770691684.769493 [asm_ratio_manager.cpp:621 update_workers] done with 1 endpoint workers
workers: (I: 31145035, hindi-asr-h100-bench:11998:16830, , reliability: 1.000000, perf: 38.883190, capacity: 0.000000/0.000000, acks: 0/0)
1770691685.769493 [asm_ratio_manager.cpp:507 update_workers]  updating 1 workers, 1 endpoint


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Observing worker loading delays**


Output:
Chunk ID: 93d505
Wall time: 5.0012 seconds
Process running with session ID 3601
Original token count: 95
Output:
02:48:17   31145035 loading H100 SXM loading bharathkumar192/hindi-nemotron-asr:v14-production
02:48:22   31145035 loading H100 SXM loading bharathkumar192/hindi-nemotron-asr:v14-production
02:48:28   31145035 loading H100 SXM loading bharathkumar192/hindi-nemotron-asr:v14-production
02:48:33   31145035 loading H100 SXM loading bharathkumar192/hindi-nemotron-asr:v14-production


Summary: auto


**Assessing worker readiness and fallback**


python3 - <<'PY'
import requests,time
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip(); break
h={'Accept':'application/json','Authorization':f'Bearer {k}'}
api='https://console.vast.ai/api/v0'; run='https://run.vast.ai'; ep='hindi-asr-h100-bench'
E=[x for x in requests.get(f'{api}/endptjobs/',headers=h).json()['results'] if x['endpoint_name']==ep][0]
workers=requests.post(f'{run}/get_endpoint_workers/',json={'id':E['id'],'api_key':E['api_key']}).json();
if isinstance(workers,dict) and 'workers' in workers: workers=workers['workers']
print(workers)
for w in workers:
    iid=w['id']; inst=requests.get(f'{api}/instances/{iid}/',headers=h).json().get('instances',{})
    print('iid',iid,'status',w.get('status'),'loaded_at',w.get('loaded_at'),'started_at',w.get('started_at'),'cur_perf',w.get('cur_perf'),'inst_state',inst.get('actual_status'),'msg',inst.get('status_msg'))
PY


Output:
Chunk ID: dc6dbe
Wall time: 0.6850 seconds
Process exited with code 0
Original token count: 133
Output:
[{'cur_load': 0.0, 'cur_load_rolling_avg': 0.0, 'cur_perf': 0.0, 'disk_usage': 7.62939453125e-06, 'dlperf': 344.89210307468466, 'id': 31145035, 'loaded_at': 0.0, 'measured_perf': 38.8831901550293, 'new_load': 0.0, 'outdated': False, 'perf': 38.8831901550293, 'reliability': 1.0, 'reqs_working': 0, 'started_at': 1770691529.7694926, 'status': 'loading'}]
iid 31145035 status loading loaded_at 0.0 started_at 1770691529.7694926 cur_perf 0.0 inst_state running msg success, running bharathkumar192/hindi-nemotron-asr:v14-production


**Investigating startup log completeness**


VAST_ENDPOINT_NAME=hindi-asr-h100-bench python3 production/deploy.py logs | rg -n "Application startup complete|Traceback|RuntimeError|CUDA out of memory|error ->|failed|NotEnoughData|benchmark|on_error|401|ImportError|Exception|No module" -i | sed -n '1,320p'


Output:
Chunk ID: 1018c4
Wall time: 0.6449 seconds
Process exited with code 0
Original token count: 131
Output:
12:[manage_workers][update_strategy] avg_perf: 38.883190, avg_perf_rolling: 69.940117, avg_idle_perf: 0.000000, targperf_3600s: 32.440601, predperf_3600s: 38.883190 perf_gap: -6.442589
28:1770691060.769493 [reconcile_workers] Restarting worker 31144514: error -> RESTARTED
30:1770691060.769493 [be_common.cpp:401 _restart_instances]   resetting started_at
35:1770691107.769493 [reconcile_workers] Destroying worker 31144514: error -> DESTROYED
40:1770691107.769493 [be_common.cpp:566 _create_instances] create 0 failed,


**Reverting worker templates to stable**


**Confirming template hash update strategy**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" update workergroup 16824 --endpoint_id 11995 --template_hash 593eea937adadfbc39c303eab5c69a75 --gpu_ram 80 --min_load 1 --target_util 0.7 --cold_mult 1.5 --cold_workers 1 --search_params 'verified=true rentable=true rented=false external=false gpu_ram>=80 num_gpus=1 reliability>0.95 direct_port_count>=2 disk_space>=30 inet_down>200 gpu_name in ["H100 NVL","H100 SXM","H100 PCIe"]' --raw | sed -n '1,120p'


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" update workergroup 16830 --endpoint_id 11998 --template_hash 593eea937adadfbc39c303eab5c69a75 --gpu_ram 80 --min_load 1 --target_util 0.7 --cold_mult 1.5 --cold_workers 1 --search_params 'verified=true rentable=true rented=false external=false gpu_ram>=80 num_gpus=1 reliability>0.95 direct_port_count>=2 disk_space>=30 inet_down>200 gpu_name in ["H100 NVL","H100 SXM","H100 PCIe"]' --raw | sed -n '1,120p'


Output:
Chunk ID: 1afa6a
Wall time: 0.3253 seconds
Process exited with code 0
Original token count: 22
Output:
failed with error 400: invalid template hash or id or template not accessible by user


Output:
Chunk ID: 202933
Wall time: 0.3174 seconds
Process exited with code 0
Original token count: 22
Output:
failed with error 400: invalid template hash or id or template not accessible by user


**Assessing template versioning and creation**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" create template --help | sed -n '1,280p'


Output:
Chunk ID: 630192
Wall time: 0.1111 seconds
Process exited with code 0
Original token count: 684
Output:
usage: vastai create template

options:
 -h, --help                     show this help message and exit
 --name NAME                    name of the template
 --image IMAGE                  docker container image to launch
 --image_tag IMAGE_TAG          docker image tag (can also be appended to end of image_path)
 --href HREF                    link you want to provide
 --repo REPO                    link to repository
 --login LOGIN                  docker login arguments for private repo authentication, surround with ''
 --env ENV                      Contents of the 'Docker options' field
 --ssh                          Launch as an ssh instance type
 --jupyter                      Launch as a jupyter instance instead of an ssh instance
 --direct                       Use (faster) direct connections for jupyter & ssh
 --jupyter-dir JUPYTER_DIR      For runtype 'jupyter', directory in instance to use to launch jupyter. Defaults to image's working directory
 --jupyter-lab                  For runtype 'jupyter', Launch instance with jupyter lab
 --onstart-cmd ONSTART_CMD      contents of onstart script as single argument
 --search_params SEARCH_PARAMS  search offers filters
 -n, --no-default               Disable default search param query args
 --disk_space DISK_SPACE        disk storage space, in GB
 --readme README                readme string
 --hide-readme                  hide the readme from users
 --desc DESC                    description string
 --public                       make template available to public
 --url URL                      server REST api url
 --retry RETRY                  retry limit
 --raw                          output machine-readable json
 --explain                      output verbose explanation of mapping of CLI calls to HTTPS API endpoints
 --curl                         show a curl equivalency to the call
 --api-key API_KEY              api key. defaults to using the one stored in /home/ubuntu/.config/vastai/vast_api_key
 --version                      show version

Create a template that can be used to create instances with

Example:
    vastai create template --name "tgi-llama2-7B-quantized" --image "ghcr.io/huggingface/text-generation-inference:1.0.3"
                            --env "-p 3000:3000 -e MODEL_ARGS='--model-id TheBloke/Llama-2-7B-chat-GPTQ --quantize gptq'"
                            --onstart-cmd 'wget -O - https://raw.githubusercontent.com/vast-ai/vast-pyworker/main/scripts/launch_tgi.sh | bash'
                            --search_params "gpu_ram>=23 num_gpus=1 gpu_name=RTX_3090 inet_down>128 direct_port_count>3 disk_space>=192 driver_version>=535086005 rented=False"
                            --disk_space 8.0 --ssh --direct


**Creating recovery template with v13**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" create template --name 'hindi-asr-v13-recover' --image 'bharathkumar192/hindi-nemotron-asr:v13-production' --env '-p 8000:8000 -p 18000:18000 -e MAX_ACTIVE_STREAMS=500' --onstart-cmd '/app/onstart.sh' --search_params 'verified=true rentable=true rented=false external=false gpu_ram>=40 num_gpus=1 reliability>0.95 direct_port_count>=2 disk_space>=30 inet_down>200' --raw | sed -n '1,260p'


Output:
Chunk ID: fc4535
Wall time: 0.3254 seconds
Process exited with code 0
Original token count: 284
Output:
New Template: {'name': 'hindi-asr-v13-recover', 'image': 'bharathkumar192/hindi-nemotron-asr:v13-production', 'tag': 'v13-production', 'href': 'https://hub.docker.com/r/bharathkumar192/hindi-nemotron-asr/', 'repo': 'bharathkumar192/hindi-nemotron-asr', 'env': '-p 8000:8000 -p 18000:18000 -e MAX_ACTIVE_STREAMS=500', 'onstart': '/app/onstart.sh', 'jup_direct': False, 'ssh_direct': False, 'use_jupyter_lab': False, 'runtype': 'args', 'use_ssh': False, 'jupyter_dir': None, 'docker_login_repo': None, 'extra_filters': '{"verified": {"eq": true}, "external": {"eq": false}, "rentable": {"eq": true}, "rented": {"eq": false}, "gpu_ram": {"gte": 40000.0}, "num_gpus": {"eq": "1"}, "reliability": {"gt": "0.95"}, "direct_port_count": {"gte": "2"}, "disk_space": {"gte": "30"}, "inet_down": {"gt": "200"}}', 'recommended_disk_space': 16, 'readme_visible': True, 'desc': None, 'private': True, 'readme_hash': None, 'hash_id': '4f07b7ec6cffedfe18bde28236c5c373', 'autoscaler': False, 'recommended': False, 'vm': False, 'sort_order': None, 'creator_id': 389181, 'created_from_id': None, 'count_created': 0, 'cached': False, 'id': 342197}
null


**Updating worker groups and filters**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" update workergroup 16824 --endpoint_id 11995 --template_hash 4f07b7ec6cffedfe18bde28236c5c373 --gpu_ram 80 --min_load 1 --target_util 0.7 --cold_mult 1.5 --cold_workers 1 --search_params 'verified=true rentable=true rented=false external=false gpu_ram>=80 num_gpus=1 reliability>0.95 direct_port_count>=2 disk_space>=30 inet_down>200 gpu_name in ["H100 NVL","H100 SXM","H100 PCIe"]' --raw | sed -n '1,120p'


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" update workergroup 16830 --endpoint_id 11998 --template_hash 4f07b7ec6cffedfe18bde28236c5c373 --gpu_ram 80 --min_load 1 --target_util 0.7 --cold_mult 1.5 --cold_workers 1 --search_params 'verified=true rentable=true rented=false external=false gpu_ram>=80 num_gpus=1 reliability>0.95 direct_port_count>=2 disk_space>=30 inet_down>200 gpu_name in ["H100 NVL","H100 SXM","H100 PCIe"]' --raw | sed -n '1,120p'


Output:
Chunk ID: 556cfc
Wall time: 0.3350 seconds
Process exited with code 0
Original token count: 11
Output:
workergroup update {'success': True}
null


Output:
Chunk ID: 6b8309
Wall time: 0.3321 seconds
Process exited with code 0
Original token count: 11
Output:
workergroup update {'success': True}
null


**Restarting worker with updated template**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" destroy instance 31145035 --raw | sed -n '1,80p'


Output:
Chunk ID: bfc44d
Wall time: 0.5684 seconds
Process exited with code 0
Original token count: 2
Output:
null


Summary: auto


**Planning polling with new worker**


python3 -u - <<'PY'
import time,requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip(); break
h={'Accept':'application/json','Authorization':f'Bearer {k}'}
api='https://console.vast.ai/api/v0'; run='https://run.vast.ai'; ep='hindi-asr-h100-bench'
E=[x for x in requests.get(f'{api}/endptjobs/',headers=h).json()['results'] if x['endpoint_name']==ep][0]
for i in range(80):
    workers=requests.post(f'{run}/get_endpoint_workers/',json={'id':E['id'],'api_key':E['api_key']}).json();
    if isinstance(workers,dict) and 'workers' in workers: workers=workers['workers']
    ts=time.strftime('%H:%M:%S')
    if not workers:
        print(ts,'workers=0')
    ready=False
    for w in workers:
        iid=w.get('id'); st=w.get('status')
        inst=requests.get(f'{api}/instances/{iid}/',headers=h).json().get('instances')
        if isinstance(inst,dict):
            img=inst.get('image_uuid')
            print(ts,' ',iid,st,inst.get('gpu_name'),inst.get('actual_status'),img)
            if st in ('idle','running') and isinstance(img,str) and 'v13-production' in img:
                ready=True
    if ready:
        print('READY_V13')
        break
    time.sleep(5)
PY


Output:
Chunk ID: 8e989c
Wall time: 1.0006 seconds
Process running with session ID 1766
Original token count: 0
Output:


Output:
Chunk ID: 0b0656
Wall time: 5.0009 seconds
Process running with session ID 1766
Original token count: 47
Output:
02:50:07   31145113 creating H100 NVL None bharathkumar192/hindi-nemotron-asr:v13-production
02:50:13   31145113 creating H100 NVL None bharathkumar192/hindi-nemotron-asr:v13-production


Output:
Chunk ID: 519fad
Wall time: 5.0016 seconds
Process running with session ID 1766
Original token count: 72
Output:
02:50:18   31145113 creating H100 NVL loading bharathkumar192/hindi-nemotron-asr:v13-production
02:50:24   31145113 loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production
02:50:29   31145113 loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production


Output:
Chunk ID: d50c01
Wall time: 5.0012 seconds
Process running with session ID 1766
Original token count: 73
Output:
02:50:35   31145113 loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production
02:50:40   31145113 loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production
02:50:46   31145113 model_loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production


Output:
Chunk ID: 34c41a
Wall time: 5.0016 seconds
Process running with session ID 1766
Original token count: 51
Output:
02:50:51   31145113 model_loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production
02:50:57   31145113 model_loading H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production


Output:
Chunk ID: 979aeb
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 26
Output:
02:51:02   31145113 idle H100 NVL running bharathkumar192/hindi-nemotron-asr:v13-production
READY_V13


**Verifying endpoint stability**


python3 production/test_smoke.py --endpoint hindi-asr-h100-bench --route-cost 10


Output:
Chunk ID: 89da91
Wall time: 5.4301 seconds
Process exited with code 0
Original token count: 130
Output:

=== Health ===
  [PASS] workers=1

=== HTTP Batch ===
  [PASS] 435ms text='<s>' proc=0.084s

=== HTTP Stream (SSE) ===
  [PASS] 2 events 421ms
  Final: '<s>'

=== WebSocket ===
  Discovered worker=31145113 ws_url=ws://104.189.178.116:2287/ws
  Connected 63ms
  Ready: {"type": "Ready", "id": 10}
  [PASS] 1 msgs 3110ms first_token=1989ms close_code=None
  Last: '<s>'

=== SUMMARY ===
  health               [PASS]
  http_batch           [PASS]
  http_stream          [PASS]
  websocket            [PASS]

All passed!


**Planning metrics parsing script**


python3 - <<'PY'
import json, csv
from pathlib import Path

# helper

def load_json(p):
    return json.loads(Path(p).read_text())

def print_ws(name,p):
    d=load_json(p)
    print('\n',name)
    for row in d:
        s=row['summary']
        print(f"c={s['concurrency']:>4} ok={s['ok']:>4}/{s['concurrency']:<4} err={s['errors']:<4} ft_p50={s['first_token_ms_p50']:>7}ms total_p50={s['total_ms_p50']:>7}ms drift_p99={s['drift_ms_p99']:>7}ms msgs={s['msgs_per_stream_avg']:>4} tput={s['throughput_x']:>6}x")

def print_full(name,p):
    d=load_json(p)
    print('\n',name)
    for row in d:
        s=row['summary']
        print(f"{s['mode']:>4} c={s['concurrency']:>4} ok={s['ok']:>4}/{s['concurrency']*s['requests_per_client']:<4} err={s['errors']:<4} qf={s['queue_full_errors']:<4} p95_total={s['total_ms_p95']:>8}ms p95_ft={s['first_token_ms_p95']:>8}ms tput={s['throughput_x']:>6}x wer={s['wer_pct']:>5}%")

def print_autoscale(p):
    d=load_json(p)
    print('\nAutoscale details')
    for row in d:
        s=row['summary']
        ws=row.get('worker_samples',[])
        max_workers=max((len(x.get('workers',[])) for x in ws), default=0)
        ids=sorted({w.get('id') for snap in ws for w in snap.get('workers',[]) if w.get('id') not in (None,0)})
        statuses={}
        for snap in ws:
            for w in snap.get('workers',[]):
                statuses[w.get('status','?')]=statuses.get(w.get('status','?'),0)+1
        print(f"c={s['concurrency']} ok={s['ok']}/{s['concurrency']*s['requests_per_client']} err={s['errors']} qf={s['queue_full_errors']} tput={s['throughput_x']}x max_workers_seen={max_workers} worker_ids={ids} status_counts={statuses}")

print_ws('WS 160ms', 'benchmark_results/ws_h100_v13_chunk160_c1_300.json')
print_ws('WS 320ms', 'benchmark_results/ws_h100_v13_chunk320_c1_127.json')
print_ws('WS 560ms', 'benchmark_results/ws_h100_v13_chunk560_c1_127.json')
print_full('HTTP+SSE clean c1-50', 'benchmark_results/full_h100_v13_fleurs_c1_50_clean.json')
print_full('HTTP+SSE unstable c1-100', 'benchmark_results/full_h100_v13_fleurs_c1_100.json')
print_full('HTTP autoscale c50-150 r4', 'benchmark_results/autoscale_http_h100_v13_c50_150_r4.json')
print_autoscale('benchmark_results/autoscale_http_h100_v13_c50_150_r4.json')
PY


Output:
Chunk ID: 62ccf3
Wall time: 0.0508 seconds
Process exited with code 0
Original token count: 1627
Output:

 WS 160ms
c=   1 ok=   1/1    err=0    ft_p50=  726.3ms total_p50= 9199.1ms drift_p99=  166.3ms msgs= 1.0 tput=  0.99x
c=  10 ok=  10/10   err=0    ft_p50= 2076.0ms total_p50= 9246.4ms drift_p99= 1518.3ms msgs= 1.0 tput=  9.86x
c=  25 ok=  25/25   err=0    ft_p50= 3793.6ms total_p50= 9257.0ms drift_p99= 3236.5ms msgs= 1.0 tput= 24.61x
c=  50 ok=  50/50   err=0    ft_p50= 6965.2ms total_p50= 9318.3ms drift_p99= 6410.7ms msgs= 1.0 tput= 48.82x
c=  75 ok=  75/75   err=0    ft_p50=10342.2ms total_p50=11346.8ms drift_p99= 9790.1ms msgs= 1.0 tput= 60.21x
c= 100 ok= 100/100  err=0    ft_p50=13538.3ms total_p50=14542.6ms drift_p99=12989.0ms msgs= 1.0 tput= 62.63x
c= 127 ok= 127/127  err=0    ft_p50=18735.2ms total_p50=22882.3ms drift_p99=22105.1ms msgs= 1.0 tput= 41.41x
c= 160 ok=   0/160  err=160  ft_p50=      0ms total_p50= 8368.7ms drift_p99=      0ms msgs= 0.0 tput=   0.0x
c= 200 ok=   0/200  err=200  ft_p50=      0ms total_p50=      0ms drift_p99=      0ms msgs= 0.0 tput=   0.0x
c= 250 ok=   0/250  err=250  ft_p50=      0ms total_p50=      0ms drift_p99=      0ms msgs= 0.0 tput=   0.0x
c= 300 ok=   0/300  err=300  ft_p50=      0ms total_p50=      0ms drift_p99=      0ms msgs= 0.0 tput=   0.0x

 WS 320ms
c=   1 ok=   1/1    err=0    ft_p50= 1821.1ms total_p50= 9352.0ms drift_p99= 1261.1ms msgs= 1.0 tput=  0.98x
c=  10 ok=  10/10   err=0    ft_p50= 1667.9ms total_p50= 9364.8ms drift_p99= 1112.9ms msgs= 1.0 tput=  9.72x
c=  25 ok=  25/25   err=0    ft_p50= 3512.8ms total_p50= 9381.5ms drift_p99= 2957.0ms msgs= 1.0 tput= 24.28x
c=  50 ok=  50/50   err=0    ft_p50= 6331.9ms total_p50= 9367.8ms drift_p99= 5778.6ms msgs= 1.0 tput= 48.48x
c=  75 ok=  75/75   err=0    ft_p50= 9206.4ms total_p50=10206.4ms drift_p99= 8654.0ms msgs= 1.0 tput= 66.93x
c= 100 ok= 100/100  err=0    ft_p50= 7108.3ms total_p50= 9686.2ms drift_p99=13171.0ms msgs= 1.0 tput= 41.38x
c= 127 ok= 127/127  err=0    ft_p50=15697.6ms total_p50=16696.7ms drift_p99=15150.1ms msgs= 1.0 tput= 69.28x

 WS 560ms
c=   1 ok=   1/1    err=0    ft_p50=  509.3ms total_p50= 9607.3ms drift_p99=  -50.7ms msgs= 1.0 tput=  0.95x
c=  10 ok=  10/10   err=0    ft_p50= 1353.8ms total_p50= 9613.0ms drift_p99=  795.1ms msgs= 1.0 tput=  9.48x
c=  25 ok=  25/25   err=0    ft_p50= 3155.0ms total_p50= 9598.9ms drift_p99= 2601.1ms msgs= 1.0 tput= 23.72x
c=  50 ok=  50/50   err=0    ft_p50= 6086.9ms total_p50= 9610.7ms drift_p99= 5534.2ms msgs= 1.0 tput= 47.31x
c=  75 ok=  75/75   err=0    ft_p50= 9793.2ms total_p50=10794.9ms drift_p99= 9242.2ms msgs= 1.0 tput= 36.67x
c= 100 ok= 100/100  err=0    ft_p50=13978.5ms total_p50= 9635.0ms drift_p99=13420.9ms msgs= 0.1 tput= 39.95x
c= 127 ok= 127/127  err=0    ft_p50=18352.5ms total_p50=10877.8ms drift_p99=17801.0ms msgs= 0.1 tput= 42.55x

 HTTP+SSE clean c1-50
http c=   1 ok=   2/2    err=0    qf=0    p95_total= 1385.37ms p95_ft= 1385.37ms tput= 7.353x wer=12.291%
http c=   5 ok=  10/10   err=0    qf=0    p95_total= 3536.05ms p95_ft= 3536.05ms tput=15.626x wer=29.632%
http c=  10 ok=  20/20   err=0    qf=0    p95_total= 5825.18ms p95_ft= 5825.18ms tput=16.632x wer=31.019%
http c=  25 ok=  50/50   err=0    qf=0    p95_total=14818.35ms p95_ft=14818.35ms tput=17.206x wer=28.096%
http c=  50 ok= 100/100  err=0    qf=0    p95_total=31326.37ms p95_ft=31326.37ms tput=17.168x wer=26.505%
 sse c=   1 ok=   2/2    err=0    qf=0    p95_total= 1413.05ms p95_ft=  846.29ms tput= 7.092x wer=12.291%
 sse c=   5 ok=  10/10   err=0    qf=0    p95_total=  3638.6ms p95_ft= 1165.68ms tput=16.216x wer=29.632%
 sse c=  10 ok=  20/20   err=0    qf=0    p95_total= 7669.36ms p95_ft= 1605.29ms tput=17.647x wer=31.269%
 sse c=  25 ok=  50/50   err=0    qf=0    p95_total= 17730.7ms p95_ft= 2995.27ms tput=18.635x wer=28.685%
 sse c=  50 ok= 100/100  err=0    qf=0    p95_total=38861.38ms p95_ft=10016.53ms tput=18.737x wer=26.889%

 HTTP+SSE unstable c1-100
http c=   1 ok=   2/2    err=0    qf=0    p95_total= 1155.14ms p95_ft= 1155.14ms tput= 9.461x wer=12.291%
http c=   5 ok=  10/10   err=0    qf=0    p95_total= 4568.35ms p95_ft= 4568.35ms tput=13.403x wer=29.632%
http c=  10 ok=  20/20   err=0    qf=0    p95_total= 7677.31ms p95_ft= 7677.31ms tput=  13.8x wer=31.019%
http c=  25 ok=  50/50   err=0    qf=0    p95_total=17917.03ms p95_ft=17917.03ms tput= 14.01x wer=28.074%
http c=  50 ok= 100/100  err=0    qf=0    p95_total=37150.87ms p95_ft=37150.87ms tput=13.993x wer=26.534%
http c=  75 ok=  58/150  err=92   qf=21   p95_total= 35453.8ms p95_ft= 35453.8ms tput= 13.75x wer=28.399%
http c= 100 ok=   0/200  err=200  qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=   1 ok=   0/2    err=2    qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=   5 ok=   0/10   err=10   qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=  10 ok=   0/20   err=20   qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=  25 ok=   0/50   err=50   qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=  50 ok=   0/100  err=100  qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c=  75 ok=   0/150  err=150  qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%
 sse c= 100 ok=   0/200  err=200  qf=0    p95_total=     0.0ms p95_ft=     0.0ms tput=   0.0x wer=  0.0%

 HTTP autoscale c50-150 r4
http c=  50 ok= 200/200  err=0    qf=0    p95_total=30713.51ms p95_ft=30713.51ms tput=17.373x wer=25.992%
http c=  75 ok= 300/300  err=0    qf=0    p95_total=44009.04ms p95_ft=44009.04ms tput= 17.49x wer=26.163%
http c= 100 ok= 287/400  err=113  qf=59   p95_total=53452.21ms p95_ft=53452.21ms tput=27.035x wer=25.659%
http c= 150 ok= 600/600  err=0    qf=0    p95_total= 39620.0ms p95_ft= 39620.0ms tput=37.356x wer=25.912%

Autoscale details
c=50 ok=200/200 err=0 qf=0 tput=17.373x max_workers_seen=1 worker_ids=[31144514] status_counts={'idle': 84}
c=75 ok=300/300 err=0 qf=0 tput=17.49x max_workers_seen=1 worker_ids=[31144514] status_counts={'idle': 128}
c=100 ok=287/400 err=113 qf=59 tput=27.035x max_workers_seen=2 worker_ids=[31144514, 31144824] status_counts={'pending': 1, 'idle': 95, 'creating': 12, 'loading': 16, 'model_loading': 22, 'error': 1, 'rebooting': 7}
c=150 ok=600/600 err=0 qf=0 tput=37.356x max_workers_seen=3 worker_ids=[31144514, 31144824, 31144875] status_counts={'idle': 159, 'rebooting': 6, 'starting': 16, 'error': 2, 'destroying': 1, 'model_loading': 56, 'creating': 14, 'loading': 18, 'stopping': 1}


**Analyzing concurrency and scaling limits**


**Querying endpoints with redaction**


python3 - <<'PY'
import requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip(); break
h={'Accept':'application/json','Authorization':f'Bearer {k}'}
api='https://console.vast.ai/api/v0'; run='https://run.vast.ai'
endpoints=[('hindi-asr-prod',11967),('hindi-asr-ws-bench',11995),('hindi-asr-h100-bench',11998)]
# map endpoint keys
res=requests.get(f'{api}/endptjobs/',headers=h).json().get('results',[])
keys={e['endpoint_name']:e['api_key'] for e in res}
ids={e['endpoint_name']:e['id'] for e in res}
for name,_ in endpoints:
    if name not in ids: continue
    eid=ids[name]; ekey=keys[name]
    workers=requests.post(f'{run}/get_endpoint_workers/',json={'id':eid,'api_key':ekey}).json()
    if isinstance(workers,dict) and 'workers' in workers: workers=workers['workers']
    print('\n',name,'id',eid,'workers',len(workers))
    for w in workers:
        iid=w.get('id'); st=w.get('status'); ld=w.get('cur_load')
        inst=requests.get(f'{api}/instances/{iid}/',headers=h).json().get('instances')
        if isinstance(inst,dict):
            print(' ',iid,st,'load',ld,'gpu',inst.get('gpu_name'),'state',inst.get('actual_status'),'img',inst.get('image_uuid'),'geo',inst.get('geolocation'))
        else:
            print(' ',iid,st,'load',ld,'instance=None')
PY


Output:
Chunk ID: 50c355
Wall time: 2.9283 seconds
Process exited with code 0
Original token count: 248
Output:

 hindi-asr-prod id 11967 workers 5
  31141471 idle load 0.0 gpu RTX PRO 6000 WS state running img bharathkumar192/hindi-nemotron-asr:v10-production geo Alberta, CA
  31144458 idle load 0.0 gpu RTX PRO 6000 S state running img bharathkumar192/hindi-nemotron-asr:v10-production geo Michigan, US
  31140781 stopped load 0.0 gpu RTX PRO 6000 S state exited img bharathkumar192/hindi-nemotron-asr:v10-production geo Texas, US
  31143658 stopped load 0.0 gpu RTX PRO 6000 S state exited img bharathkumar192/hindi-nemotron-asr:v10-production geo Texas, US
  31141312 stopped load 0.0 gpu H100 SXM state exited img bharathkumar192/hindi-nemotron-asr:v10-production geo Montana, US

 hindi-asr-ws-bench id 11995 workers 1
  31143842 idle load 0.0 gpu H100 SXM state running img bharathkumar192/hindi-nemotron-asr:v13-production geo , US

 hindi-asr-h100-bench id 11998 workers 1
  31145113 idle load 0.0 gpu H100 NVL state running img bharathkumar192/hindi-nemotron-asr:v13-production geo Kansas, US


**Planning worker group status collection**


VAST_API_KEY=$(python3 - <<'PY'
from pathlib import Path
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): print(line.split('=',1)[1].strip()); break
PY
); ~/.local/bin/vastai --api-key "$VAST_API_KEY" show workergroups --raw | python3 - <<'PY'
import sys,json
arr=json.load(sys.stdin)
for w in arr:
    if w.get('endpoint_name') in ['hindi-asr-ws-bench','hindi-asr-h100-bench']:
        print({'id':w['id'],'endpoint':w['endpoint_name'],'template_hash':w['template_hash'],'gpu_ram':w['gpu_ram'],'target_util':w['target_util'],'max_ports':w.get('search_query',{}).get('direct_port_count'),'gpu_name_filter':w.get('search_query',{}).get('gpu_name')})
PY


Output:
Chunk ID: 696f2b
Wall time: 0.3298 seconds
Process exited with code 1
Original token count: 182
Output:
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe


**Switching to requests for workergroups**


python3 - <<'PY'
import requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip(); break
h={'Accept':'application/json','Authorization':f'Bearer {k}'}
d=requests.get('https://console.vast.ai/api/v0/workergroups/',headers=h).json().get('results',[])
for w in d:
    if w.get('endpoint_name') in ['hindi-asr-ws-bench','hindi-asr-h100-bench']:
        print({'id':w['id'],'endpoint':w['endpoint_name'],'template_hash':w['template_hash'],'template_id':w['template_id'],'gpu_ram':w['gpu_ram'],'target_util':w['target_util'],'search_gpu_name':w.get('search_query',{}).get('gpu_name')})
PY


Output:
Chunk ID: db465a
Wall time: 0.2343 seconds
Process exited with code 0
Original token count: 114
Output:
{'id': 16824, 'endpoint': 'hindi-asr-ws-bench', 'template_hash': '4f07b7ec6cffedfe18bde28236c5c373', 'template_id': 342197, 'gpu_ram': 80.0, 'target_util': 0.7, 'search_gpu_name': {'in': ['H100 NVL', 'H100 SXM', 'H100 PCIe']}}
{'id': 16830, 'endpoint': 'hindi-asr-h100-bench', 'template_hash': '4f07b7ec6cffedfe18bde28236c5c373', 'template_id': 342197, 'gpu_ram': 80.0, 'target_util': 0.7, 'search_gpu_name': {'in': ['H100 NVL', 'H100 SXM', 'H100 PCIe']}}


**Creating detailed benchmark report**


I’ve gathered all benchmark artifacts and I’m writing a concise H100 serverless report in the repo with concrete scaling rules, failure points, and next engineering actions so we can iterate from hard data instead of anecdotes.


cat > benchmark_results/h100_serverless_report_20260210.md <<'MD'
# H100 Serverless Benchmark Report (2026-02-10)

## Scope
- Model/service: `bharathkumar192/hindi-nemotron-asr:v13-production` (Hindi fine-tuned model)
- Platform: Vast.ai serverless
- Benchmark endpoints:
  - `hindi-asr-ws-bench` (H100 SXM)
  - `hindi-asr-h100-bench` (H100 NVL)
- Audio quality set: FLEURS Hindi (real speech) for HTTP/SSE quality runs

## Artifacts
- WS chunk sweeps:
  - `benchmark_results/ws_h100_v13_chunk160_c1_300.csv`
  - `benchmark_results/ws_h100_v13_chunk160_c1_300.json`
  - `benchmark_results/ws_h100_v13_chunk320_c1_127.csv`
  - `benchmark_results/ws_h100_v13_chunk320_c1_127.json`
  - `benchmark_results/ws_h100_v13_chunk560_c1_127.csv`
  - `benchmark_results/ws_h100_v13_chunk560_c1_127.json`
- HTTP+SSE clean baseline (H100):
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean_details.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean.json`
- HTTP autoscaling stress:
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4.csv`
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4_details.csv`
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4.json`
- Unstable overload run (documented failure mode):
  - `benchmark_results/full_h100_v13_fleurs_c1_100.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_100_details.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_100.json`

## Key Results

### 1) WebSocket (single worker, direct mapped port)
- Throughput increased with concurrency, but stream behavior was not true incremental streaming:
  - `msgs_per_stream_avg` stayed around `1.0` (or lower at high load)
  - first-token latency scaled toward end-of-utterance instead of remaining low
- 160ms sweep:
  - Stable through `c=127`, then hard failure from `c=160+`
  - At `c=127`: `first_token_p50 ~18.7s`, `drift_p99 ~22.1s`
- 320ms/560ms sweeps showed same structural pattern: non-incremental output and high drift growth.

Interpretation: current WS path is not matching NVIDIA’s benchmark condition (continuous low-drift partial streaming).

### 2) HTTP+SSE clean baseline (H100, FLEURS Hindi)
- Zero errors through `c=50` for both HTTP and SSE.
- Throughput (realtime factor):
  - HTTP: ~`17.2x` at `c=50`
  - SSE: ~`18.7x` at `c=50`
- Quality (FLEURS subset):
  - WER roughly `26%–31%` in loaded conditions
  - CER roughly in the same band as prior runs (see details CSV/JSON)
- VRAM used stayed near ~`10.45 GB`, indicating compute/scheduling/network path dominates before memory.

### 3) HTTP autoscaling stress (r=4 requests/client)
- `c=50` and `c=75`: zero errors, one worker observed.
- `c=100`: partial failure (`113` errors, `59` queue-full) while autoscaler created additional workers.
- `c=150`: recovered to zero errors once more workers were active.
- Worker samples confirm scale-out happened after overload onset (lag window), not before it.

Interpretation: autoscaler works, but reactive scale-up lag causes transient failures at burst boundaries.

## Production Failure Modes Observed
1. WS stream path emitted near-final-only outputs (not rich partial stream), inflating drift.
2. Under high burst load, worker entered restart/reboot cycles in one run.
3. Queue-full spikes occurred before additional workers became ready (scale lag).
4. Endpoint auth path (`/route` -> worker) showed transient 401 behavior on one recycled endpoint state; resolved by fresh endpoint/workergroup.

## Recommended Scaling Rules (HTTP/SSE, current stack)
1. Single-worker safe zone: keep per-worker effective concurrency around `<= 75` requests in flight for reliability.
2. Preferred quality+latency zone: target `<= 50` concurrent requests per worker.
3. Scale-out trigger: pre-scale before hitting `c=75` equivalent load; do not wait for queue-full events.
4. Capacity planning: if you need sustained `c=150` behavior, maintain at least `3` warm-capable workers (or equivalent cold buffer).
5. Endpoint config guardrails:
   - keep `max_workers >= 4` for burst absorption
   - increase cold capacity (`cold_workers`) when you expect sharp demand spikes
   - tune `target_util` lower (more headroom) if queue-full events recur

## Why This Is Below NVIDIA Blog Numbers
- Their reported benchmarks are on true cache-aware streaming conditions with low-drift partial output behavior.
- Current WS behavior here is not equivalent yet (`~1 msg/stream`), so concurrency numbers are not apples-to-apples.
- Hardware class differences (H100 SXM/NVL vs mixed offers), serverless control-plane delays, and endpoint routing behavior add additional variance.

## Next Actions
1. Fix WS incremental emission path and verify `msgs_per_stream` rises with utterance duration.
2. Re-run apples-to-apples WS test matrix (3-minute real-time sessions, chunk mode parity, drift curves).
3. Add pre-warm autoscaling policy tuned from observed lag window.
4. Recompute scaling law once WS path is corrected.
MD


Output:
Chunk ID: eff87e
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 0
Output:


Summary: auto


**Considering file existence check**


ls -l benchmark_results/h100_serverless_report_20260210.md && sed -n '1,220p' benchmark_results/h100_serverless_report_20260210.md


Output:
Chunk ID: d14318
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 1257
Output:
-rw-rw-r-- 1 ubuntu ubuntu 4929 Feb 10 02:53 benchmark_results/h100_serverless_report_20260210.md
# H100 Serverless Benchmark Report (2026-02-10)

## Scope
- Model/service: `bharathkumar192/hindi-nemotron-asr:v13-production` (Hindi fine-tuned model)
- Platform: Vast.ai serverless
- Benchmark endpoints:
  - `hindi-asr-ws-bench` (H100 SXM)
  - `hindi-asr-h100-bench` (H100 NVL)
- Audio quality set: FLEURS Hindi (real speech) for HTTP/SSE quality runs

## Artifacts
- WS chunk sweeps:
  - `benchmark_results/ws_h100_v13_chunk160_c1_300.csv`
  - `benchmark_results/ws_h100_v13_chunk160_c1_300.json`
  - `benchmark_results/ws_h100_v13_chunk320_c1_127.csv`
  - `benchmark_results/ws_h100_v13_chunk320_c1_127.json`
  - `benchmark_results/ws_h100_v13_chunk560_c1_127.csv`
  - `benchmark_results/ws_h100_v13_chunk560_c1_127.json`
- HTTP+SSE clean baseline (H100):
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean_details.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_50_clean.json`
- HTTP autoscaling stress:
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4.csv`
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4_details.csv`
  - `benchmark_results/autoscale_http_h100_v13_c50_150_r4.json`
- Unstable overload run (documented failure mode):
  - `benchmark_results/full_h100_v13_fleurs_c1_100.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_100_details.csv`
  - `benchmark_results/full_h100_v13_fleurs_c1_100.json`

## Key Results

### 1) WebSocket (single worker, direct mapped port)
- Throughput increased with concurrency, but stream behavior was not true incremental streaming:
  - `msgs_per_stream_avg` stayed around `1.0` (or lower at high load)
  - first-token latency scaled toward end-of-utterance instead of remaining low
- 160ms sweep:
  - Stable through `c=127`, then hard failure from `c=160+`
  - At `c=127`: `first_token_p50 ~18.7s`, `drift_p99 ~22.1s`
- 320ms/560ms sweeps showed same structural pattern: non-incremental output and high drift growth.

Interpretation: current WS path is not matching NVIDIA’s benchmark condition (continuous low-drift partial streaming).

### 2) HTTP+SSE clean baseline (H100, FLEURS Hindi)
- Zero errors through `c=50` for both HTTP and SSE.
- Throughput (realtime factor):
  - HTTP: ~`17.2x` at `c=50`
  - SSE: ~`18.7x` at `c=50`
- Quality (FLEURS subset):
  - WER roughly `26%–31%` in loaded conditions
  - CER roughly in the same band as prior runs (see details CSV/JSON)
- VRAM used stayed near ~`10.45 GB`, indicating compute/scheduling/network path dominates before memory.

### 3) HTTP autoscaling stress (r=4 requests/client)
- `c=50` and `c=75`: zero errors, one worker observed.
- `c=100`: partial failure (`113` errors, `59` queue-full) while autoscaler created additional workers.
- `c=150`: recovered to zero errors once more workers were active.
- Worker samples confirm scale-out happened after overload onset (lag window), not before it.

Interpretation: autoscaler works, but reactive scale-up lag causes transient failures at burst boundaries.

## Production Failure Modes Observed
1. WS stream path emitted near-final-only outputs (not rich partial stream), inflating drift.
2. Under high burst load, worker entered restart/reboot cycles in one run.
3. Queue-full spikes occurred before additional workers became ready (scale lag).
4. Endpoint auth path (`/route` -> worker) showed transient 401 behavior on one recycled endpoint state; resolved by fresh endpoint/workergroup.

## Recommended Scaling Rules (HTTP/SSE, current stack)
1. Single-worker safe zone: keep per-worker effective concurrency around `<= 75` requests in flight for reliability.
2. Preferred quality+latency zone: target `<= 50` concurrent requests per worker.
3. Scale-out trigger: pre-scale before hitting `c=75` equivalent load; do not wait for queue-full events.
4. Capacity planning: if you need sustained `c=150` behavior, maintain at least `3` warm-capable workers (or equivalent cold buffer).
5. Endpoint config guardrails:
   - keep `max_workers >= 4` for burst absorption
   - increase cold capacity (`cold_workers`) when you expect sharp demand spikes
   - tune `target_util` lower (more headroom) if queue-full events recur

## Why This Is Below NVIDIA Blog Numbers
- Their reported benchmarks are on true cache-aware streaming conditions with low-drift partial output behavior.
- Current WS behavior here is not equivalent yet (`~1 msg/stream`), so concurrency numbers are not apples-to-apples.
- Hardware class differences (H100 SXM/NVL vs mixed offers), serverless control-plane delays, and endpoint routing behavior add additional variance.

## Next Actions
1. Fix WS incremental emission path and verify `msgs_per_stream` rises with utterance duration.
2. Re-run apples-to-apples WS test matrix (3-minute real-time sessions, chunk mode parity, drift curves).
3. Add pre-warm autoscaling policy tuned from observed lag window.
4. Recompute scaling law once WS path is corrected.


**Weighing patch deployment and cleanup**


ps -eo pid,etimes,cmd | rg 'python3 -u - <<' | rg -v rg


Output:
Chunk ID: bacbcc
Wall time: 0.0514 seconds
Process exited with code 1
Original token count: 0
Output:


**Crafting final benchmark summary**


You’re right to challenge the `~75` number. I re-ran this on H100 serverless and debugged the bottlenecks end-to-end.

**What I ran (Feb 10, 2026)**
- Pinned serverless workergroups to H100-only offers and benchmarked on:
- `hindi-asr-ws-bench` (H100 SXM)
- `hindi-asr-h100-bench` (H100 NVL)
- Used your Hindi model (`v13-production`) and FLEURS Hindi for WER/CER.
- Full report written to `benchmark_results/h100_serverless_report_20260210.md`.

**Key findings**
- No, this is not reproducing NVIDIA blog conditions yet.
- Biggest blocker: WS path behaves like near-final output, not true incremental streaming.
- Evidence: `msgs_per_stream_avg` stays ~`1.0` (or below) across WS sweeps.
- That causes “first token” and drift to blow up with concurrency, so comparisons vs NVIDIA’s low-drift WS benchmark are not apples-to-apples.

**H100 results (current stack)**
- WS (single worker, direct port):
- Stable up to `c=127`, hard fail from `c=160+` in 160ms sweep.
- At `c=127` (160ms): first-token p50 ~18.7s, drift p99 ~22.1s.
- HTTP+SSE (clean run, H100, FLEURS, c=1..50):
- Zero errors through `c=50`.
- HTTP throughput at `c=50`: ~`17.17x` realtime.
- SSE throughput at `c=50`: ~`18.74x` realtime.
- WER roughly `26%–31%` in loaded conditions.
- HTTP autoscaling run (c=50,75,100,150; 4 req/client):
- `c=50` and `c=75`: 0 errors, single worker.
- `c=100`: transient failures (`113` errors, `59` queue-full) while scale-out started.
- `c=150`: recovered to 0 errors once more workers were active (max 3 workers seen).

**Why your 500-stream expectation doesn’t match yet**
- NVIDIA’s numbers were under true cache-aware incremental WS behavior and specific benchmark harness assumptions.
- Your current WS service path is not emitting frequent partials (primary bottleneck).
- Serverless adds scale-up lag; burst tiers can queue-fail before new workers are ready.
- Hardware/offer variance also matters (H100 SXM vs NVL, host/network differences).

**Scaling rules you can use now (HTTP/SSE prod)**
- Safe per-worker zone: `<= 50` concurrent in-flight requests.
- Upper practical single-worker zone: `~75` (beyond that, pre-scale).
- Pre-warm rule: expected concurrency `C` => warm workers `ceil(C/50)`.
- Keep `max_workers >= 4` and raise cold capacity for bursty traffic.
- Trigger scale-out before queue-full (don’t wait for failures).

**Code changes made (WS improvement path, not deployed yet)**
- Patched `production/server.py` to improve WS incremental handling:
- `production/server.py:64` (new pipeline batching env knobs)
- `production/server.py:240` (streaming config uses those knobs)
- `production/server.py:883` (feed smaller frame chunks instead of coarse buffering)
- `production/server.py:1036` (prefer final/partial/step transcript fallbacks)
- Built/pushed `v14-production`, but I rolled workergroups back to stable `v13` to keep endpoints healthy during this phase.

**Current live status**
- `hindi-asr-h100-bench` is healthy on H100 NVL, `v13-production`.
- `hindi-asr-ws-bench` is healthy on H100 SXM, `v13-production`.
- H100-only filters are active on both benchmark workergroups.

Next step I recommend:
1. Deploy the WS patch on a dedicated endpoint and re-run the same H100 WS sweeps to verify `msgs_per_stream` rises above 1 and drift drops.
2. If that passes, run 3-minute real-time WS parity runs (closer to NVIDIA’s condition) and refit scaling laws.