Output:
Chunk ID: 4991d8
Wall time: 22.2639 seconds
Process exited with code 0
Original token count: 257
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['sse']
Levels: [100]
Audio source: fleurs (pool=240)
Requests/client: 2

==========================================================================================
Level mode=sse c=100 reqs/client=2
==========================================================================================
OK=196/200 err=4 qfull=0 tput=9.45x total_p95=180267ms ft_p95=180267ms wer=29.74%

==========================================================================================
SUMMARY
==========================================================================================
 sse c=100 ok= 196/200  err=  4 qfull=  0 p95_total=180266.9ms p95_ft=180266.9ms tput=  9.45x wer= 29.74%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck.json


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Confirming HTTP load test results**


python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode http --levels 100 --requests-per-client 2 --audio-source fleurs --sample-pool 240 --min-duration 3 --max-duration 20 --shuffle-samples --seed 202 --metrics-interval 2 --cooldown 0 --output-prefix sustained_v10_http_c100_r2_recheck


Output:
Chunk ID: e902b3
Wall time: 30.0011 seconds
Process running with session ID 30777
Original token count: 0
Output:


Summary: auto


**Estimating poll duration**


Output:
Chunk ID: ec2531
Wall time: 25.8657 seconds
Process exited with code 0
Original token count: 258
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['http']
Levels: [100]
Audio source: fleurs (pool=240)
Requests/client: 2

==========================================================================================
Level mode=http c=100 reqs/client=2
==========================================================================================
OK=200/200 err=0 qfull=0 tput=37.73x total_p95=30590ms ft_p95=30590ms wer=24.53%

==========================================================================================
SUMMARY
==========================================================================================
http c=100 ok= 200/200  err=  0 qfull=  0 p95_total= 30589.8ms p95_ft= 30589.8ms tput= 37.73x wer= 24.53%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_http_c100_r2_recheck.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_http_c100_r2_recheck_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_http_c100_r2_recheck.json


**Parsing new outputs and setting autoscaling rules**


The main and sustained datasets are complete. I’m now aggregating all new JSON/CSV artifacts into one scaling summary: latency curves, WER drift, throughput ceilings, error signatures, and autoscaler behavior under overload.


python3 - <<'PY'
import json,glob,os
files=sorted(glob.glob('benchmark_results/full_v10_fleurs_c1_100_r2.json')+glob.glob('benchmark_results/sustained_v10_*.json'))
for f in files:
    data=json.load(open(f))
    print('\n===',os.path.basename(f),'entries',len(data))
    for item in data:
        s=item['summary']
        print(f" {s['mode']} c={s['concurrency']} req/client={s['requests_per_client']} ok={s['ok']} err={s['errors']} qf={s['queue_full_errors']} "+
              f"tput={s['throughput_x']:.2f}x p95_total={s['total_ms_p95']:.0f} p95_ft={s['first_token_ms_p95']:.0f} wer={s['wer_pct']:.2f} cer={s['cer_pct']:.2f} "+
              f"vram_max={s.get('vram_used_gb_max',0):.2f} worker_load_max={s.get('worker_cur_load_max',0):.1f} p95_load={s.get('worker_cur_load_p95',0):.1f}")
PY


python3 - <<'PY'
import json
from collections import Counter
f='benchmark_results/full_v10_fleurs_c1_100_r2.json'
data=json.load(open(f))
for item in data:
    s=item['summary']
    if s['errors']:
        c=Counter()
        for r in item['records']:
            if not r['ok']:
                msg=(r.get('error') or '')[:120]
                c[msg]+=1
        print('---',s['mode'],'c',s['concurrency'],'errors',s['errors'])
        for k,v in c.most_common(10):
            print(v,k)
PY


python3 - <<'PY'
import json
import statistics as st
f='benchmark_results/full_v10_fleurs_c1_100_r2.json'
data=json.load(open(f))
# inspect worker sample behavior at worst level sse c100
for item in data:
    s=item['summary']
    if s['mode']=='sse' and s['concurrency']==100:
        ws=item['worker_samples']
        statuses=[]
        loads=[]
        for snap in ws:
            for w in snap.get('workers',[]):
                statuses.append(w.get('status'))
                try: loads.append(float(w.get('cur_load',0)))
                except: pass
        from collections import Counter
        print('worker snapshots',len(ws),'status_counts',Counter(statuses))
        if loads:
            print('load max',max(loads),'p95',sorted(loads)[int(0.95*len(loads))-1])
PY


Output:
Chunk ID: a76251
Wall time: 0.0505 seconds
Process exited with code 0
Original token count: 849
Output:

=== full_v10_fleurs_c1_100_r2.json entries 14
 http c=1 req/client=2 ok=2 err=0 qf=0 tput=6.91x p95_total=1185 p95_ft=1185 wer=21.23 cer=9.64 vram_max=10.43 worker_load_max=20.0 p95_load=20.0
 http c=5 req/client=2 ok=10 err=0 qf=0 tput=28.91x p95_total=2175 p95_ft=2175 wer=21.03 cer=7.82 vram_max=10.43 worker_load_max=70.0 p95_load=70.0
 http c=10 req/client=2 ok=20 err=0 qf=0 tput=33.17x p95_total=4143 p95_ft=4143 wer=21.92 cer=9.57 vram_max=10.43 worker_load_max=110.0 p95_load=110.0
 http c=25 req/client=2 ok=50 err=0 qf=0 tput=36.52x p95_total=8453 p95_ft=8453 wer=26.11 cer=12.46 vram_max=10.43 worker_load_max=270.0 p95_load=270.0
 http c=50 req/client=2 ok=100 err=0 qf=0 tput=37.48x p95_total=15370 p95_ft=15370 wer=25.12 cer=11.49 vram_max=10.43 worker_load_max=610.0 p95_load=610.0
 http c=75 req/client=2 ok=150 err=0 qf=0 tput=37.46x p95_total=23891 p95_ft=23891 wer=24.82 cer=11.89 vram_max=10.43 worker_load_max=830.0 p95_load=820.0
 http c=100 req/client=2 ok=200 err=0 qf=0 tput=37.51x p95_total=32123 p95_ft=32123 wer=24.47 cer=11.23 vram_max=10.43 worker_load_max=1090.0 p95_load=1070.0
 sse c=1 req/client=2 ok=2 err=0 qf=0 tput=9.46x p95_total=1137 p95_ft=952 wer=21.23 cer=9.64 vram_max=10.43 worker_load_max=20.0 p95_load=20.0
 sse c=5 req/client=2 ok=10 err=0 qf=0 tput=26.30x p95_total=2575 p95_ft=1224 wer=21.03 cer=7.82 vram_max=10.43 worker_load_max=110.0 p95_load=110.0
 sse c=10 req/client=2 ok=20 err=0 qf=0 tput=35.16x p95_total=4314 p95_ft=1596 wer=22.28 cer=9.68 vram_max=10.43 worker_load_max=140.0 p95_load=140.0
 sse c=25 req/client=2 ok=50 err=0 qf=0 tput=38.98x p95_total=10335 p95_ft=3651 wer=26.80 cer=12.91 vram_max=10.43 worker_load_max=320.0 p95_load=320.0
 sse c=50 req/client=2 ok=100 err=0 qf=0 tput=40.03x p95_total=19637 p95_ft=5218 wer=25.56 cer=11.92 vram_max=10.43 worker_load_max=570.0 p95_load=570.0
 sse c=75 req/client=2 ok=150 err=0 qf=0 tput=40.20x p95_total=30527 p95_ft=9484 wer=25.33 cer=12.32 vram_max=10.43 worker_load_max=820.0 p95_load=820.0
 sse c=100 req/client=2 ok=159 err=41 qf=0 tput=8.40x p95_total=180876 p95_ft=180876 wer=37.35 cer=26.54 vram_max=10.43 worker_load_max=1130.0 p95_load=10.0

=== sustained_v10_http_c100_r2_recheck.json entries 1
 http c=100 req/client=2 ok=200 err=0 qf=0 tput=37.73x p95_total=30590 p95_ft=30590 wer=24.53 cer=11.18 vram_max=10.43 worker_load_max=1090.0 p95_load=1060.0

=== sustained_v10_http_c50_r6.json entries 1
 http c=50 req/client=6 ok=300 err=0 qf=0 tput=51.76x p95_total=17081 p95_ft=17081 wer=23.67 cer=10.91 vram_max=10.43 worker_load_max=560.0 p95_load=510.0

=== sustained_v10_sse_c100_r1.json entries 1
 sse c=100 req/client=1 ok=100 err=0 qf=0 tput=38.88x p95_total=27783 p95_ft=15212 wer=26.18 cer=13.24 vram_max=10.43 worker_load_max=1010.0 p95_load=1000.0

=== sustained_v10_sse_c100_r2_recheck.json entries 1
 sse c=100 req/client=2 ok=196 err=4 qf=0 tput=9.45x p95_total=180267 p95_ft=180267 wer=29.74 cer=16.88 vram_max=10.43 worker_load_max=1120.0 p95_load=930.0

=== sustained_v10_sse_c50_r6.json entries 1
 sse c=50 req/client=6 ok=300 err=0 qf=0 tput=39.71x p95_total=22071 p95_ft=3935 wer=24.18 cer=11.46 vram_max=10.43 worker_load_max=610.0 p95_load=590.0

=== sustained_v10_sse_c75_r3.json entries 1
 sse c=75 req/client=3 ok=225 err=0 qf=0 tput=39.45x p95_total=32516 p95_ft=9122 wer=24.86 cer=11.75 vram_max=10.43 worker_load_max=890.0 p95_load=840.0


Output:
Chunk ID: 10f099
Wall time: 0.0510 seconds
Process exited with code 0
Original token count: 314
Output:
--- sse c 100 errors 41
1 route failed: {'__request_id': '8e407ec5-940c-4f27-bb36-8ffd681b6217', 'endpoint': 'hindi-asr-prod', 'request_idx': 2241
1 route failed: {'__request_id': '1527c60e-bfca-4cf3-a1d7-0459c9e3e5e4', 'endpoint': 'hindi-asr-prod', 'request_idx': 2242
1 route failed: {'__request_id': '964922ea-bb13-4c38-a62b-1031930d73fd', 'endpoint': 'hindi-asr-prod', 'request_idx': 2243
1 route failed: {'__request_id': 'f3df8d4d-3fa9-4a18-8a99-31ab2fb163f8', 'endpoint': 'hindi-asr-prod', 'request_idx': 2244
1 route failed: {'__request_id': '19604311-72d6-448b-956e-062f5aa43f53', 'endpoint': 'hindi-asr-prod', 'request_idx': 2245
1 route failed: {'__request_id': '6e62bdf8-3168-4db8-8e40-05df73446481', 'endpoint': 'hindi-asr-prod', 'request_idx': 2247
1 route failed: {'__request_id': 'fff534aa-3c8b-4a8f-9b4d-2bd5298bb0ab', 'endpoint': 'hindi-asr-prod', 'request_idx': 2248
1 route failed: {'__request_id': '02f54a32-49a8-44fa-aa62-720a6eef82ca', 'endpoint': 'hindi-asr-prod', 'request_idx': 2249
1 route failed: {'__request_id': '20272efa-8ad1-4a63-9798-d3fd621a717b', 'endpoint': 'hindi-asr-prod', 'request_idx': 2250
1 route failed: {'__request_id': '023e7687-d8aa-47dd-91fd-5656c0febfdd', 'endpoint': 'hindi-asr-prod', 'request_idx': 2252


Output:
Chunk ID: 53746b
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 49
Output:
worker snapshots 91 status_counts Counter({'loading': 204, 'idle': 67, 'model_loading': 30, 'created': 28, 'creating': 25, 'error': 16, 'pending': 2, 'destroying': 1})
load max 1130.0 p95 10.0


**Analyzing detailed error messages**


python3 - <<'PY'
import json
f='benchmark_results/full_v10_fleurs_c1_100_r2.json'
data=json.load(open(f))
for item in data:
    s=item['summary']
    if s['mode']=='sse' and s['concurrency']==100:
        errs=[r['error'] for r in item['records'] if not r['ok']]
        print('errors',len(errs))
        for e in errs[:8]:
            print(e)
        break
PY


Output:
Chunk ID: cfbd24
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 417
Output:
errors 41
route failed: {'__request_id': '8e407ec5-940c-4f27-bb36-8ffd681b6217', 'endpoint': 'hindi-asr-prod', 'request_idx': 2241, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': '1527c60e-bfca-4cf3-a1d7-0459c9e3e5e4', 'endpoint': 'hindi-asr-prod', 'request_idx': 2242, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': '964922ea-bb13-4c38-a62b-1031930d73fd', 'endpoint': 'hindi-asr-prod', 'request_idx': 2243, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': 'f3df8d4d-3fa9-4a18-8a99-31ab2fb163f8', 'endpoint': 'hindi-asr-prod', 'request_idx': 2244, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': '19604311-72d6-448b-956e-062f5aa43f53', 'endpoint': 'hindi-asr-prod', 'request_idx': 2245, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': '6e62bdf8-3168-4db8-8e40-05df73446481', 'endpoint': 'hindi-asr-prod', 'request_idx': 2247, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': 'fff534aa-3c8b-4a8f-9b4d-2bd5298bb0ab', 'endpoint': 'hindi-asr-prod', 'request_idx': 2248, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}
route failed: {'__request_id': '02f54a32-49a8-44fa-aa62-720a6eef82ca', 'endpoint': 'hindi-asr-prod', 'request_idx': 2249, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}


**Identifying autoscaler churn cause**


python3 - <<'PY'
import json
f='benchmark_results/sustained_v10_sse_c100_r2_recheck.json'
data=json.load(open(f))
item=data[0]
errs=[r for r in item['records'] if not r['ok']]
print('errors',len(errs))
for r in errs:
    print('status',r['status_code'],'err',r['error'][:220])
PY


Output:
Chunk ID: 3187a0
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 223
Output:
errors 4
status 0 err route failed: {'__request_id': 'c8aba9d8-11c5-4f3e-840b-8d6e367a7251', 'endpoint': 'hindi-asr-prod', 'request_idx': 3601, 'status': 'total workers: 3 loading workers: 1 standby workers: 0 error workers: 1'}
status 0 err route failed: {'__request_id': '68364ef8-dd38-4c1a-84cf-23693be88b13', 'endpoint': 'hindi-asr-prod', 'request_idx': 3602, 'status': 'total workers: 2 loading workers: 1 standby workers: 0 error workers: 0'}
status 0 err route failed: {'__request_id': '92c0e728-1500-4675-a9ac-b1a32dd49df6', 'endpoint': 'hindi-asr-prod', 'request_idx': 3604, 'status': 'total workers: 2 loading workers: 1 standby workers: 0 error workers: 0'}
status 0 err route failed: {'__request_id': '2d1e5d05-62e5-4d24-adef-f9ed8fbbcd42', 'endpoint': 'hindi-asr-prod', 'request_idx': 3605, 'status': 'total workers: 2 loading workers: 1 standby workers: 0 error workers: 0'}


**Analyzing request duration distribution**


python3 - <<'PY'
import csv,statistics
path='benchmark_results/sustained_v10_sse_c100_r2_recheck_details.csv'
rows=list(csv.DictReader(open(path)))
vals=[float(r['total_ms']) for r in rows if r['ok']=='True']
vals_sorted=sorted(vals)
print('n',len(vals),'min',vals_sorted[0],'p50',vals_sorted[len(vals)//2],'p95',vals_sorted[int(len(vals)*0.95)-1],'p99',vals_sorted[int(len(vals)*0.99)-1],'max',vals_sorted[-1])
# count over thresholds
for th in [30000,60000,120000,180000]:
    c=sum(1 for v in vals if v>=th)
    print('>=',th,c)
# show top 8 entries
rows_ok=[r for r in rows if r['ok']=='True']
rows_ok.sort(key=lambda r: float(r['total_ms']), reverse=True)
for r in rows_ok[:8]:
    print('client',r['client_idx'],'req',r['req_idx'],'total',r['total_ms'],'first',r['first_token_ms'],'status',r['status_code'],'err',r['error'][:80])
PY


Output:
Chunk ID: 013b9c
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 205
Output:
n 196 min 1135.8662229031324 p50 25452.23716273904 p95 180105.5681416765 p99 180802.112467587 max 181033.32439996302
>= 30000 77
>= 60000 11
>= 120000 11
>= 180000 11
client 96 req 1 total 181033.32439996302 first 181033.32439996302 status 200 err
client 73 req 1 total 180824.34107270092 first 180824.34107270092 status 200 err
client 43 req 1 total 180802.112467587 first 180802.112467587 status 200 err
client 91 req 1 total 180723.00837934017 first 180723.00837934017 status 200 err
client 83 req 1 total 180722.57518209517 first 180722.57518209517 status 200 err
client 18 req 1 total 180722.0215946436 first 180722.0215946436 status 200 err
client 21 req 1 total 180388.45834787935 first 180388.45834787935 status 200 err
client 70 req 1 total 180315.58490544558 first 180315.58490544558 status 200 err


**Analyzing silent 180s responses**


python3 - <<'PY'
import csv
rows=list(csv.DictReader(open('benchmark_results/sustained_v10_sse_c100_r2_recheck_details.csv')))
rows_ok=[r for r in rows if r['ok']=='True' and float(r['total_ms'])>180000]
print('count',len(rows_ok))
for r in rows_ok[:5]:
    print('sample',r['sample_id'],'hyp_len',len(r['hypothesis']),'wer',r['wer'],'first',r['first_token_ms'],'total',r['total_ms'])
PY


Output:
Chunk ID: 3d0aba
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 102
Output:
count 11
sample 1938 hyp_len 0 wer 1.0 first 180723.00837934017 total 180723.00837934017
sample 1954 hyp_len 0 wer 1.0 first 180722.57518209517 total 180722.57518209517
sample 1803 hyp_len 0 wer 1.0 first 180105.5681416765 total 180105.5681416765
sample 1823 hyp_len 0 wer 1.0 first 180722.0215946436 total 180722.0215946436
sample 1718 hyp_len 0 wer 1.0 first 180313.85784875602 total 180313.85784875602


**Identifying SSE empty output bug**


I found a correctness bug in the new stress harness: SSE requests returning HTTP 200 with no transcript were counted as success, which inflated `ok` at extreme load. I’m patching the harness to mark “no events/no done frame” as failure and then re-running the affected `SSE c100 r2` check.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/nemo_hindi/production/test_stress_full.py
@@
 async def run_sse_request(
@@
-                got_first = False
+                got_first = False
+                saw_any_event = False
+                saw_done = False
                 last_text = ""
                 async for line in r.content:
                     l = line.decode(errors="ignore").strip()
                     if not l.startswith("data: "):
                         continue
                     try:
                         d = json.loads(l[6:])
                     except Exception:
                         continue
+                    saw_any_event = True
                     txt = str(d.get("text", "") or "")
                     if txt:
                         last_text = txt
                         if not got_first:
                             first_token_ms = (time.perf_counter() - t0) * 1000.0
                             got_first = True
                     if d.get("done"):
+                        saw_done = True
                         break
                 hypothesis = last_text
+                if not saw_any_event or not saw_done:
+                    raise RuntimeError("sse stream incomplete: no terminal done event")
         else:
             if not direct_url:
                 raise RuntimeError("direct_url required when endpoint is not set")
@@
-                got_first = False
+                got_first = False
+                saw_any_event = False
+                saw_done = False
                 last_text = ""
                 async for line in r.content:
                     l = line.decode(errors="ignore").strip()
                     if not l.startswith("data: "):
                         continue
                     try:
                         d = json.loads(l[6:])
                     except Exception:
                         continue
+                    saw_any_event = True
                     txt = str(d.get("text", "") or "")
                     if txt:
                         last_text = txt
                         if not got_first:
                             first_token_ms = (time.perf_counter() - t0) * 1000.0
                             got_first = True
                     if d.get("done"):
+                        saw_done = True
                         break
                 hypothesis = last_text
+                if not saw_any_event or not saw_done:
+                    raise RuntimeError("sse stream incomplete: no terminal done event")
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/nemo_hindi/production/test_stress_full.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Compiling script**


python3 -m py_compile production/test_stress_full.py


Output:
Chunk ID: 71d9ee
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Rerunning tests with new prefix**


python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode sse --levels 100 --requests-per-client 2 --audio-source fleurs --sample-pool 240 --min-duration 3 --max-duration 20 --shuffle-samples --seed 202 --metrics-interval 2 --cooldown 0 --output-prefix sustained_v10_sse_c100_r2_recheck2


Output:
Chunk ID: 95eec8
Wall time: 30.0013 seconds
Process running with session ID 3312
Original token count: 0
Output:


Output:
Chunk ID: f37382
Wall time: 30.0018 seconds
Process running with session ID 3312
Original token count: 0
Output:


ps -eo pid,etimes,pcpu,pmem,cmd | rg "sustained_v10_sse_c100_r2_recheck2"


Output:
Chunk ID: f227cf
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 127
Output:
  63528      72 10.5  0.7 python3 production/test_stress_full.py --endpoint hindi-asr-prod --mode sse --levels 100 --requests-per-client 2 --audio-source fleurs --sample-pool 240 --min-duration 3 --max-duration 20 --shuffle-samples --seed 202 --metrics-interval 2 --cooldown 0 --output-prefix sustained_v10_sse_c100_r2_recheck2
  64031       0  0.0  0.0 /bin/bash -lc ps -eo pid,etimes,pcpu,pmem,cmd | rg "sustained_v10_sse_c100_r2_recheck2"
  64036       0  0.0  0.0 rg sustained_v10_sse_c100_r2_recheck2


Output:
Chunk ID: 8695db
Wall time: 24.5428 seconds
Process exited with code 0
Original token count: 257
Output:
=== Full Stress Test ===
Target: endpoint hindi-asr-prod
Modes: ['sse']
Levels: [100]
Audio source: fleurs (pool=240)
Requests/client: 2

==========================================================================================
Level mode=sse c=100 reqs/client=2
==========================================================================================
OK=200/200 err=0 qfull=0 tput=22.99x total_p95=40930ms ft_p95=28602ms wer=25.23%

==========================================================================================
SUMMARY
==========================================================================================
 sse c=100 ok= 200/200  err=  0 qfull=  0 p95_total= 40929.6ms p95_ft= 28601.5ms tput= 22.99x wer= 25.23%

Summary CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck2.csv
Details CSV : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck2_details.csv
Full JSON   : /home/ubuntu/nemo_hindi/benchmark_results/sustained_v10_sse_c100_r2_recheck2.json


**Inspecting worker load distribution**


python3 - <<'PY'
import json
f='benchmark_results/sustained_v10_sse_c100_r2_recheck2.json'
d=json.load(open(f))[0]
print(d['summary'])
# worker snapshot count and unique ids
ids=set()
statuses=[]
for s in d['worker_samples']:
    for w in s.get('workers',[]):
        ids.add(w.get('id'))
        statuses.append((w.get('id'),w.get('status')))
print('worker_ids',sorted(ids))
from collections import Counter
print('status counts',Counter([x[1] for x in statuses]))
PY


Output:
Chunk ID: 6ff395
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 191
Output:
{'mode': 'sse', 'concurrency': 100, 'requests_per_client': 2, 'wall_time_s': 94.732, 'ok': 200, 'errors': 0, 'queue_full_errors': 0, 'throughput_x': 22.987, 'route_ms_p50': 274.71, 'route_ms_p95': 456.49, 'route_ms_p99': 466.44, 'first_token_ms_p50': 9045.76, 'first_token_ms_p95': 28601.5, 'first_token_ms_p99': 48359.66, 'total_ms_p50': 23280.15, 'total_ms_p95': 40929.61, 'total_ms_p99': 48709.74, 'server_proc_ms_p50': 0.0, 'server_proc_ms_p95': 0.0, 'server_proc_ms_p99': 0.0, 'vram_used_gb_max': 10.43, 'worker_cur_load_max': 1110.0, 'worker_cur_load_p95': 1040.0, 'worker_cur_perf_avg': 5.4746, 'wer_pct': 25.226, 'cer_pct': 11.783, 'status_counts': {'200': 200}}
worker_ids [31138743, 31138893, 31138896]
status counts Counter({'loading': 84, 'idle': 42})


**Planning autoscaling log analysis**


python3 production/deploy.py logs > /tmp/hindi_endpoint_logs.txt && wc -l /tmp/hindi_endpoint_logs.txt && rg -n "error|ERROR|RuntimeError|Traceback|queue|All worker queues full|model_loading|destroy|restart|timeout" /tmp/hindi_endpoint_logs.txt | tail -n 80


Output:
Chunk ID: a29c97
Wall time: 0.6597 seconds
Process exited with code 0
Original token count: 805
Output:
70 /tmp/hindi_endpoint_logs.txt
21:1770675551.191368 [reconcile_workers] Destroying worker 31138581: error -> DESTROYED
22:1770675551.191368 [be_common.cpp:728 _destroy_instances] 1:  31138581,51154, error,0.123450,63.526763
26:1770675623.191 [asm_ratio_manager.cpp:1096 manage_workers] destroying 1/1 225.355541 from: 31138759,25.021671
28:1770675623.191368 [be_common.cpp:728 _destroy_instances] 1:  31138759,48284, stopping,25.021671,28.252540
30:1770675713.191 [asm_ratio_manager.cpp:1096 manage_workers] destroying 1/1 188.003883 from: 31138735,31.109352
32:1770675713.191368 [be_common.cpp:728 _destroy_instances] 1:  31138735,51732, stopping,31.109352,30.643710
34:1770675797.191 [asm_ratio_manager.cpp:1096 manage_workers] destroying 1/1 111.059457 from: 31138721,24.779945
36:1770675797.191368 [be_common.cpp:728 _destroy_instances] 1:  31138721,48300, stopping,24.779945,28.111112
37:1770676075.191 [asm_ratio_manager.cpp:1142 manage_workers] errored_workers (0/4)  1: 31138733
38:1770676075.191368 [reconcile_workers] Destroying worker 31138733: error -> DESTROYED
42:1770676104.290 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3618 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 80
43:1770676106.359 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3619 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 18
44:1770676108.428 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3620 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 16
45:1770676110.498 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3621 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
46:1770676112.567 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3622 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
47:1770676114.641 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3623 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
48:1770676116.716 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3624 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
49:1770676118.786 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3625 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
50:1770676120.855 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3626 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
51:1770676122.926 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3627 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
52:1770676124.997 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3628 new_load: 10.000000 replay_timeout: 30.000000 |tasks|: 15
53:1770676127.068 autoscaler.cpp[560] error queue_work: endpoint 11967 failed to find rdy worker for req: 3629 new_load: 10.000000 replay_timeout: 30.000000 |tasks|:
60:1770676507.358 server-endpoints.cpp[238] error: delete_requests request_idx: 4121 not found


**Planning autoscaling parameter updates**


Output:
Chunk ID: f85ea0
Wall time: 0.0825 seconds
Process exited with code 0
Original token count: 357
Output:
usage: vastai update endpoint ID [OPTIONS]

positional arguments:
 id                             id of endpoint group to update

options:
 -h, --help                     show this help message and exit
 --min_load MIN_LOAD            minimum floor load in perf units/s  (token/s for LLms)
 --target_util TARGET_UTIL      target capacity utilization (fraction, max 1.0, default 0.9)
 --cold_mult COLD_MULT          cold/stopped instance capacity target as multiple of hot capacity target (default 2.5)
 --cold_workers COLD_WORKERS    min number of workers to keep 'cold' when you have no load (default 5)
 --max_workers MAX_WORKERS      max number of workers your endpoint group can have (default 20)
 --endpoint_name ENDPOINT_NAME  deployment endpoint name (allows multiple workergroups to share same deployment endpoint)
 --url URL                      server REST api url
 --retry RETRY                  retry limit
 --raw                          output machine-readable json
 --explain                      output verbose explanation of mapping of CLI calls to HTTPS API endpoints
 --curl                         show a curl equivalency to the call
 --api-key API_KEY              api key. defaults to using the one stored in /home/ubuntu/.config/vastai/vast_api_key
 --version                      show version

Example: vastai update endpoint 4242 --min_load 100 --target_util 0.9 --cold_mult 2.0 --endpoint_name "LLama"


**Defining autoscaling parameter updates**


I’ve finished enough evidence to set concrete autoscaling defaults now. I’m applying conservative production values to reduce the worker-churn/no-ready-worker failures we observed at high SSE load, then I’ll verify the endpoint/workergroup settings.


VAST_KEY=$(awk -F= '/^VAST_KEY=/{print $2}' .env)
vastai update endpoint 11967 --min_load 40 --target_util 0.65 --cold_mult 2.5 --cold_workers 2 --max_workers 16 --api-key "$VAST_KEY"


VAST_KEY=$(awk -F= '/^VAST_KEY=/{print $2}' .env)
vastai update workergroup 16780 --endpoint_id 11967 --min_load 40 --target_util 0.65 --cold_mult 2.5 --cold_workers 2 --template_hash 503f03778b9c2bc36e5ae520d1625380 --gpu_ram 40 --api-key "$VAST_KEY"


python3 - <<'PY'
import requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip()
headers={'Authorization':f'Bearer {k}','Accept':'application/json'}
eps=requests.get('https://console.vast.ai/api/v0/endptjobs/',headers=headers,timeout=30).json().get('results',[])
for ep in eps:
    if ep.get('id')==11967:
        print('endpoint', {k: ep.get(k) for k in ['id','endpoint_name','min_load','target_util','cold_mult','cold_workers','max_workers']})
wgs=requests.get('https://console.vast.ai/api/v0/workergroups/',headers=headers,timeout=30).json().get('results',[])
for wg in wgs:
    if wg.get('id')==16780:
        print('workergroup', {k: wg.get(k) for k in ['id','endpoint_id','min_load','target_util','cold_mult','cold_workers','template_hash','gpu_ram']})
PY


Output:
Chunk ID: 8e8643
Wall time: 0.3752 seconds
Process exited with code 0
Original token count: 9
Output:
update endpoint {'success': True}


Output:
Chunk ID: 7cb45c
Wall time: 0.3663 seconds
Process exited with code 0
Original token count: 10
Output:
workergroup update {'success': True}


Output:
Chunk ID: 5134f2
Wall time: 0.5506 seconds
Process exited with code 0
Original token count: 86
Output:
endpoint {'id': 11967, 'endpoint_name': 'hindi-asr-prod', 'min_load': 1.0, 'target_util': 0.8, 'cold_mult': 2.0, 'cold_workers': 1, 'max_workers': 5}
workergroup {'id': 16780, 'endpoint_id': 11967, 'min_load': 40.0, 'target_util': 0.65, 'cold_mult': 2.5, 'cold_workers': 1, 'template_hash': '503f03778b9c2bc36e5ae520d1625380', 'gpu_ram': 40.0}


**Analyzing endpoint update inconsistencies**


VAST_KEY=$(awk -F= '/^VAST_KEY=/{print $2}' .env)
vastai --raw update workergroup 16780 --endpoint_id 11967 --cold_workers 2 --api-key "$VAST_KEY"


Output:
Chunk ID: 71a415
Wall time: 0.3031 seconds
Process exited with code 0
Original token count: 10
Output:
workergroup update {'success': True}


Summary: auto


python3 - <<'PY'
import requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip()
headers={'Authorization':f'Bearer {k}','Accept':'application/json'}
wg=[x for x in requests.get('https://console.vast.ai/api/v0/workergroups/',headers=headers,timeout=30).json().get('results',[]) if x.get('id')==16780][0]
print({k:wg.get(k) for k in ['id','min_load','target_util','cold_mult','cold_workers','template_hash']})
PY


Output:
Chunk ID: e29a74
Wall time: 0.2307 seconds
Process exited with code 0
Original token count: 36
Output:
{'id': 16780, 'min_load': 40.0, 'target_util': 0.65, 'cold_mult': 2.5, 'cold_workers': 1, 'template_hash': '503f03778b9c2bc36e5ae520d1625380'}


**Testing direct API update**


python3 - <<'PY'
import requests, json
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip()
url='https://console.vast.ai/api/v0/endptjobs/11967/'
payload={'min_load':40,'target_util':0.65,'cold_mult':2.5,'cold_workers':2,'max_workers':16}
r=requests.put(url,headers={'Authorization':f'Bearer {k}','Accept':'application/json','Content-Type':'application/json'},json=payload,timeout=30)
print('status',r.status_code)
print(r.text[:500])
# fetch again
r2=requests.get('https://console.vast.ai/api/v0/endptjobs/',headers={'Authorization':f'Bearer {k}','Accept':'application/json'},timeout=30)
for ep in r2.json().get('results',[]):
    if ep.get('id')==11967:
        print('updated', {k:ep.get(k) for k in ['id','min_load','target_util','cold_mult','cold_workers','max_workers']})
PY


Output:
Chunk ID: 06853f
Wall time: 0.4340 seconds
Process exited with code 0
Original token count: 37
Output:
status 200
{"success": true}
updated {'id': 11967, 'min_load': 40.0, 'target_util': 0.65, 'cold_mult': 2.5, 'cold_workers': 2, 'max_workers': 16}


**Updating deployment defaults**


I’m updating `production/deploy.py` defaults to match the final validated state (`v10` image and the new autoscaling baselines), so future recreate/update runs won’t silently revert to old parameters.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/nemo_hindi/production/deploy.py
@@
 DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v6-production")
 TEMPLATE_HASH = os.environ.get("VAST_TEMPLATE_HASH", "")
-LAUNCH_ARGS = os.environ.get("VAST_LAUNCH_ARGS", "-p 8000:8000 -p 18000:18000")
+LAUNCH_ARGS = os.environ.get("VAST_LAUNCH_ARGS", "-p 8000:8000")
@@
 ENDPOINT_CONFIG = {
     "endpoint_name": ENDPOINT_NAME,
-    "min_load": 1,
-    "target_util": 0.8,
-    "cold_mult": 2.0,
-    "cold_workers": 1,
-    "max_workers": 5,
+    "min_load": 40,
+    "target_util": 0.65,
+    "cold_mult": 2.5,
+    "cold_workers": 2,
+    "max_workers": 16,
 }
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/nemo_hindi/production/deploy.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Updating default Docker image version**


Input:
*** Begin Patch
*** Update File: /home/ubuntu/nemo_hindi/production/deploy.py
@@
-DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v6-production")
+DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v10-production")
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/nemo_hindi/production/deploy.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Creating deploy script without errors**


python3 -m py_compile production/deploy.py production/test_stress_full.py production/server.py


Output:
Chunk ID: 394150
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Planning smoke test with endpoint key**


python3 production/test_smoke.py --endpoint hindi-asr-prod --route-cost 10 --http-only


Output:
Chunk ID: b711d4
Wall time: 2.7951 seconds
Process exited with code 0
Original token count: 70
Output:

=== Health ===
  [PASS] workers=3

=== HTTP Batch ===
  [PASS] 937ms text='<s>' proc=0.077s

=== HTTP Stream (SSE) ===
  [PASS] 2 events 964ms
  Final: '<s>'

=== SUMMARY ===
  health               [PASS]
  http_batch           [PASS]
  http_stream          [PASS]

All passed!


**Checking worker endpoint status**


**Checking deployment status**


**Checking worker endpoint status**

**Checking deployment status**


Output:
Chunk ID: 746a63
Wall time: 1.2458 seconds
Process exited with code 0
Original token count: 50
Output:
Endpoint: hindi-asr-prod (id=11967)
  Workers: 3
    id=31138896 status=loading load=0.0
    id=31138743 status=loading load=0.0
    id=31138893 status=idle load=0.0
  Route: http://79.112.1.66:35189


**Planning final worker config retrieval**


python3 - <<'PY'
import requests
from pathlib import Path
k=''
for line in Path('.env').read_text().splitlines():
    if line.startswith('VAST_KEY='): k=line.split('=',1)[1].strip()
headers={'Authorization':f'Bearer {k}','Accept':'application/json'}
ep=[x for x in requests.get('https://console.vast.ai/api/v0/endptjobs/',headers=headers,timeout=30).json().get('results',[]) if x.get('id')==11967][0]
wg=[x for x in requests.get('https://console.vast.ai/api/v0/workergroups/',headers=headers,timeout=30).json().get('results',[]) if x.get('id')==16780][0]
print('endpoint', {k:ep.get(k) for k in ['id','endpoint_name','min_load','target_util','cold_mult','cold_workers','max_workers']})
print('workergroup', {k:wg.get(k) for k in ['id','endpoint_id','min_load','target_util','cold_mult','cold_workers','gpu_ram','template_hash']})
PY


Output:
Chunk ID: 524381
Wall time: 0.4345 seconds
Process exited with code 0
Original token count: 87
Output:
endpoint {'id': 11967, 'endpoint_name': 'hindi-asr-prod', 'min_load': 40.0, 'target_util': 0.65, 'cold_mult': 2.5, 'cold_workers': 2, 'max_workers': 16}
workergroup {'id': 16780, 'endpoint_id': 11967, 'min_load': 40.0, 'target_util': 0.65, 'cold_mult': 2.5, 'cold_workers': 1, 'gpu_ram': 40.0, 'template_hash': '503f03778b9c2bc36e5ae520d1625380'}


**Confirming endpoint and template versions**


VAST_KEY=$(awk -F= '/^VAST_KEY=/{print $2}' .env)
vastai --raw search templates --api-key "$VAST_KEY" "id=341822"


Output:
Chunk ID: 71a0a8
Wall time: 0.3669 seconds
Process exited with code 0
Original token count: 372
Output:
[
 {
  "args_str": null,
  "autoscaler": false,
  "cached": false,
  "command": null,
  "count_created": 0,
  "created_at": 1770627204.5825045,
  "created_from": null,
  "created_from_id": null,
  "creator_id": 389181,
  "default_tag": null,
  "deleted_at": null,
  "desc": null,
  "desc_count": 0,
  "docker_login_pass": null,
  "docker_login_repo": null,
  "docker_login_user": null,
  "env": "-p 8000:8000",
  "extra_filters": "{\"verified\": {\"eq\": true}, \"external\": {\"eq\": false}, \"rentable\": {\"eq\": true}, \"rented\": {\"eq\": false}, \"gpu_ram\": {\"gte\": 40000.0}, \"num_gpus\": {\"eq\": \"1\"}, \"reliability\": {\"gt\": \"0.95\"}, \"direct_port_count\": {\"gte\": \"1\"}, \"disk_space\": {\"gte\": \"30\"}, \"inet_down\": {\"gt\": \"200\"}}",
  "hash_id": "503f03778b9c2bc36e5ae520d1625380",
  "href": null,
  "id": 341822,
  "image": "bharathkumar192/hindi-nemotron-asr:v10-production",
  "jup_direct": false,
  "jupyter_dir": null,
  "jupyter_tested": null,
  "jupyterlab_tested": null,
  "lang_utf8": null,
  "max_cuda": null,
  "min_cuda": null,
  "name": null,
  "onstart": "/app/onstart.sh",
  "private": true,
  "python_utf8": null,
  "readme_hash": null,
  "readme_visible": true,
  "recent_create_date": null,
  "recommended": false,
  "recommended_disk_space": null,
  "repo": null,
  "runtype": "args",
  "sort_order": null,
  "ssh_direct": false,
  "tag": null,
  "use_jupyter_lab": false,
  "use_ssh": false,
  "vm": false,
  "volume_info": null
 }
]


Summary: auto


**Analyzing concurrency thresholds**


python3 - <<'PY'
import json
for f in ['benchmark_results/sustained_v10_http_c50_r6.json','benchmark_results/sustained_v10_sse_c50_r6.json','benchmark_results/sustained_v10_sse_c75_r3.json','benchmark_results/sustained_v10_http_c100_r2_recheck.json']:
    d=json.load(open(f))[0]
    ids=set(); sts={}
    for s in d['worker_samples']:
        for w in s.get('workers',[]):
            ids.add(w.get('id'))
    print(f.split('/')[-1], 'workers_seen',len(ids), sorted(ids)[:10], 'tput',d['summary']['throughput_x'])
PY


Output:
Chunk ID: aaadac
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 89
Output:
sustained_v10_http_c50_r6.json workers_seen 3 [31138721, 31138733, 31138743] tput 51.757
sustained_v10_sse_c50_r6.json workers_seen 2 [31138733, 31138743] tput 39.712
sustained_v10_sse_c75_r3.json workers_seen 2 [31138733, 31138743] tput 39.455
sustained_v10_http_c100_r2_recheck.json workers_seen 4 [31138743, 31138893, 31138896, 31138903] tput 37.731


**Assessing worker count impact**


python3 - <<'PY'
import json
f='benchmark_results/full_v10_fleurs_c1_100_r2.json'
d=json.load(open(f))
for item in d:
    s=item['summary']
    ids=set()
    for snap in item['worker_samples']:
        for w in snap.get('workers',[]):
            ids.add(w.get('id'))
    print(s['mode'],s['concurrency'],'workers_seen',len(ids),sorted(ids))
PY


Output:
Chunk ID: 3b7f4f
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 135
Output:
http 1 workers_seen 1 [31138581]
http 5 workers_seen 1 [31138581]
http 10 workers_seen 1 [31138581]
http 25 workers_seen 1 [31138581]
http 50 workers_seen 1 [31138581]
http 75 workers_seen 2 [31138581, 31138614]
http 100 workers_seen 2 [31138581, 31138614]
sse 1 workers_seen 1 [31138581]
sse 5 workers_seen 1 [31138581]
sse 10 workers_seen 1 [31138581]
sse 25 workers_seen 1 [31138581]
sse 50 workers_seen 1 [31138581]
sse 75 workers_seen 1 [31138581]
sse 100 workers_seen 7 [0, 31138581, 31138721, 31138733, 31138735, 31138743, 31138759]


**Deriving per-worker concurrency scaling**


Output:
Chunk ID: b24672
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 396
Output:
benchmark_results/full_v10_fleurs_c1_100_r2.csv
benchmark_results/full_v10_fleurs_c1_100_r2.json
benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
benchmark_results/sanity_v10_http_c1_r10.csv
benchmark_results/sanity_v10_http_c1_r10.json
benchmark_results/sanity_v10_http_c1_r10_details.csv
benchmark_results/sanity_v10_sse_c1_r10.csv
benchmark_results/sanity_v10_sse_c1_r10.json
benchmark_results/sanity_v10_sse_c1_r10_details.csv
benchmark_results/sustained_v10_http_c100_r2_recheck.csv
benchmark_results/sustained_v10_http_c100_r2_recheck.json
benchmark_results/sustained_v10_http_c100_r2_recheck_details.csv
benchmark_results/sustained_v10_http_c50_r6.csv
benchmark_results/sustained_v10_http_c50_r6.json
benchmark_results/sustained_v10_http_c50_r6_details.csv
benchmark_results/sustained_v10_sse_c100_r1.csv
benchmark_results/sustained_v10_sse_c100_r1.json
benchmark_results/sustained_v10_sse_c100_r1_details.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck.json
benchmark_results/sustained_v10_sse_c100_r2_recheck2.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck2.json
benchmark_results/sustained_v10_sse_c100_r2_recheck2_details.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck_details.csv
benchmark_results/sustained_v10_sse_c50_r6.csv
benchmark_results/sustained_v10_sse_c50_r6.json
benchmark_results/sustained_v10_sse_c50_r6_details.csv
benchmark_results/sustained_v10_sse_c75_r3.csv
benchmark_results/sustained_v10_sse_c75_r3.json
benchmark_results/sustained_v10_sse_c75_r3_details.csv


**Creating consolidated metrics table**


python3 - <<'PY'
import json

def load(path):
    return json.load(open(path))

main=load('benchmark_results/full_v10_fleurs_c1_100_r2.json')
print('MAIN SWEEP')
for item in main:
    s=item['summary']
    print(f"{s['mode']:>4} c={s['concurrency']:>3} ok={s['ok']:>3}/{s['concurrency']*s['requests_per_client']:<3} err={s['errors']:>3} tput={s['throughput_x']:>6.2f}x p95_total={s['total_ms_p95']:>8.0f}ms p95_ft={s['first_token_ms_p95']:>8.0f}ms wer={s['wer_pct']:>5.2f}%")
print('\nSUSTAINED CHECKS')
for f in [
    'benchmark_results/sustained_v10_http_c50_r6.json',
    'benchmark_results/sustained_v10_sse_c50_r6.json',
    'benchmark_results/sustained_v10_sse_c75_r3.json',
    'benchmark_results/sustained_v10_http_c100_r2_recheck.json',
    'benchmark_results/sustained_v10_sse_c100_r1.json',
    'benchmark_results/sustained_v10_sse_c100_r2_recheck2.json',
]:
    s=load(f)[0]['summary']
    print(f"{f.split('/')[-1]:42s} ok={s['ok']:>3}/{s['concurrency']*s['requests_per_client']:<3} err={s['errors']:>3} tput={s['throughput_x']:>6.2f}x p95_total={s['total_ms_p95']:>8.0f}ms p95_ft={s['first_token_ms_p95']:>8.0f}ms wer={s['wer_pct']:>5.2f}%")
PY


Output:
Chunk ID: ad2075
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 521
Output:
MAIN SWEEP
http c=  1 ok=  2/2   err=  0 tput=  6.91x p95_total=    1185ms p95_ft=    1185ms wer=21.23%
http c=  5 ok= 10/10  err=  0 tput= 28.91x p95_total=    2175ms p95_ft=    2175ms wer=21.03%
http c= 10 ok= 20/20  err=  0 tput= 33.17x p95_total=    4143ms p95_ft=    4143ms wer=21.92%
http c= 25 ok= 50/50  err=  0 tput= 36.52x p95_total=    8453ms p95_ft=    8453ms wer=26.11%
http c= 50 ok=100/100 err=  0 tput= 37.48x p95_total=   15370ms p95_ft=   15370ms wer=25.12%
http c= 75 ok=150/150 err=  0 tput= 37.46x p95_total=   23891ms p95_ft=   23891ms wer=24.82%
http c=100 ok=200/200 err=  0 tput= 37.51x p95_total=   32123ms p95_ft=   32123ms wer=24.47%
 sse c=  1 ok=  2/2   err=  0 tput=  9.46x p95_total=    1137ms p95_ft=     952ms wer=21.23%
 sse c=  5 ok= 10/10  err=  0 tput= 26.30x p95_total=    2575ms p95_ft=    1224ms wer=21.03%
 sse c= 10 ok= 20/20  err=  0 tput= 35.16x p95_total=    4314ms p95_ft=    1596ms wer=22.28%
 sse c= 25 ok= 50/50  err=  0 tput= 38.98x p95_total=   10335ms p95_ft=    3651ms wer=26.80%
 sse c= 50 ok=100/100 err=  0 tput= 40.03x p95_total=   19637ms p95_ft=    5218ms wer=25.56%
 sse c= 75 ok=150/150 err=  0 tput= 40.20x p95_total=   30527ms p95_ft=    9484ms wer=25.33%
 sse c=100 ok=159/200 err= 41 tput=  8.40x p95_total=  180876ms p95_ft=  180876ms wer=37.35%

SUSTAINED CHECKS
sustained_v10_http_c50_r6.json             ok=300/300 err=  0 tput= 51.76x p95_total=   17081ms p95_ft=   17081ms wer=23.67%
sustained_v10_sse_c50_r6.json              ok=300/300 err=  0 tput= 39.71x p95_total=   22071ms p95_ft=    3935ms wer=24.18%
sustained_v10_sse_c75_r3.json              ok=225/225 err=  0 tput= 39.45x p95_total=   32516ms p95_ft=    9122ms wer=24.86%
sustained_v10_http_c100_r2_recheck.json    ok=200/200 err=  0 tput= 37.73x p95_total=   30590ms p95_ft=   30590ms wer=24.53%
sustained_v10_sse_c100_r1.json             ok=100/100 err=  0 tput= 38.88x p95_total=   27783ms p95_ft=   15212ms wer=26.18%
sustained_v10_sse_c100_r2_recheck2.json    ok=200/200 err=  0 tput= 22.99x p95_total=   40930ms p95_ft=   28602ms wer=25.23%


**Estimating average request duration**


python3 - <<'PY'
import csv,statistics
rows=list(csv.DictReader(open('benchmark_results/full_v10_fleurs_c1_100_r2_details.csv')))
durs=[float(r['duration_s']) for r in rows]
print('avg_duration_s',sum(durs)/len(durs),'min',min(durs),'max',max(durs),'n',len(durs))
PY


Output:
Chunk ID: 3b569b
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 15
Output:
avg_duration_s 10.775413533834595 min 4.08 max 19.74 n 1064


**Analyzing concurrency scaling needs**


python3 - <<'PY'
import json,math,datetime
from pathlib import Path

full=json.load(open('benchmark_results/full_v10_fleurs_c1_100_r2.json'))

summary={
  'created_utc': datetime.datetime.utcnow().isoformat()+'Z',
  'gpu_observed':'Vast serverless workers (vram_total ~102GB class GPUs)',
  'data_sources':[
    'benchmark_results/full_v10_fleurs_c1_100_r2.json',
    'benchmark_results/sustained_v10_http_c50_r6.json',
    'benchmark_results/sustained_v10_sse_c50_r6.json',
    'benchmark_results/sustained_v10_sse_c75_r3.json',
    'benchmark_results/sustained_v10_http_c100_r2_recheck.json',
    'benchmark_results/sustained_v10_sse_c100_r1.json',
    'benchmark_results/sustained_v10_sse_c100_r2_recheck2.json',
  ],
  'main_sweep': [item['summary'] for item in full],
  'recommended_per_gpu_limits':{
    'http':{
      'target_concurrency':75,
      'hard_cap_concurrency':100,
      'notes':'c100 stays functional, but p95 total latency ~30s on 10.8s average clips.'
    },
    'sse':{
      'target_concurrency':50,
      'hard_cap_concurrency':75,
      'burst_cap_concurrency':100,
      'notes':'c100 can work for one-shot traffic but sustained/back-to-back shows heavy tail risk.'
    }
  },
  'quality_guardrails':{
    'wer_target_pct':30.0,
    'expected_wer_pct_range':'~21-27 in stable zone (FLEURS hi_in samples)',
    'action_if_exceeded':'scale out and/or reduce per-worker concurrency; inspect worker error churn'
  },
  'autoscaling_formula':{
    'workload_unit':'audio_seconds',
    'effective_perf_per_gpu_audio_seconds_per_sec':35.0,
    'workers_needed':'ceil(incoming_audio_seconds_per_sec / (effective_perf_per_gpu_audio_seconds_per_sec * target_util))',
    'example_500_concurrent_10s_clips_every_10s':{
      'incoming_audio_seconds_per_sec':500,
      'target_util':0.65,
      'workers_needed':math.ceil(500/(35*0.65))
    }
  },
  'endpoint_settings_applied':{
    'endpoint_id':11967,
    'min_load':40,
    'target_util':0.65,
    'cold_mult':2.5,
    'cold_workers':2,
    'max_workers':16,
    'workergroup_id':16780,
    'template_hash':'503f03778b9c2bc36e5ae520d1625380'
  },
  'known_failure_modes':[
    'Autoscaler worker churn at high SSE load can produce route failures when no standby workers are ready.',
    'At extreme SSE sustained load, tail latency can spike sharply even when median throughput remains high.',
    'Route allocation failures can occur before model queue full if workers are loading/error states.'
  ]
}

out=Path('benchmark_results')/'scaling_law_v10_20260209.json'
out.write_text(json.dumps(summary,ensure_ascii=False,indent=2))
print(out)
PY


Output:
Chunk ID: 4d4485
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 12
Output:
benchmark_results/scaling_law_v10_20260209.json


sed -n '1,200p' benchmark_results/scaling_law_v10_20260209.json


Output:
Chunk ID: c2a4dd
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 1433
Output:
{
  "created_utc": "2026-02-09T22:38:52.981391Z",
  "gpu_observed": "Vast serverless workers (vram_total ~102GB class GPUs)",
  "data_sources": [
    "benchmark_results/full_v10_fleurs_c1_100_r2.json",
    "benchmark_results/sustained_v10_http_c50_r6.json",
    "benchmark_results/sustained_v10_sse_c50_r6.json",
    "benchmark_results/sustained_v10_sse_c75_r3.json",
    "benchmark_results/sustained_v10_http_c100_r2_recheck.json",
    "benchmark_results/sustained_v10_sse_c100_r1.json",
    "benchmark_results/sustained_v10_sse_c100_r2_recheck2.json"
  ],
  "main_sweep": [
    {
      "mode": "http",
      "concurrency": 1,
      "requests_per_client": 2,
      "wall_time_s": 2.283,
      "ok": 2,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 6.912,
      "route_ms_p50": 149.54,
      "route_ms_p95": 149.54,
      "route_ms_p99": 149.54,
      "first_token_ms_p50": 1184.58,
      "first_token_ms_p95": 1184.58,
      "first_token_ms_p99": 1184.58,
      "total_ms_p50": 1184.58,
      "total_ms_p95": 1184.58,
      "total_ms_p99": 1184.58,
      "server_proc_ms_p50": 229.0,
      "server_proc_ms_p95": 229.0,
      "server_proc_ms_p99": 229.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 20.0,
      "worker_cur_load_p95": 20.0,
      "worker_cur_perf_avg": 17.9512,
      "wer_pct": 21.228,
      "cer_pct": 9.637,
      "status_counts": {
        "200": 2
      }
    },
    {
      "mode": "http",
      "concurrency": 5,
      "requests_per_client": 2,
      "wall_time_s": 3.272,
      "ok": 10,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 28.914,
      "route_ms_p50": 140.57,
      "route_ms_p95": 150.54,
      "route_ms_p99": 150.54,
      "first_token_ms_p50": 1484.13,
      "first_token_ms_p95": 2175.04,
      "first_token_ms_p99": 2175.04,
      "total_ms_p50": 1484.13,
      "total_ms_p95": 2175.04,
      "total_ms_p99": 2175.04,
      "server_proc_ms_p50": 252.0,
      "server_proc_ms_p95": 375.0,
      "server_proc_ms_p99": 375.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 70.0,
      "worker_cur_load_p95": 70.0,
      "worker_cur_perf_avg": 15.4236,
      "wer_pct": 21.03,
      "cer_pct": 7.815,
      "status_counts": {
        "200": 10
      }
    },
    {
      "mode": "http",
      "concurrency": 10,
      "requests_per_client": 2,
      "wall_time_s": 6.293,
      "ok": 20,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 33.17,
      "route_ms_p50": 134.83,
      "route_ms_p95": 159.03,
      "route_ms_p99": 159.03,
      "first_token_ms_p50": 2686.05,
      "first_token_ms_p95": 4142.99,
      "first_token_ms_p99": 4142.99,
      "total_ms_p50": 2686.05,
      "total_ms_p95": 4142.99,
      "total_ms_p99": 4142.99,
      "server_proc_ms_p50": 277.0,
      "server_proc_ms_p95": 438.0,
      "server_proc_ms_p99": 438.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 110.0,
      "worker_cur_load_p95": 110.0,
      "worker_cur_perf_avg": 18.8462,
      "wer_pct": 21.923,
      "cer_pct": 9.572,
      "status_counts": {
        "200": 20
      }
    },
    {
      "mode": "http",
      "concurrency": 25,
      "requests_per_client": 2,
      "wall_time_s": 15.645,
      "ok": 50,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 36.518,
      "route_ms_p50": 142.29,
      "route_ms_p95": 202.2,
      "route_ms_p99": 204.86,
      "first_token_ms_p50": 7295.39,
      "first_token_ms_p95": 8452.92,
      "first_token_ms_p99": 8656.5,
      "total_ms_p50": 7295.39,
      "total_ms_p95": 8452.92,
      "total_ms_p99": 8656.5,
      "server_proc_ms_p50": 264.0,
      "server_proc_ms_p95": 498.0,
      "server_proc_ms_p99": 516.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 270.0,
      "worker_cur_load_p95": 270.0,
      "worker_cur_perf_avg": 18.9776,
      "wer_pct": 26.106,
      "cer_pct": 12.458,
      "status_counts": {
        "200": 50
      }
    },
    {
      "mode": "http",
      "concurrency": 50,
      "requests_per_client": 2,
      "wall_time_s": 27.89,
      "ok": 100,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 37.484,
      "route_ms_p50": 183.34,
      "route_ms_p95": 263.48,
      "route_ms_p99": 270.98,
      "first_token_ms_p50": 13072.52,
      "first_token_ms_p95": 15370.32,
      "first_token_ms_p99": 15539.29,
      "total_ms_p50": 13072.52,
      "total_ms_p95": 15370.32,
      "total_ms_p99": 15539.29,
      "server_proc_ms_p50": 253.0,
      "server_proc_ms_p95": 443.0,
      "server_proc_ms_p99": 509.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 610.0,
      "worker_cur_load_p95": 610.0,
      "worker_cur_perf_avg": 26.7597,
      "wer_pct": 25.122,
      "cer_pct": 11.489,
      "status_counts": {
        "200": 100
      }
    },
    {
      "mode": "http",
      "concurrency": 75,
      "requests_per_client": 2,
      "wall_time_s": 43.392,
      "ok": 150,
      "errors": 0,
      "queue_full_errors": 0,
      "throughput_x": 37.463,
      "route_ms_p50": 334.68,
      "route_ms_p95": 479.71,
      "route_ms_p99": 493.65,
      "first_token_ms_p50": 21222.07,
      "first_token_ms_p95": 23891.23,
      "first_token_ms_p99": 24296.59,
      "total_ms_p50": 21222.07,
      "total_ms_p95": 23891.23,
      "total_ms_p99": 24296.59,
      "server_proc_ms_p50": 261.0,
      "server_proc_ms_p95": 477.0,
      "server_proc_ms_p99": 511.0,
      "vram_used_gb_max": 10.43,
      "worker_cur_load_max": 830.0,
      "worker_cur_load_p95": 820.0,
      "worker_cur_perf_avg": 12.8943,
      "wer_pct": 24.82,
      "cer_pct": 11.891,
      "status_counts": {
        "200": 150
      }
    },
    {


**Confirming no leftover processes**


ps -eo pid,cmd | rg "test_stress_full.py" | rg -v rg || true


Output:
Chunk ID: f74d3d
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 0
Output:


**Checking worker statuses**


Output:
Chunk ID: 2fcc33
Wall time: 1.3057 seconds
Process exited with code 0
Original token count: 50
Output:
Endpoint: hindi-asr-prod (id=11967)
  Workers: 3
    id=31138896 status=loading load=0.0
    id=31138743 status=loading load=0.0
    id=31138893 status=idle load=0.0
  Route: http://79.112.1.66:35189


**Reviewing code change locations**


nl -ba production/server.py | sed -n '420,620p'


nl -ba production/server.py | sed -n '680,910p'


nl -ba production/deploy.py | sed -n '1,120p'


nl -ba production/test_stress_full.py | sed -n '1,260p'


Output:
Chunk ID: 434d2e
Wall time: 0.0519 seconds
Process exited with code 0
Original token count: 2207
Output:
   420	                requests, outputs = await state.pipeline.process_streaming_batch()
   421	            except Exception as e:
   422	                if type(e).__name__ == "NotEnoughDataException":
   423	                    outputs = []
   424	                else:
   425	                    break
   426	            if not outputs:
   427	                break
   428	            for output in outputs:
   429	                if output.stream_id != stream_id:
   430	                    continue
   431	                if output.final_transcript:
   432	                    full_segments.append(output.final_transcript.strip())
   433	                    last_partial = ""
   434	                if output.partial_transcript:
   435	                    last_partial = output.partial_transcript.strip()
   436
   437	    finally:
   438	        # Cleanup stream state
   439	        if stream_id in state.pipeline._state_pool:
   440	            del state.pipeline._state_pool[stream_id]
   441	        if stream_id in state.pipeline._streaming_request_generator.streams:
   442	            state.pipeline._streaming_request_generator.streams.pop(stream_id, None)
   443
   444	    # Keep tail partial to avoid truncating long utterances with no final EOU flush.
   445	    combined = " ".join([x for x in [*full_segments, last_partial] if x]).strip()
   446	    final_only = " ".join([x for x in full_segments if x]).strip()
   447	    if len(combined) >= len(final_only):
   448	        return combined
   449	    return final_only or last_partial
   450
   451
   452	# ============================================================
   453	# HTTP Endpoints
   454	# ============================================================
   455
   456	async def _parse_audio_request(request: Request):
   457	    """
   458	    Parse audio payload from either JSON (audio_base64) or multipart form-data (file).
   459	    Returns (audio_bytes, language, response_format, chunk_duration_ms).
   460	    """
   461	    content_type = request.headers.get("content-type", "")
   462
   463	    if content_type.startswith("application/json"):
   464	        body = await request.json()
   465	        audio_b64 = (
   466	            body.get("audio_base64")
   467	            or body.get("audio_b64")
   468	            or body.get("audio")
   469	        )
   470	        if not audio_b64:
   471	            raise HTTPException(status_code=400, detail="audio_base64 required")
   472	        try:
   473	            audio_data = base64.b64decode(audio_b64)
   474	        except Exception as e:
   475	            raise HTTPException(status_code=400, detail=f"invalid audio_base64: {e}")
   476	        language = body.get("language", "hi")
   477	        response_format = body.get("response_format", "json")
   478	        chunk_duration_ms = int(body.get("chunk_duration_ms", 160) or 160)
   479	        return audio_data, language, response_format, chunk_duration_ms
   480
   481	    # Fallback: multipart form-data
   482	    form = await request.form()
   483	    file_obj = form.get("file")
   484	    if file_obj is None:
   485	        raise HTTPException(status_code=400, detail="file required")
   486
   487	    if hasattr(file_obj, "read"):
   488	        audio_data = await file_obj.read()
   489	    else:
   490	        audio_data = file_obj
   491
   492	    language = form.get("language", "hi")
   493	    response_format = form.get("response_format", "json")
   494	    chunk_duration_ms = int(form.get("chunk_duration_ms", 160) or 160)
   495	    return audio_data, language, response_format, chunk_duration_ms
   496
   497	@app.on_event("startup")
   498	async def startup():
   499	    """Load model on server startup"""
   500	    await load_model()
   501
   502
   503	@app.get("/health")
   504	async def health_check():
   505	    """
   506	    Health check for load balancers and PyWorker.
   507	    Returns 503 when overloaded so /route/ skips this worker.
   508	    """
   509	    try:
   510	        vram_used = torch.cuda.memory_allocated() / 1e9
   511	        vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
   512	        gpu_name = torch.cuda.get_device_name(0)
   513	    except Exception:
   514	        vram_used = vram_total = 0
   515	        gpu_name = "unknown"
   516
   517	    active = len(state.client_queues)
   518	    status = "healthy" if active < MAX_ACTIVE_STREAMS else "overloaded"
   519	    uptime = round(time.perf_counter() - state.start_time, 0) if state.start_time else 0
   520
   521	    return JSONResponse(
   522	        status_code=200 if status == "healthy" else 503,
   523	        content={
   524	            "status": status,
   525	            "model_loaded": state.model_loaded,
   526	            "active_streams": active,
   527	            "max_streams": MAX_ACTIVE_STREAMS,
   528	            "gpu": gpu_name,
   529	            "vram_used_gb": round(vram_used, 2),
   530	            "vram_total_gb": round(vram_total, 0),
   531	            "inference_running": state.inference_running,
   532	            "uptime_s": uptime,
   533	        },
   534	    )
   535
   536
   537	def _build_metrics_payload():
   538	    """Shared metrics payload for GET/POST /metrics."""
   539	    m = state.metrics
   540
   541	    def pct(arr, p):
   542	        if not arr:
   543	            return 0
   544	        s = sorted(arr)
   545	        idx = min(int(len(s) * p / 100), len(s) - 1)
   546	        return round(s[idx], 2)
   547
   548	    cuda_available = False
   549	    device_count = 0
   550	    gpu_name = "unknown"
   551	    vram_used = vram_total = vram_reserved = 0
   552	    try:
   553	        cuda_available = torch.cuda.is_available()
   554	        device_count = torch.cuda.device_count() if cuda_available else 0
   555	        if cuda_available and device_count > 0:
   556	            gpu_name = torch.cuda.get_device_name(0)
   557	            vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
   558	            vram_used = torch.cuda.memory_allocated() / 1e9
   559	            vram_reserved = torch.cuda.memory_reserved() / 1e9
   560	    except Exception:
   561	        pass
   562
   563	    return {
   564	        "active_streams": len(state.client_queues),
   565	        "max_streams": MAX_ACTIVE_STREAMS,
   566	        "inference": {
   567	            "total_batches": m["total_batches"],
   568	            "total_outputs": m["total_outputs"],
   569	            "empty_cycles": m["empty_cycles"],
   570	            "vad_filtered_chunks": m["vad_filtered_chunks"],
   571	            "gpu_ms_p50": pct(m["gpu_times_ms"], 50),
   572	            "gpu_ms_p99": pct(m["gpu_times_ms"], 99),
   573	            "cycle_ms_p50": pct(m["cycle_times_ms"], 50),
   574	            "cycle_ms_p99": pct(m["cycle_times_ms"], 99),
   575	            "batch_size_avg": round(sum(m["batch_sizes"]) / max(len(m["batch_sizes"]), 1), 1),
   576	        },
   577	        "http": {
   578	            "requests_total": m["http_requests_total"],
   579	            "requests_ok": m["http_requests_ok"],
   580	            "requests_err": m["http_requests_err"],
   581	        },
   582	        "ws": {
   583	            "connections_total": m["ws_connections_total"],
   584	            "connections_active": m["ws_connections_active"],
   585	            "connections_rejected": m["ws_connections_rejected"],
   586	        },
   587	        "cuda_available": cuda_available,
   588	        "cuda_device_count": device_count,
   589	        "gpu_name": gpu_name,
   590	        "vram_used_gb": round(vram_used, 2),
   591	        "vram_reserved_gb": round(vram_reserved, 2),
   592	        "vram_total_gb": round(vram_total, 0),
   593	        "uptime_s": round(time.perf_counter() - state.start_time, 0) if state.start_time else 0,
   594	    }
   595
   596
   597	@app.get("/metrics")
   598	async def metrics_get():
   599	    """
   600	    Detailed metrics endpoint for monitoring/alerting.
   601	    Returns structured JSON with rolling-window percentiles.
   602	    """
   603	    return _build_metrics_payload()
   604
   605
   606	@app.post("/metrics")
   607	async def metrics_post(_request: Request):
   608	    """POST variant for serverless PyWorker proxy."""
   609	    return _build_metrics_payload()
   610
   611
   612	@app.post("/v1/audio/transcriptions/json")
   613	async def transcribe_json(request: Request):
   614	    """
   615	    JSON-based transcription endpoint for PyWorker benchmarks.
   616	    Accepts: {"audio_base64": "...", "language": "hi"}
   617	    WHY: PyWorker sends benchmark payloads as JSON, not multipart.
   618	    """
   619	    state.metrics["http_requests_total"] += 1
   620	    if not state.model_loaded:


Output:
Chunk ID: 56b131
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 2872
Output:
   680	            "text": text,
   681	            "language": language,
   682	            "duration": round(audio_duration, 2),
   683	            "processing_time": round(t_done - t_infer, 3),
   684	            "total_time": round(t_done - t_start, 3),
   685	        }
   686
   687	        state.metrics["http_requests_ok"] += 1
   688	        logger.info(f"HTTP batch: {audio_duration:.1f}s audio -> {t_done - t_infer:.2f}s inference")
   689	        return result
   690
   691	    except HTTPException:
   692	        raise
   693	    except Exception as e:
   694	        state.metrics["http_requests_err"] += 1
   695	        logger.error(f"HTTP batch error: {e}", exc_info=True)
   696	        raise HTTPException(status_code=500, detail=str(e))
   697
   698
   699	@app.post("/v1/audio/transcriptions/stream")
   700	async def transcribe_stream(request: Request):
   701	    """
   702	    HTTP streaming transcription via Server-Sent Events (SSE).
   703	    Client uploads audio file, gets progressive transcription events.
   704
   705	    WHY SSE: Works through HTTP proxies (including Vast.ai PyWorker).
   706	    Compatible with standard HTTP clients — no WebSocket library needed.
   707	    """
   708	    state.metrics["http_requests_total"] += 1
   709
   710	    if not state.model_loaded:
   711	        state.metrics["http_requests_err"] += 1
   712	        raise HTTPException(status_code=503, detail="Model not loaded yet")
   713
   714	    try:
   715	        audio_data, language, response_format, chunk_duration_ms = await _parse_audio_request(request)
   716	        if not audio_data:
   717	            raise HTTPException(status_code=400, detail="Empty audio file")
   718
   719	        from asr_utils import preprocess_audio
   720	        pcm_bytes = preprocess_audio(audio_data, target_sample_rate=16000)
   721
   722	        async def sse_generator():
   723	            """Stream transcription results as SSE events"""
   724	            from nemo.collections.asr.inference.streaming.framing.request_options import ASRRequestOptions
   725
   726	            options = ASRRequestOptions()
   727	            options.asr_output_granularity = "word"
   728	            stream_id = state.pipeline.open_streaming_session(options=options)
   729
   730	            chunk_size = int(chunk_duration_ms / 1000.0 * 16000 * 2)  # bytes per chunk
   731	            att_ctx = state.cfg.streaming.att_context_size[1] + 1
   732	            frame_bytes = att_ctx * 1280 * 2  # samples per frame * 2 bytes
   733
   734	            audio_buffer = b""
   735	            full_segments: list[str] = []
   736	            last_partial = ""
   737
   738	            try:
   739	                for i in range(0, len(pcm_bytes), chunk_size):
   740	                    chunk = pcm_bytes[i:i + chunk_size]
   741	                    audio_buffer += chunk
   742
   743	                    # Feed to pipeline when we have enough for a frame
   744	                    if len(audio_buffer) >= frame_bytes:
   745	                        state.pipeline.append_streaming_audio(stream_id, audio_buffer)
   746	                        audio_buffer = b""
   747
   748	                        # Process batch
   749	                        requests, outputs = await state.pipeline.process_streaming_batch()
   750	                        for output in outputs:
   751	                            if output.stream_id == stream_id:
   752	                                text = output.final_transcript or output.partial_transcript or ""
   753	                                is_final = bool(output.final_transcript)
   754	                                if text.strip():
   755	                                    if is_final:
   756	                                        full_segments.append(text.strip())
   757	                                        last_partial = ""
   758	                                    else:
   759	                                        last_partial = text.strip()
   760	                                    event_data = json.dumps({
   761	                                        "text": text,
   762	                                        "is_final": is_final,
   763	                                        "timestamp": time.time(),
   764	                                    })
   765	                                    yield f"data: {event_data}\n\n"
   766
   767	                    # Simulate real-time pacing (optional, for more realistic SSE)
   768	                    await asyncio.sleep(0)
   769
   770	                # Flush remaining audio
   771	                if audio_buffer:
   772	                    state.pipeline.append_streaming_audio(stream_id, audio_buffer)
   773
   774	                # Close stream and drain remaining outputs
   775	                state.pipeline.close_streaming_session(stream_id)
   776	                for _ in range(10):
   777	                    try:
   778	                        requests, outputs = await state.pipeline.process_streaming_batch()
   779	                        if not outputs:
   780	                            break
   781	                        for output in outputs:
   782	                            if output.stream_id == stream_id:
   783	                                text = output.final_transcript or output.partial_transcript or ""
   784	                                is_final = bool(output.final_transcript)
   785	                                if text.strip():
   786	                                    if is_final:
   787	                                        full_segments.append(text.strip())
   788	                                        last_partial = ""
   789	                                    else:
   790	                                        last_partial = text.strip()
   791	                                    event_data = json.dumps({
   792	                                        "text": text,
   793	                                        "is_final": is_final,
   794	                                        "timestamp": time.time(),
   795	                                    })
   796	                                    yield f"data: {event_data}\n\n"
   797	                    except Exception:
   798	                        break
   799
   800	                # Final event with complete transcription
   801	                combined = " ".join([x for x in [*full_segments, last_partial] if x]).strip()
   802	                yield f"data: {json.dumps({'text': combined, 'is_final': True, 'done': True})}\n\n"
   803
   804	            finally:
   805	                # Cleanup
   806	                if stream_id in state.pipeline._state_pool:
   807	                    del state.pipeline._state_pool[stream_id]
   808	                if stream_id in state.pipeline._streaming_request_generator.streams:
   809	                    state.pipeline._streaming_request_generator.streams.pop(stream_id, None)
   810
   811	        state.metrics["http_requests_ok"] += 1
   812	        return StreamingResponse(
   813	            sse_generator(),
   814	            media_type="text/event-stream",
   815	            headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
   816	        )
   817
   818	    except HTTPException:
   819	        raise
   820	    except Exception as e:
   821	        state.metrics["http_requests_err"] += 1
   822	        logger.error(f"HTTP stream error: {e}", exc_info=True)
   823	        raise HTTPException(status_code=500, detail=str(e))
   824
   825
   826	# ============================================================
   827	# WebSocket Streaming
   828	# ============================================================
   829
   830	@app.websocket("/ws")
   831	async def websocket_streaming(ws: WebSocket):
   832	    """
   833	    Production WebSocket endpoint for real-time streaming ASR.
   834
   835	    Protocol:
   836	      Client -> Server: binary audio chunks (raw PCM 16-bit 16kHz mono)
   837	      Client -> Server: text "END" to signal end of audio
   838	      Server -> Client: JSON {"text": "...", "is_final": bool, "timestamp": float}
   839	      Server -> Client: text "END" to confirm stream complete
   840
   841	    Backpressure: Rejects with 1013 when at capacity.
   842	    """
   843	    state.metrics["ws_connections_total"] += 1
   844
   845	    # Backpressure: fast-fail when overloaded
   846	    active = len(state.client_queues)
   847	    if active >= MAX_ACTIVE_STREAMS:
   848	        state.metrics["ws_connections_rejected"] += 1
   849	        await ws.accept()
   850	        await ws.close(code=1013, reason=f"Overloaded: {active}/{MAX_ACTIVE_STREAMS}")
   851	        return
   852
   853	    await ws.accept()
   854	    state.metrics["ws_connections_active"] += 1
   855
   856	    stream_id = None
   857	    try:
   858	        # Start inference loop if needed
   859	        start_inference_loop_if_needed()
   860
   861	        # Open streaming session
   862	        stream_id = state.pipeline.open_streaming_session()
   863	        transcription_queue = await register_client(stream_id)
   864	        elapsed = time.perf_counter() - state.start_time
   865	        logger.info(f"[+{elapsed:.1f}s] WS client {stream_id} connected ({len(state.client_queues)} total)")
   866
   867	    except Exception as e:
   868	        logger.error(f"WS connection setup error: {e}", exc_info=True)
   869	        state.metrics["ws_connections_active"] -= 1
   870	        if stream_id is not None:
   871	            await unregister_client(stream_id)
   872	        if ws.application_state is WebSocketState.CONNECTED:
   873	            await ws.close()
   874	        return
   875
   876	    # Send ready message
   877	    await ws.send_text(json.dumps({"type": "Ready", "id": stream_id}))
   878
   879	    async def recv_loop():
   880	        """Receive audio chunks from client, feed to pipeline"""
   881	        att_ctx = state.cfg.streaming.att_context_size[1] + 1
   882	        frame_threshold = att_ctx * 1280 * 2  # bytes
   883	        audio_buffer = b""
   884
   885	        try:
   886	            while True:
   887	                message = await ws.receive()
   888
   889	                # Text message = control signal
   890	                if "text" in message:
   891	                    text_msg = message["text"]
   892	                    if text_msg == "END":
   893	                        await transcription_queue.put({"type": "marker"})
   894	                        break
   895
   896	                # Binary message = audio data
   897	                if "bytes" in message:
   898	                    raw_bytes = message["bytes"]
   899	                    if isinstance(raw_bytes, str):
   900	                        raw_bytes = raw_bytes.encode("latin-1")
   901	                    audio_buffer += raw_bytes
   902
   903	                    # Feed to pipeline when buffer exceeds frame threshold
   904	                    if len(audio_buffer) >= frame_threshold:
   905	                        # VAD gate: skip silence before GPU
   906	                        if state.vad is not None and not state.vad.is_speech(audio_buffer, stream_id):
   907	                            state.metrics["vad_filtered_chunks"] += 1
   908	                            audio_buffer = b""
   909	                            continue
   910


Output:
Chunk ID: 8c77b3
Wall time: 0.0515 seconds
Process exited with code 0
Original token count: 1177
Output:
     1	#!/usr/bin/env python3
     2	"""
     3	Vast.ai Serverless Deployment for Hindi ASR.
     4	Usage: python deploy.py {create|status|workers|logs|route|destroy}
     5	"""
     6	import os, sys, json, argparse, requests
     7	sys.path.insert(0, os.path.dirname(__file__))
     8	try:
     9	    from dotenv import load_dotenv
    10	    load_dotenv(os.path.join(os.path.dirname(__file__), "..", ".env"))
    11	except ImportError:
    12	    pass
    13
    14	VAST_API_KEY = os.environ.get("VAST_KEY", "")
    15	VAST_API = "https://console.vast.ai/api/v0"
    16	VAST_RUN = "https://run.vast.ai"
    17	ENDPOINT_NAME = os.environ.get("VAST_ENDPOINT_NAME", "hindi-asr-prod")
    18	DOCKER_IMAGE = os.environ.get("VAST_DOCKER_IMAGE", "bharathkumar192/hindi-nemotron-asr:v10-production")
    19	TEMPLATE_HASH = os.environ.get("VAST_TEMPLATE_HASH", "")
    20	LAUNCH_ARGS = os.environ.get("VAST_LAUNCH_ARGS", "-p 8000:8000")
    21
    22	ENDPOINT_CONFIG = {
    23	    "endpoint_name": ENDPOINT_NAME,
    24	    "min_load": 40,
    25	    "target_util": 0.65,
    26	    "cold_mult": 2.5,
    27	    "cold_workers": 2,
    28	    "max_workers": 16,
    29	}
    30
    31	SEARCH_PARAMS = {
    32	    "verified": {"eq": True}, "rentable": {"eq": True},
    33	    "rented": {"eq": False}, "gpu_ram": {"gte": 40},
    34	    "num_gpus": {"eq": 1}, "reliability": {"gte": 0.95},
    35	    "inet_down": {"gte": 200},
    36	}
    37
    38	def headers():
    39	    return {"Accept": "application/json", "Content-Type": "application/json",
    40	            "Authorization": f"Bearer {VAST_API_KEY}"}
    41
    42	def get_endpoints():
    43	    try:
    44	        r = requests.get(f"{VAST_API}/endptjobs/", headers=headers())
    45	        data = r.json()
    46	        if isinstance(data, dict):
    47	            return data.get("results", [])
    48	        if isinstance(data, list):
    49	            return data
    50	    except Exception:
    51	        pass
    52	    return []
    53
    54	def get_endpoint_info():
    55	    for ep in get_endpoints():
    56	        if ep.get("endpoint_name") == ENDPOINT_NAME:
    57	            return ep
    58	    return None
    59
    60	def create_endpoint():
    61	    print(f"Creating endpoint: {ENDPOINT_NAME}")
    62	    r = requests.post(f"{VAST_API}/endptjobs/", headers=headers(), json=ENDPOINT_CONFIG)
    63	    d = r.json()
    64	    if d.get("success"):
    65	        print(f"  Created: id={d['result']}")
    66	        return d["result"]
    67	    print(f"  Error: {d}")
    68	    return get_endpoint_id()
    69
    70	def get_endpoint_id():
    71	    ep = get_endpoint_info()
    72	    return ep.get("id") if ep else None
    73
    74	def get_endpoint_key():
    75	    ep = get_endpoint_info()
    76	    return ep.get("api_key") if ep else None
    77
    78	def get_template_hash():
    79	    if TEMPLATE_HASH:
    80	        return TEMPLATE_HASH
    81	    # Try to infer from existing workergroup
    82	    try:
    83	        r = requests.get(f"{VAST_API}/workergroups/", headers=headers())
    84	        data = r.json()
    85	        results = data.get("results", []) if isinstance(data, dict) else data
    86	        for wg in results or []:
    87	            if wg.get("endpoint_name") == ENDPOINT_NAME or wg.get("endpoint_id") == get_endpoint_id():
    88	                return wg.get("template_hash") or ""
    89	    except Exception:
    90	        pass
    91	    return ""
    92
    93	def create_workergroup(eid=None):
    94	    eid = eid or get_endpoint_id()
    95	    if not eid: print("No endpoint found"); return
    96	    print(f"Creating workergroup for endpoint {eid}")
    97	    payload = {
    98	        "endpoint_id": eid,
    99	        "gpu_ram": 40,
   100	    }
   101	    template_hash = get_template_hash()
   102	    if template_hash:
   103	        payload["template_hash"] = template_hash
   104	        payload["search_params"] = ""
   105	    else:
   106	        payload["search_params"] = json.dumps(SEARCH_PARAMS)
   107	    if LAUNCH_ARGS:
   108	        payload["launch_args"] = LAUNCH_ARGS
   109	    r = requests.post(f"{VAST_API}/workergroups/", headers=headers(), json=payload)
   110	    d = r.json()
   111	    print(f"  {'Created: id=' + str(d['result']) if d.get('success') else 'Error: ' + str(d)}")
   112
   113	def get_workers():
   114	    eid = get_endpoint_id()
   115	    ekey = get_endpoint_key()
   116	    if not eid or not ekey:
   117	        return []
   118	    r = requests.post(f"{VAST_RUN}/get_endpoint_workers/",
   119	        json={"id": eid, "api_key": ekey},
   120	        headers={"Content-Type": "application/json"})


Output:
Chunk ID: 06e7f7
Wall time: 0.0512 seconds
Process exited with code 0
Original token count: 2336
Output:
     1	#!/usr/bin/env python3
     2	"""
     3	Full-fidelity stress + quality test for Hindi ASR over Vast serverless (HTTP/SSE).
     4
     5	Features:
     6	- Customer-like concurrent clients with multiple requests per client
     7	- Route timing + queue-full detection
     8	- First-token/total latency + server processing time
     9	- Optional FLEURS Hindi audio (real speech) with WER/CER checks
    10	- Time-series snapshots from /metrics and /get_endpoint_workers/
    11	- CSV + JSON outputs suitable for scaling-law fitting
    12	"""
    13
    14	import argparse
    15	import asyncio
    16	import base64
    17	import csv
    18	import io
    19	import json
    20	import math
    21	import os
    22	import random
    23	import re
    24	import struct
    25	import sys
    26	import time
    27	import wave
    28	from collections import Counter
    29	from dataclasses import dataclass, asdict
    30	from datetime import datetime
    31	from pathlib import Path
    32	from typing import Any, Dict, List, Optional, Tuple
    33
    34	import aiohttp
    35	import numpy as np
    36	import requests
    37
    38	try:
    39	    import jiwer
    40	except Exception:
    41	    jiwer = None
    42
    43	VAST_API = "https://console.vast.ai/api/v0"
    44	VAST_RUN = "https://run.vast.ai"
    45
    46
    47	def _env_or_dotenv(name: str) -> Optional[str]:
    48	    if os.environ.get(name):
    49	        return os.environ.get(name)
    50	    env_path = Path(__file__).resolve().parent.parent / ".env"
    51	    if env_path.exists():
    52	        for line in env_path.read_text().splitlines():
    53	            if line.startswith(name + "="):
    54	                return line.split("=", 1)[1].strip()
    55	    return None
    56
    57
    58	def get_endpoint_info(endpoint_name: str, account_key: Optional[str] = None) -> Dict[str, Any]:
    59	    key = account_key or _env_or_dotenv("VAST_KEY")
    60	    if not key:
    61	        raise RuntimeError("VAST_KEY not set; pass --endpoint-key and --endpoint-id")
    62	    r = requests.get(
    63	        f"{VAST_API}/endptjobs/",
    64	        headers={"Accept": "application/json", "Authorization": f"Bearer {key}"},
    65	        timeout=30,
    66	    )
    67	    data = r.json()
    68	    results = data.get("results", []) if isinstance(data, dict) else data
    69	    for ep in results or []:
    70	        if ep.get("endpoint_name") == endpoint_name:
    71	            return ep
    72	    raise RuntimeError(f"endpoint '{endpoint_name}' not found")
    73
    74
    75	async def get_route(session: aiohttp.ClientSession, endpoint_name: str, endpoint_key: str, cost: float) -> Tuple[Optional[Dict[str, Any]], float, Optional[str], bool]:
    76	    t0 = time.perf_counter()
    77	    try:
    78	        async with session.post(
    79	            f"{VAST_RUN}/route/",
    80	            json={"endpoint": endpoint_name, "api_key": endpoint_key, "cost": cost},
    81	            timeout=aiohttp.ClientTimeout(total=15),
    82	        ) as r:
    83	            data = await r.json(content_type=None)
    84	    except Exception as e:
    85	        return None, (time.perf_counter() - t0) * 1000.0, str(e), False
    86
    87	    dt = (time.perf_counter() - t0) * 1000.0
    88	    if isinstance(data, dict) and "url" in data:
    89	        return data, dt, None, False
    90
    91	    status = ""
    92	    if isinstance(data, dict):
    93	        status = str(data.get("status", ""))
    94	    queue_full = "all worker queues full" in status.lower()
    95	    return None, dt, f"route failed: {data}", queue_full
    96
    97
    98	def wrap_auth(route: Dict[str, Any], payload: Dict[str, Any]) -> Dict[str, Any]:
    99	    auth = {k: route[k] for k in ["signature", "cost", "endpoint", "reqnum", "url", "request_idx"] if k in route}
   100	    return {"auth_data": auth, "payload": payload}
   101
   102
   103	def pct(vals: List[float], p: int) -> float:
   104	    if not vals:
   105	        return 0.0
   106	    s = sorted(vals)
   107	    idx = min(int(len(s) * p / 100), len(s) - 1)
   108	    return float(s[idx])
   109
   110
   111	def safe_mean(vals: List[float]) -> float:
   112	    return float(sum(vals) / len(vals)) if vals else 0.0
   113
   114
   115	def normalize_text(text: str) -> str:
   116	    if not text:
   117	        return ""
   118	    text = text.lower().strip()
   119	    text = re.sub(r"<[^>]+>", " ", text)
   120	    text = re.sub(r"[^\u0900-\u097fa-z0-9\s]", " ", text)
   121	    text = re.sub(r"\s+", " ", text).strip()
   122	    return text
   123
   124
   125	def compute_wer_cer(reference: str, hypothesis: str) -> Tuple[Optional[float], Optional[float]]:
   126	    if not jiwer:
   127	        return None, None
   128	    ref = normalize_text(reference)
   129	    hyp = normalize_text(hypothesis)
   130	    if not ref:
   131	        return None, None
   132	    try:
   133	        return float(jiwer.wer(ref, hyp)), float(jiwer.cer(ref, hyp))
   134	    except Exception:
   135	        return None, None
   136
   137
   138	@dataclass
   139	class AudioSample:
   140	    sample_id: str
   141	    duration_s: float
   142	    wav_bytes: bytes
   143	    reference: Optional[str] = None
   144
   145
   146	def _pcm_to_wav_bytes(pcm_int16: np.ndarray, sample_rate: int = 16000) -> bytes:
   147	    bio = io.BytesIO()
   148	    with wave.open(bio, "wb") as wf:
   149	        wf.setnchannels(1)
   150	        wf.setsampwidth(2)
   151	        wf.setframerate(sample_rate)
   152	        wf.writeframes(pcm_int16.astype(np.int16).tobytes())
   153	    return bio.getvalue()
   154
   155
   156	def make_sine_samples(durations: List[float]) -> List[AudioSample]:
   157	    samples: List[AudioSample] = []
   158	    for i, d in enumerate(durations):
   159	        n = int(16000 * d)
   160	        pcm = np.array(
   161	            [
   162	                min(32767, max(-32768, int(16000 * math.sin(2 * math.pi * 440 * k / 16000))))
   163	                for k in range(n)
   164	            ],
   165	            dtype=np.int16,
   166	        )
   167	        wav = _pcm_to_wav_bytes(pcm)
   168	        samples.append(AudioSample(sample_id=f"sine_{i}_{d:.2f}s", duration_s=float(d), wav_bytes=wav, reference=None))
   169	    return samples
   170
   171
   172	def load_fleurs_samples(max_samples: int, min_duration: float, max_duration: float, seed: int) -> List[AudioSample]:
   173	    from datasets import load_dataset
   174
   175	    ds = load_dataset("google/fleurs", "hi_in", split="test")
   176	    rows = list(ds)
   177	    rnd = random.Random(seed)
   178	    rnd.shuffle(rows)
   179
   180	    out: List[AudioSample] = []
   181	    for row in rows:
   182	        audio = row["audio"]
   183	        arr = np.asarray(audio["array"])  # float32 [-1, 1]
   184	        sr = int(audio["sampling_rate"])
   185	        if sr != 16000:
   186	            # FLEURS hi_in is normally 16kHz, but keep explicit guard.
   187	            continue
   188	        duration = len(arr) / float(sr)
   189	        if duration < min_duration or duration > max_duration:
   190	            continue
   191	        pcm = np.clip(arr, -1.0, 1.0)
   192	        pcm = (pcm * 32767.0).astype(np.int16)
   193	        wav = _pcm_to_wav_bytes(pcm, sample_rate=sr)
   194	        out.append(
   195	            AudioSample(
   196	                sample_id=str(row["id"]),
   197	                duration_s=round(duration, 3),
   198	                wav_bytes=wav,
   199	                reference=row.get("transcription"),
   200	            )
   201	        )
   202	        if len(out) >= max_samples:
   203	            break
   204
   205	    if not out:
   206	        raise RuntimeError("No FLEURS samples matched filters")
   207	    return out
   208
   209
   210	@dataclass
   211	class RequestRecord:
   212	    client_idx: int
   213	    req_idx: int
   214	    mode: str
   215	    sample_id: str
   216	    duration_s: float
   217	    route_ms: float
   218	    connect_ms: float
   219	    first_token_ms: float
   220	    total_ms: float
   221	    server_processing_ms: float
   222	    status_code: int
   223	    ok: bool
   224	    queue_full: bool
   225	    error: str
   226	    reference: str
   227	    hypothesis: str
   228	    wer: float
   229	    cer: float
   230
   231
   232	@dataclass
   233	class LevelResult:
   234	    mode: str
   235	    concurrency: int
   236	    requests_per_client: int
   237	    wall_time_s: float
   238	    ok: int
   239	    errors: int
   240	    queue_full_errors: int
   241	    throughput_x: float
   242	    route_ms_p50: float
   243	    route_ms_p95: float
   244	    route_ms_p99: float
   245	    first_token_ms_p50: float
   246	    first_token_ms_p95: float
   247	    first_token_ms_p99: float
   248	    total_ms_p50: float
   249	    total_ms_p95: float
   250	    total_ms_p99: float
   251	    server_proc_ms_p50: float
   252	    server_proc_ms_p95: float
   253	    server_proc_ms_p99: float
   254	    vram_used_gb_max: float
   255	    worker_cur_load_max: float
   256	    worker_cur_load_p95: float
   257	    worker_cur_perf_avg: float
   258	    wer_pct: float
   259	    cer_pct: float
   260	    status_counts: Dict[str, int]


Summary: auto


**Summarizing deployment results**


Output:
Chunk ID: 28e66b
Wall time: 0.0513 seconds
Process exited with code 0
Original token count: 405
Output:
total 10600
-rw-rw-r-- 1 ubuntu ubuntu   14293 Feb  9 22:38 scaling_law_v10_20260209.json
-rw-rw-r-- 1 ubuntu ubuntu  334035 Feb  9 22:35 sustained_v10_sse_c100_r2_recheck2.json
-rw-rw-r-- 1 ubuntu ubuntu  152200 Feb  9 22:35 sustained_v10_sse_c100_r2_recheck2_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     566 Feb  9 22:35 sustained_v10_sse_c100_r2_recheck2.csv
-rw-rw-r-- 1 ubuntu ubuntu  293609 Feb  9 22:32 sustained_v10_http_c100_r2_recheck.json
-rw-rw-r-- 1 ubuntu ubuntu  153237 Feb  9 22:32 sustained_v10_http_c100_r2_recheck_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     574 Feb  9 22:32 sustained_v10_http_c100_r2_recheck.csv
-rw-rw-r-- 1 ubuntu ubuntu  493699 Feb  9 22:30 sustained_v10_sse_c100_r2_recheck.json
-rw-rw-r-- 1 ubuntu ubuntu  146777 Feb  9 22:30 sustained_v10_sse_c100_r2_recheck_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     588 Feb  9 22:30 sustained_v10_sse_c100_r2_recheck.csv
-rw-rw-r-- 1 ubuntu ubuntu  137199 Feb  9 22:26 sustained_v10_sse_c100_r1.json
-rw-rw-r-- 1 ubuntu ubuntu   76173 Feb  9 22:26 sustained_v10_sse_c100_r1_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     567 Feb  9 22:26 sustained_v10_sse_c100_r1.csv
-rw-rw-r-- 1 ubuntu ubuntu  308241 Feb  9 22:26 sustained_v10_sse_c75_r3.json
-rw-rw-r-- 1 ubuntu ubuntu  174278 Feb  9 22:26 sustained_v10_sse_c75_r3_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     562 Feb  9 22:26 sustained_v10_sse_c75_r3.csv
-rw-rw-r-- 1 ubuntu ubuntu  405438 Feb  9 22:24 sustained_v10_sse_c50_r6.json
-rw-rw-r-- 1 ubuntu ubuntu  228581 Feb  9 22:24 sustained_v10_sse_c50_r6_details.csv
-rw-rw-r-- 1 ubuntu ubuntu     561 Feb  9 22:24 sustained_v10_sse_c50_r6.csv


**Verifying deployment and logs setup**


VAST_KEY=$(awk -F= '/^VAST_KEY=/{print $2}' .env)
vastai --raw show instance 31138893 --api-key "$VAST_KEY"


Output:
Chunk ID: adce9b
Wall time: 0.3790 seconds
Process exited with code 0
Original token count: 119
Output:
ID        Machine  Status   Num  Model            Util. %  vCPUs    RAM  Storage  SSH Addr  SSH Port  $/hr    Image                                              Net up  Net down  R     Label                       age(hours)  uptime(mins)
31138893  51154    running   1x  RTX_PRO_6000_WS  0.0      32.0   190.8  10       -         -         0.8028  bharathkumar192/hindi-nemotron-asr:v10-production  2207.8  2273.7    99.7  hindi-asr-prod:11967:16780  0.20        -


**Summarizing final report structure**