Output:
Chunk ID: 0ee4b6
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 14166
Output:
Total output lines: 773

benchmark_results/stress_sweep_s2.0.json
benchmark_results/stress_c250_s2.0_delays.csv
benchmark_results/capacity_c10_s60_health.csv
benchmark_results/capacity_c110_s60_health.csv
benchmark_results/capacity_c150_s60.json
benchmark_results/capacity_c120_s60_streams.csv
benchmark_results/sustained_v10_sse_c100_r1_details.csv
benchmark_results/fleurs_hindi_comparison.json
benchmark_results/capacity_c250_s60_streams.csv
benchmark_results/capacity_c20_s45_streams.csv
benchmark_results/stress_c50_s2.0_delays.csv
benchmark_results/stress_results_20260209_202718.json
benchmark_results/stress_c250_s2.0.json
benchmark_results/fleurs_hindi_details.csv
benchmark_results/stress_c1000_s2.0.json
benchmark_results/capacity_c80_s60.json
benchmark_results/indicvoices_hindi_summary.json
benchmark_results/stress_results_20260209_202433.csv
benchmark_results/capacity_c200_s60_health.csv
benchmark_results/sustained_v10_sse_c75_r3.csv
benchmark_results/capacity_c150_s60_streams.csv
benchmark_results/stress_c50_s2.0_streams.csv
benchmark_results/capacity_c100_s60.json
benchmark_results/stress_c500_s2.0_streams.csv
benchmark_results/sustained_v10_sse_c50_r6.csv
benchmark_results/stress_results_20260209_201434.json
benchmark_results/fleurs_hindi_summary.json
benchmark_results/sanity_full_sse_c1.csv
benchmark_results/capacity_c140_s60_health.csv
benchmark_results/sustained_v10_http_c100_r2_recheck.csv
benchmark_results/capacity_c50_s45_health.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck2.csv
benchmark_results/capacity_c200_s60_streams.csv
benchmark_results/full_v10_fleurs_c1_100_r2.json
benchmark_results/gpu_ceiling_test.json
benchmark_results/sustained_v10_sse_c100_r2_recheck2_details.csv
benchmark_results/sustained_v10_sse_c50_r6_details.csv
benchmark_results/capacity_c1_s45_streams.csv
benchmark_results/sanity_full_http_c1_r10_details.csv
benchmark_results/capacity_c5_sustained20_streams.csv
benchmark_results/capacity_c80_s60_health.csv
benchmark_results/capacity_c80_s45_health.csv
benchmark_results/capacity_c30_s45.json
benchmark_results/capacity_c5_sustained45_health.csv
benchmark_results/stress_c50_s2.0.json
benchmark_results/stress_c1000_s2.0_health.csv
benchmark_results/loadtest_concurrent_10_summary.json
benchmark_results/capacity_c80_s45.json
benchmark_results/capacity_c140_s60.json
benchmark_results/stress_c50_s2.0_health.csv
benchmark_results/capacity_c1_s60.json
benchmark_results/stress_results_20260209_201434.csv
benchmark_results/overall_concurrent_p99.json
benchmark_results/scaling_law_v10_20260209.json
benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
benchmark_results/capacity_c120_s60.json
benchmark_results/sanity_full_http_c1.csv
benchmark_results/capacity_c1_sustained15_delays.csv
benchmark_results/capacity_c250_s60.json
benchmark_results/loadtest_concurrent_5_details.csv
benchmark_results/capacity_c110_s60_streams.csv
benchmark_results/capacity_c50_s45.json
benchmark_results/capacity_c20_s60.json
benchmark_results/loadtest_concurrent_25_details.csv
benchmark_results/capacity_c10_s45_health.csv
benchmark_results/sustained_v10_http_c50_r6.csv
benchmark_results/stress_c10_s2.0_streams.csv
benchmark_results/capacity_c5_sustained20.json
benchmark_results/capacity_c50_s60_health.csv
benchmark_results/capacity_c20_s45_health.csv
benchmark_results/capacity_c1_sustained15.json
benchmark_results/stress_c100_s2.0_health.csv
benchmark_results/capacity_c30_s60.json
benchmark_results/sustained_v10_sse_c100_r2_recheck.json
benchmark_results/capacity_c5_sustained60_health.csv
benchmark_results/capacity_c1_sustained15_streams.csv
benchmark_results/capacity_c5_s60_health.csv
benchmark_results/local_concurrent_proper.json
benchmark_results/sanity_v10_http_c1_r10.csv
benchmark_results/indicvoices_hindi_details.csv
benchmark_results/loadtest_concurrent_25_summary.json
benchmark_results/capacity_c100_s45.json
benchmark_results/capacity_c5_s60_streams.csv
benchmark_results/stress_c5_s2.0_delays.csv
benchmark_results/capacity_c80_s45_streams.csv
benchmark_results/capacity_c10_s45_streams.csv
benchmark_results/sustained_v10_sse_c50_r6.json
benchmark_results/stress_c1000_s2.0_streams.csv
benchmark_results/capacity_c50_s60_streams.csv
benchmark_results/sustained_v10_sse_c100_r1.json
benchmark_results/capacity_c5_sustained45_delays.csv
benchmark_results/capacity_c10_s45.json
benchmark_results/stress_c10_s2.0_delays.csv
benchmark_results/stress_c500_s2.0_health.csv
benchmark_results/capacity_c5_s45_health.csv
benchmark_results/stress_c1_s1.0.json
benchmark_results/stress_c100_s2.0.json
benchmark_results/stress_c500_s2.0_delays.csv
benchmark_results/stress_c250_s2.0_health.csv
benchmark_results/loadtest_concurrent_50_summary.json
benchmark_results/sustained_v10_http_c50_r6_details.csv
benchmark_results/stress_results_20260209_202718.csv
benchmark_results/capacity_c120_s60_health.csv
benchmark_results/sanity_v10_http_c1_r10.json
benchmark_results/capacity_c100_s60_health.csv
benchmark_results/capacity_c30_s45_streams.csv
benchmark_results/capacity_c175_s60.json
benchmark_results/local_batched_sweep.json
benchmark_results/sanity_v10_sse_c1_r10.json
benchmark_results/sanity_full_http_c1_r10.json
benchmark_results/capacity_c175_s60_health.csv
benchmark_results/stress_c1_s1.0_delays.csv
benchmark_results/sustained_v10_sse_c100_r1.csv
benchmark_results/capacity_c110_s60.json
benchmark_results/capacity_c300_s60_streams.csv
benchmark_results/stress_c1_s1.0_streams.csv
benchmark_results/capacity_c100_s45_streams.csv
benchmark_results/sanity_v10_sse_c1_r10_details.csv
benchmark_results/stress_c100_s2.0_streams.csv
benchmark_results/sustained_v10_http_c100_r2_recheck_details.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck.csv
benchmark_results/stress_c500_s2.0.json
benchmark_results/stress_results_20260209_195730.json
benchmark_results/local_direct_latency.json
benchmark_results/capacity_c200_s60.json
benchmark_results/capacity_c5_s45_streams.csv
benchmark_results/capacity_c5_sustained30_streams.csv
benchmark_results/capacity_c5_sustained60_streams.csv
benchmark_results/capacity_c1_s45_health.csv
benchmark_results/loadtest_concurrent_50_details.csv
benchmark_results/capacity_c140_s60_streams.csv
benchmark_results/capacity_c100_s60_streams.csv
benchmark_results/capacity_c5_sustained45_streams.csv
benchmark_results/capacity_c150_s60_health.csv
benchmark_results/sustained_v10_sse_c75_r3_details.csv
benchmark_results/capacity_c1_s60_streams.csv
benchmark_results/stress_sweep_h100_s2.0.json
benchmark_results/capacity_c130_s60.json
benchmark_results/sanity_full_http_c1_r10.csv
benchmark_results/stress_c1_s2.0_health.csv
benchmark_results/stress_c1_s2.0_delays.csv
benchmark_results/stress_results_20260209_195631.json
benchmark_results/capacity_c30_s45_health.csv
benchmark_results/stress_c1_s2.0_streams.csv
benchmark_results/stress_results_20260209_195631.csv
benchmark_results/capacity_c5_sustained20_health.csv
benchmark_results/capacity_c5_s45.json
benchmark_results/loadtest_concurrent_1_summary.json
benchmark_results/sustained_v10_sse_c100_r2_recheck_details.csv
benchmark_results/stress_c5_s2.0_health.csv
benchmark_results/indicvoices_hindi_comparison.json
benchmark_results/capacity_c5_sustained30_health.csv
benchmark_results/capacity_c175_s60_streams.csv
benchmark_results/stress_c10_s2.0_health.csv
benchmark_results/capacity_c80_s60_streams.csv
benchmark_results/stress_c10_s2.0.json
benchmark_results/capacity_c130_s60_streams.csv
benchmark_results/capacity_c50_s45_streams.csv
benchmark_results/capacity_c20_s45.json
benchmark_results/capacity_c100_s45_health.csv
benchmark_results/capacity_c5_sustained45.json
benchmark_results/sanity_full_sse_c1.json
benchmark_results/capacity_c30_s60_streams.csv
benchmark_results/capacity_c300_s60_health.csv
benchmark_results/sustained_v10_http_c50_r6.json
benchmark_results/capacity_c5_sustained30_delays.csv
benchmark_results/capacity_c1_s45.json
benchmark_results/capacity_c20_s60_streams.csv
benchmark_results/sustained_v10_sse_c100_r2_recheck2.json
benchmark_results/capacity_c1_s60_health.csv
benchmark_results/loadtest_concurrent_5_summary.json
benchmark_results/capacity_c5_sustained60.json
benchmark_results/capacity_sweep_a100_80gb.json
benchmark_results/capacity_c10_s60.json
benchmark_results/capacity_c5_sustained30.json
benchmark_results/capacity_c130_s60_health.csv
benchmark_results/gpu_ceiling_honest.json
benchmark_results/capacity_c250_s60_health.csv
benchmark_results/capacity_c5_sustained20_delays.csv
benchmark_results/stress_c5_s2.0.json
benchmark_results/capacity_c300_s60.json
benchmark_results/capacity_c5_s60.json
benchmark_results/sanity_full_http_c1.json
benchmark_results/stress_c1000_s2.0_delays.csv
benchmark_results/stress_c5_s2.0_streams.csv
benchmark_results/stress_results_20260209_202433.json
benchmark_results/stress_c1_s1.0_health.csv
benchmark_results/stress_c250_s2.0_streams.csv
benchmark_results/capacity_c50_s60.json
benchmark_results/capacity_c20_s60_health.csv
benchmark_results/stress_c1_s2.0.json
benchmark_results/capacity_c5_sustained60_delays.csv
benchmark_results/sanity_v10_sse_c1_r10.csv
benchmark_results/sustained_v10_http_c100_r2_recheck.json
benchmark_results/stress_c100_s2.0_delays.csv
benchmark_results/capacity_c30_s60_health.csv
benchmark_results/stress_results_20260209_195730.csv
benchmark_results/capacity_c10_s60_streams.csv
benchmark_results/loadtest_concurrent_10_details.csv
benchmark_results/loadtest_concurrent_1_details.csv
benchmark_results/sanity_full_sse_c1_details.csv
production/__pycache__/test_stress_full.cpython-310.pyc
benchmark_results/capacity_c1_sustained15_health.csv
production/__pycache__/test_stress.cpython-310.pyc
benchmark_results/overall_concurrent.json
benchmark_results/sanity_full_http_c1_details.csv
benchmark_results/sanity_v10_http_c1_r10_details.csv
benchmark_results/full_v10_fleurs_c1_100_r2.csv
benchmark_results/realistic_full_pipeline.json
benchmark_results/sustained_v10_sse_c75_r3.json
production/test_stress_full.py
production/test_stress.py
venv/bin/websockets
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/INSTALLER
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/REQUESTED
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/entry_points.txt
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/METADATA
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/WHEEL
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/top_level.txt
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/RECORD
venv/lib/python3.10/site-packages/websockets-16.0.dist-info/licenses/LICENSE
venv/lib/python3.10/site-packages/filelock/__pycache__/_windows.cpython-310.pyc
venv/lib/python3.10/site-packages/filelock/_windows.py
venv/lib/python3.10/site-packages/pydantic_settings/sources/providers/__pycache__/aws.cpython-310.pyc
venv/lib/python3.10/site-packages/pydantic_settings/sources/providers/aws.py
venv/lib/python3.10/site-packages/lhotse/recipes/__pycache__/broadcast_news.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/recipes/__pycache__/iwslt22_ta.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/recipes/iwslt22_ta.py
venv/lib/python3.10/site-packages/lhotse/recipes/broadcast_news.py
venv/lib/python3.10/site-packages/starlette/__pycache__/websockets.cpython-310.pyc
venv/lib/python3.10/site-packages/starlette/middleware/__pycache__/wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/starlette/middleware/wsgi.py
venv/lib/python3.10/site-packages/starlette/websockets.py
venv/lib/python3.10/site-packages/websockets/version.py
venv/lib/python3.10/site-packages/websockets/__pycache__/proxy.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/server.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/cli.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/frames.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/http.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/headers.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/client.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/http11.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/utils.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/exceptions.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/streams.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/auth.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/datastructures.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/imports.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/__main__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/uri.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/connection.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/version.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/typing.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/__pycache__/protocol.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/extensions/__pycache__/permessage_deflate.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/extensions/__pycache__/base.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/extensions/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/extensions/permessage_deflate.py
venv/lib/python3.10/site-packages/websockets/extensions/base.py
venv/lib/python3.10/site-packages/websockets/extensions/__init__.py
venv/lib/python3.10/site-packages/websockets/auth.py
venv/lib/python3.10/site-packages/websockets/streams.py
venv/lib/python3.10/site-packages/websockets/utils.py
venv/lib/python3.10/site-packages/websockets/py.typed
venv/lib/python3.10/site-packages/websockets/speedups.c
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/server.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/client.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/router.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/async_timeout.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/messages.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/connection.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/__pycache__/compatibility.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/asyncio/router.py
venv/lib/python3.10/site-packages/websockets/asyncio/__init__.py
venv/lib/python3.10/site-packages/websockets/asyncio/server.py
venv/lib/python3.10/site-packages/websockets/asyncio/async_timeout.py
venv/lib/python3.10/site-packages/websockets/asyncio/messages.py
venv/lib/python3.10/site-packages/websockets/asyncio/connection.py
venv/lib/python3.10/site-packages/websockets/asyncio/compatibility.py
venv/lib/python3.10/site-packages/websockets/asyncio/client.py
venv/lib/python3.10/site-packages/websockets/frames.py
venv/lib/python3.10/site-packages/websockets/speedups.cpython-310-x86_64-linux-gnu.so
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/server.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/http.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/framing.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/client.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/exceptions.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/auth.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/handshake.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/__pycache__/protocol.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/legacy/auth.py
venv/lib/python3.10/site-packages/websockets/legacy/http.py
venv/lib/python3.10/site-packages/websockets/legacy/protocol.py
venv/lib/python3.10/site-packages/websockets/legacy/__init__.py
venv/lib/python3.10/site-packages/websockets/legacy/server.py
venv/lib/python3.10/site-packages/websockets/legacy/framing.py
venv/lib/python3.10/site-packages/websockets/legacy/exceptions.py
venv/lib/python3.10/site-packages/websockets/legacy/client.py
venv/lib/python3.10/site-packages/websockets/legacy/handshake.py
venv/lib/python3.10/site-packages/lhotse/workflows/__pycache__/dnsmos.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/__pycache__/whisper.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/server.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/client.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/router.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/utils.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/messages.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/__pycache__/connection.cpython-310.pyc
venv/lib/python3.10/site-packages/websockets/sync/router.py
venv/lib/python3.10/site-packages/websockets/sync/utils.py
venv/lib/python3.10/site-packages/websockets/sync/__init__.py
venv/lib/python3.10/site-packages/websockets/sync/server.py
venv/lib/python3.10/site-packages/websockets/sync/messages.py
venv/lib/python3.10/site-packages/websockets/sync/connection.py
venv/lib/python3.10/site-packages/websockets/sync/client.py
venv/lib/python3.10/site-packages/websockets/http.py
venv/lib/python3.10/site-packages/websockets/http11.py
venv/lib/python3.10/site-packages/websockets/headers.py
venv/lib/python3.10/site-packages/websockets/protocol.py
venv/lib/python3.10/site-packages/websockets/speedups.pyi
venv/lib/python3.10/site-packages/websockets/__init__.py
venv/lib/python3.10/site-packages/websockets/server.py
venv/lib/python3.10/site-packages/websockets/__main__.py
venv/lib/python3.10/site-packages/websockets/typing.py
venv/lib/python3.10/site-packages/websockets/exceptions.py
venv/lib/python3.10/site-packages/websockets/proxy.py
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/silero_vad.py
venv/lib/python3.10/site-packages/websockets/connection.py
venv/lib/python3.10/site-packages/websockets/uri.py
venv/lib/python3.10/site-packages/websockets/datastructures.py
venv/lib/python3.10/site-packages/websockets/cli.py
venv/lib/python3.10/site-packages/websockets/client.py
venv/lib/python3.10/site-packages/websockets/imports.py
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/__pycache__/silero_vad.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/__pycache__/base.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/base.py
venv/lib/python3.10/site-packages/lhotse/workflows/activity_detection/__init__.py
venv/lib/python3.10/site-packages/lhotse/workflows/meeting_simulation/speaker_independent.py
venv/lib/python3.10/site-packages/lhotse/workflows/meeting_simulation/__pycache__/base.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/meeting_simulation/__pycache__/speaker_independent.cpython-310.pyc
venv/lib/python3.10/site-packages/lhotse/workflows/meeting_simulation/__pycache__/__init__.cpython-3…4166 tokens truncated…s/sympy/core/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__pycache__/bench_sympify.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__pycache__/bench_basic.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__pycache__/bench_expand.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__pycache__/bench_arit.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__pycache__/bench_assumptions.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_expand.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_assumptions.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_basic.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_sympify.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_numbers.py
venv/lib/python3.10/site-packages/sympy/core/benchmarks/bench_arit.py
venv/lib/python3.10/site-packages/botocore/__pycache__/awsrequest.cpython-310.pyc
venv/lib/python3.10/site-packages/psutil/_pswindows.py
venv/lib/python3.10/site-packages/psutil/__pycache__/_pswindows.cpython-310.pyc
venv/lib/python3.10/site-packages/psutil/tests/__pycache__/test_windows.cpython-310.pyc
venv/lib/python3.10/site-packages/psutil/tests/test_windows.py
venv/lib/python3.10/site-packages/scipy/signal/windows/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/scipy/signal/windows/__pycache__/_windows.cpython-310.pyc
venv/lib/python3.10/site-packages/scipy/signal/windows/__pycache__/windows.cpython-310.pyc
venv/lib/python3.10/site-packages/scipy/signal/windows/_windows.py
venv/lib/python3.10/site-packages/scipy/signal/windows/windows.py
venv/lib/python3.10/site-packages/scipy/signal/windows/__init__.py
venv/lib/python3.10/site-packages/scipy/signal/tests/__pycache__/test_windows.cpython-310.pyc
venv/lib/python3.10/site-packages/scipy/signal/tests/test_windows.py
venv/lib/python3.10/site-packages/sympy/series/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/series/benchmarks/__pycache__/bench_order.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/series/benchmarks/__pycache__/bench_limit.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/series/benchmarks/bench_limit.py
venv/lib/python3.10/site-packages/sympy/series/benchmarks/bench_order.py
venv/lib/python3.10/site-packages/sympy/series/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/matrices/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/matrices/benchmarks/__pycache__/bench_matrix.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/matrices/benchmarks/bench_matrix.py
venv/lib/python3.10/site-packages/sympy/matrices/benchmarks/__init__.py
venv/lib/python3.10/site-packages/botocore/awsrequest.py
venv/lib/python3.10/site-packages/nltk/corpus/reader/__pycache__/reviews.cpython-310.pyc
venv/lib/python3.10/site-packages/nltk/corpus/reader/reviews.py
venv/lib/python3.10/site-packages/nltk/__pycache__/wsd.cpython-310.pyc
venv/lib/python3.10/site-packages/nltk/wsd.py
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/bench_galoispolys.py
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/__pycache__/bench_solvers.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/__pycache__/bench_galoispolys.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/__pycache__/bench_groebnertools.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/bench_groebnertools.py
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/polys/benchmarks/bench_solvers.py
venv/lib/python3.10/site-packages/nltk/test/unit/__pycache__/test_corpus_views.cpython-310.pyc
venv/lib/python3.10/site-packages/nltk/test/unit/test_corpus_views.py
venv/lib/python3.10/site-packages/nltk/test/wsd.doctest
venv/lib/python3.10/site-packages/tzdata/zoneinfo/America/Dawson
venv/lib/python3.10/site-packages/torch/include/torch/csrc/utils/throughput_benchmark.h
venv/lib/python3.10/site-packages/torch/include/torch/csrc/utils/throughput_benchmark-inl.h
venv/lib/python3.10/site-packages/tzdata/zoneinfo/America/Dawson_Creek
venv/lib/python3.10/site-packages/sympy/functions/elementary/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/functions/elementary/benchmarks/__pycache__/bench_exp.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/functions/elementary/benchmarks/bench_exp.py
venv/lib/python3.10/site-packages/sympy/functions/elementary/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/functions/special/benchmarks/bench_special.py
venv/lib/python3.10/site-packages/sympy/functions/special/benchmarks/__pycache__/bench_special.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/functions/special/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/functions/special/benchmarks/__init__.py
venv/lib/python3.10/site-packages/tzdata/zoneinfo/Antarctica/Mawson
venv/lib/python3.10/site-packages/sympy/simplify/__pycache__/powsimp.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/simplify/powsimp.py
venv/lib/python3.10/site-packages/sympy/simplify/tests/__pycache__/test_powsimp.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/simplify/tests/test_powsimp.py
venv/lib/python3.10/site-packages/torch/include/torch/csrc/jit/frontend/tree_views.h
venv/lib/python3.10/site-packages/torch/include/torch/csrc/jit/python/python_tree_views.h
venv/lib/python3.10/site-packages/sympy/solvers/benchmarks/__pycache__/bench_solvers.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/solvers/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/solvers/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/solvers/benchmarks/bench_solvers.py
venv/lib/python3.10/site-packages/torch/include/torch/csrc/jit/ir/ir_views.h
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/__pycache__/bench_integrate.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/__pycache__/bench_trigintegrate.cpython-310.pyc
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/bench_trigintegrate.py
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/__init__.py
venv/lib/python3.10/site-packages/sympy/integrals/benchmarks/bench_integrate.py
venv/lib/python3.10/site-packages/sentry_sdk/integrations/django/views.py
venv/lib/python3.10/site-packages/sentry_sdk/integrations/django/__pycache__/views.cpython-310.pyc
venv/lib/python3.10/site-packages/sentry_sdk/integrations/__pycache__/wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/sentry_sdk/integrations/__pycache__/_wsgi_common.cpython-310.pyc
venv/lib/python3.10/site-packages/sentry_sdk/integrations/__pycache__/aws_lambda.cpython-310.pyc
venv/lib/python3.10/site-packages/sentry_sdk/integrations/_wsgi_common.py
venv/lib/python3.10/site-packages/sentry_sdk/integrations/wsgi.py
venv/lib/python3.10/site-packages/sentry_sdk/integrations/aws_lambda.py
venv/lib/python3.10/site-packages/wandb/sdk/launch/environment/__pycache__/aws_environment.cpython-310.pyc
venv/lib/python3.10/site-packages/wandb/sdk/launch/environment/aws_environment.py
venv/lib/python3.10/site-packages/fastapi/__pycache__/websockets.cpython-310.pyc
venv/lib/python3.10/site-packages/fastapi/middleware/__pycache__/wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/fastapi/middleware/wsgi.py
venv/lib/python3.10/site-packages/fastapi/websockets.py
venv/lib/python3.10/site-packages/torch/include/c10/util/overflows.h
venv/lib/python3.10/site-packages/jsonschema/benchmarks/import_benchmark.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/useless_keywords.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/json_schema_test_suite.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/json_schema_test_suite.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/import_benchmark.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/unused_registry.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/useless_keywords.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/const_vs_enum.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/contains.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/validator_creation.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/issue232.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/useless_applicator_schemas.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/subcomponents.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__pycache__/nested_schemas.cpython-310.pyc
venv/lib/python3.10/site-packages/jsonschema/benchmarks/issue232/issue.json
venv/lib/python3.10/site-packages/jsonschema/benchmarks/unused_registry.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/const_vs_enum.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/__init__.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/useless_applicator_schemas.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/nested_schemas.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/validator_creation.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/subcomponents.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/issue232.py
venv/lib/python3.10/site-packages/jsonschema/benchmarks/contains.py
venv/lib/python3.10/site-packages/nemo/agents/voice_agent/pipecat/transports/network/__pycache__/websocket_server.cpython-310.pyc
venv/lib/python3.10/site-packages/nemo/agents/voice_agent/pipecat/transports/network/websocket_server.py
venv/lib/python3.10/site-packages/torch/lib/libshm_windows/libshm.h
venv/lib/python3.10/site-packages/rich/__pycache__/_windows_renderer.cpython-310.pyc
venv/lib/python3.10/site-packages/rich/__pycache__/_windows.cpython-310.pyc
venv/lib/python3.10/site-packages/rich/_windows.py
venv/lib/python3.10/site-packages/rich/_windows_renderer.py
venv/lib/python3.10/site-packages/httpx/_transports/__pycache__/wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/httpx/_transports/wsgi.py
venv/lib/python3.10/site-packages/torch/_inductor/__pycache__/wrapper_benchmark.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/_inductor/runtime/__pycache__/benchmarking.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/_inductor/runtime/benchmarking.py
venv/lib/python3.10/site-packages/torch/_inductor/wrapper_benchmark.py
venv/lib/python3.10/site-packages/torch/_inductor/codegen/rocm/__pycache__/rocm_benchmark_request.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/_inductor/codegen/rocm/rocm_benchmark_request.py
venv/lib/python3.10/site-packages/torch/_inductor/kernel/templates/triton_blackwell_ws_persistent_device_tma_mm.py.jinja
venv/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/__pycache__/asr_confidence_benchmarking_utils.cpython-310.pyc
venv/lib/python3.10/site-packages/nemo/collections/asr/parts/utils/asr_confidence_benchmarking_utils.py
venv/lib/python3.10/site-packages/torch/_functorch/__pycache__/benchmark_utils.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/_functorch/benchmark_utils.py
venv/lib/python3.10/site-packages/wandb/vendor/promise-2.3.0/tests/__pycache__/test_benchmark.cpython-310.pyc
venv/lib/python3.10/site-packages/wandb/vendor/promise-2.3.0/tests/test_benchmark.py
venv/lib/python3.10/site-packages/torch/signal/windows/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/signal/windows/__pycache__/windows.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/signal/windows/windows.py
venv/lib/python3.10/site-packages/torch/signal/windows/__init__.py
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/3/asyncio/windows_events.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/3/asyncio/windows_utils.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/handlers.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/__init__.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/types.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/validate.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/simple_server.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/headers.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/wsgiref/util.pyi
venv/lib/python3.10/site-packages/Crypto/PublicKey/_ec_ws.abi3.so
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/_typeshed/wsgi.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/stdlib/2and3/webbrowser.pyi
venv/lib/python3.10/site-packages/werkzeug/__pycache__/wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/werkzeug/wsgi.py
venv/lib/python3.10/site-packages/torch/_C/_jit_tree_views.pyi
venv/lib/python3.10/site-packages/aiohttp/__pycache__/web_ws.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/__pycache__/client_ws.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/__pycache__/http_websocket.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/http_websocket.py
venv/lib/python3.10/site-packages/aiohttp/client_ws.py
venv/lib/python3.10/site-packages/aiohttp/web_ws.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/reader_c.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/reader_py.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/reader.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/writer.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/__init__.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/helpers.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/models.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/reader_py.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/__pycache__/reader_c.cpython-310.pyc
venv/lib/python3.10/site-packages/aiohttp/_websocket/mask.pyx
venv/lib/python3.10/site-packages/aiohttp/_websocket/mask.pxd
venv/lib/python3.10/site-packages/aiohttp/_websocket/writer.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/reader.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/__init__.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/models.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/reader_c.cpython-310-x86_64-linux-gnu.so
venv/lib/python3.10/site-packages/aiohttp/_websocket/helpers.py
venv/lib/python3.10/site-packages/aiohttp/_websocket/reader_c.pxd
venv/lib/python3.10/site-packages/aiohttp/_websocket/mask.cpython-310-x86_64-linux-gnu.so
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/third_party/2and3/werkzeug/wsgi.pyi
venv/lib/python3.10/site-packages/jedi/third_party/typeshed/third_party/2and3/flask/views.pyi
venv/lib/python3.10/site-packages/prompt_toolkit/output/__pycache__/windows10.cpython-310.pyc
venv/lib/python3.10/site-packages/prompt_toolkit/output/windows10.py
venv/lib/python3.10/site-packages/modal/_utils/__pycache__/browser_utils.cpython-310.pyc
venv/lib/python3.10/site-packages/modal/_utils/browser_utils.py
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/sitemaps/views.pyi
venv/lib/python3.10/site-packages/modal/_vendor/__pycache__/a2wsgi_wsgi.cpython-310.pyc
venv/lib/python3.10/site-packages/modal/_vendor/a2wsgi_wsgi.py
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/auth/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/auth/handlers/modwsgi.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/admin/views/autocomplete.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/admin/views/__init__.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/admin/views/decorators.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/admin/views/main.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/messages/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/admindocs/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/syndication/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/flatpages/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/contenttypes/views.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/contrib/staticfiles/views.pyi
venv/lib/python3.10/site-packages/torch/ao/ns/fx/__pycache__/n_shadows_utils.cpython-310.pyc
venv/lib/python3.10/site-packages/torch/ao/ns/fx/n_shadows_utils.py
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/csrf.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/__init__.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/static.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/csrf.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/cache.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/gzip.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/__init__.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/debug.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/clickjacking.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/vary.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/decorators/http.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/edit.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/__init__.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/list.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/dates.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/base.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/generic/detail.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/debug.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/i18n.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/views/defaults.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/core/wsgi.pyi
venv/lib/python3.10/site-packages/jedi/third_party/django-stubs/django-stubs/core/handlers/wsgi.pyi


Output:
Chunk ID: 80b316
Wall time: 1.0281 seconds
Process exited with code 0
Original token count: 21
Output:
Endpoint: hindi-asr-prod (id=11967)
  Workers: 0
  Route: http://72.19.32.135:43287


Output:
Chunk ID: 11177c
Wall time: 0.6483 seconds
Process exited with code 0
Original token count: 547
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.6541290283203125,
    "dlperf": 283.67722901688677,
    "id": 31138896,
    "loaded_at": 1770680410.6239877,
    "measured_perf": 60.0201953645223,
    "new_load": 0.0,
    "outdated": false,
    "perf": 60.02019500732422,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770680377.1913676,
    "status": "idle"
  },
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.6442108154296875,
    "dlperf": 377.7898826488518,
    "id": 31140799,
    "loaded_at": 0.0,
    "measured_perf": 55.416038513183594,
    "new_load": 0.0,
    "outdated": false,
    "perf": 55.416038513183594,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770680401.1913676,
    "status": "model_loading"
  },
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 1.1045961106299e-05,
    "cur_perf": 0.0,
    "disk_usage": 4.634525299072266,
    "dlperf": 283.48244628500436,
    "id": 31139135,
    "loaded_at": 0.0,
    "measured_perf": 62.81435810315692,
    "new_load": 0.0,
    "outdated": false,
    "perf": 9.196084022521973,
    "reliability": 0.6185659363701174,
    "reqs_working": 0,
    "started_at": 0.0,
    "status": "stopped"
  },
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644908905029297,
    "dlperf": 279.33293848751566,
    "id": 31140781,
    "loaded_at": 1770680460.8428197,
    "measured_perf": 27.557987324331773,
    "new_load": 0.0,
    "outdated": false,
    "perf": 27.557987213134766,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770680374.1913676,
    "status": "idle"
  },
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 7.713806611729567e-06,
    "cur_perf": 2.054178697149056,
    "disk_usage": 4.808258056640625,
    "dlperf": 283.1902643586163,
    "id": 31138893,
    "loaded_at": 0.0,
    "measured_perf": 64.446012519946,
    "new_load": 0.0,
    "outdated": false,
    "perf": 11.66126823425293,
    "reliability": 0.6522099642728176,
    "reqs_working": 0,
    "started_at": 0.0,
    "status": "stopped"
  }
]


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Checking endpoint availability**


I verified the production endpoint is alive, but it now has multiple worker lifecycle entries and is not ideal for apples-to-apples single-GPU WS testing. I’m checking the dedicated `hindi-asr-ws-bench` endpoint status and then I’ll run a controlled WS benchmark there.


VAST_ENDPOINT_NAME=hindi-asr-ws-bench python3 production/deploy.py status


VAST_ENDPOINT_NAME=hindi-asr-ws-bench python3 production/deploy.py workers | sed -n '1,260p'


python3 - << 'PY'
import requests, os
from pathlib import Path
# pull key from env/.env
key=os.getenv('VAST_KEY')
if not key:
    for ln in Path('.env').read_text().splitlines():
        if ln.startswith('VAST_KEY='):
            key=ln.split('=',1)[1].strip(); break
h={'Accept':'application/json','Authorization':f'Bearer {key}'}
resp=requests.get('https://console.vast.ai/api/v0/endptjobs/',headers=h,timeout=30).json()
rows=resp.get('results',[]) if isinstance(resp,dict) else resp
for r in rows:
    if r.get('endpoint_name') in ('hindi-asr-prod','hindi-asr-ws-bench'):
        print(r.get('endpoint_name'), 'id=',r.get('id'),'workers',r.get('workers'),'min_load',r.get('min_load'),'max_workers',r.get('max_workers'),'target_util',r.get('target_util'))
PY


Output:
Chunk ID: 0c8fb6
Wall time: 1.3385 seconds
Process exited with code 0
Original token count: 32
Output:
Endpoint: hindi-asr-ws-bench (id=11995)
  Workers: 1
    id=31140830 status=idle load=0.0
  Route: http://140.150.159.2:28077


Output:
Chunk ID: 0c8c11
Wall time: 0.6963 seconds
Process exited with code 0
Original token count: 110
Output:
[
  {
    "cur_load": 0.0,
    "cur_load_rolling_avg": 0.0,
    "cur_perf": 0.0,
    "disk_usage": 4.644920349121094,
    "dlperf": 282.3526239563256,
    "id": 31140830,
    "loaded_at": 1770680568.9050314,
    "measured_perf": 31.342262522851918,
    "new_load": 0.0,
    "outdated": false,
    "perf": 31.342262268066406,
    "reliability": 1.0,
    "reqs_working": 0,
    "started_at": 1770680499.9882426,
    "status": "idle"
  }
]


Output:
Chunk ID: 2629ae
Wall time: 0.4995 seconds
Process exited with code 0
Original token count: 43
Output:
hindi-asr-prod id= 11967 workers None min_load 40.0 max_workers 16 target_util 0.65
hindi-asr-ws-bench id= 11995 workers None min_load 1.0 max_workers 1 target_util 0.7


Summary: auto


**Planning robust WS benchmarking script**


python3 - << 'PY'
import json,glob,os
for p in ['benchmark_results/stress_sweep_h100_s2.0.json','benchmark_results/capacity_sweep_a100_80gb.json','benchmark_results/scaling_law_v10_20260209.json','benchmark_results/gpu_ceiling_honest.json','benchmark_results/gpu_ceiling_test.json']:
    if os.path.exists(p):
        print('\n==',p)
        d=json.load(open(p))
        if isinstance(d,dict):
            for k in list(d.keys())[:8]:
                if isinstance(d[k],(int,float,str)): print(k,d[k])
            if 'results' in d:
                print('results len',len(d['results']))
                for r in d['results'][:3]:
                    print(r)
        else:
            print(type(d),len(d))
PY


python3 - << 'PY'
import csv
p='benchmark_results/full_v10_fleurs_c1_100_r2.csv'
print('exists',p)
with open(p) as f:
    r=list(csv.DictReader(f))
print('rows',len(r))
for row in r[:5]:
    print({k:row[k] for k in ['mode','concurrency','ok','errors','queue_full_errors','throughput_x','total_ms_p95','first_token_ms_p95','wer_pct','vram_used_gb_max','worker_cur_load_max'] if k in row})
PY


python3 - << 'PY'
import csv
p='benchmark_results/full_v10_fleurs_c1_100_r2_details.csv'
with open(p) as f:
    r=list(csv.DictReader(f))
print('rows',len(r),'ok',sum(1 for x in r if x['ok']=='True'),'err',sum(1 for x in r if x['ok']!='True'))
# show top error messages if any
from collections import Counter
c=Counter(x['error'] for x in r if x['error'])
print('errors',c.most_common(5))
# status codes
s=Counter(x['status_code'] for x in r)
print('status',s)
PY


Output:
Chunk ID: 8db767
Wall time: 0.0504 seconds
Process exited with code 0
Original token count: 561
Output:

== benchmark_results/stress_sweep_h100_s2.0.json
<class 'list'> 7

== benchmark_results/capacity_sweep_a100_80gb.json
gpu NVIDIA A100-SXM4-80GB
sustained_s 60.0
timestamp 2026-02-09 07:01:17
results len 13
{'concurrency': 1, 'sustained_s': 60.0, 'valid_streams': 1, 'errored_streams': 0, 'error_rate': 0.0, 'clean_disconnects': 1, 'wer_pct': 37.5, 'cer_pct': 19.83, 'total_audio_sec': 9.1, 'wall_time_sec': 11.8, 'throughput_rtx': 0.78, 'total_chunks': 16, 'avg_loops': 1.0, 'chunk_p50_ms': -419.7, 'chunk_p95_ms': -379.4, 'chunk_p99_ms': -377.4, 'chunk_max_ms': -376.9, 'ft_p50_ms': 137.7, 'ft_p95_ms': 137.7, 'ft_p99_ms': 137.7, 'conn_p50_ms': 73.2, 'conn_p99_ms': 73.2, 'drift': {}, 'gpu': {'streams_peak': 1, 'streams_avg': 1.0, 'polls': 6}}
{'concurrency': 10, 'sustained_s': 60.0, 'valid_streams': 10, 'errored_streams': 0, 'error_rate': 0.0, 'clean_disconnects': 10, 'wer_pct': 22.85, 'cer_pct': 10.11, 'total_audio_sec': 113.6, 'wall_time_sec': 19.6, 'throughput_rtx': 5.79, 'total_chunks': 185, 'avg_loops': 1.0, 'chunk_p50_ms': -378.4, 'chunk_p95_ms': -336.3, 'chunk_p99_ms': -325.3, 'chunk_max_ms': -322.7, 'ft_p50_ms': 191.0, 'ft_p95_ms': 199.3, 'ft_p99_ms': 201.3, 'conn_p50_ms': 78.2, 'conn_p99_ms': 86.4, 'drift': {'early_ms': -376.8, 'late_ms': -378.2, 'drift_ms': -1.5}, 'gpu': {'streams_peak': 10, 'streams_avg': 7.0, 'polls': 10}}
{'concurrency': 30, 'sustained_s': 60.0, 'valid_streams': 30, 'errored_streams': 0, 'error_rate': 0.0, 'clean_disconnects': 30, 'wer_pct': 29.85, 'cer_pct': 16.55, 'total_audio_sec': 348.3, 'wall_time_sec': 23.6, 'throughput_rtx': 14.76, 'total_chunks': 563, 'avg_loops': 1.0, 'chunk_p50_ms': -356.7, 'chunk_p95_ms': -296.8, 'chunk_p99_ms': -216.4, 'chunk_max_ms': -197.3, 'ft_p50_ms': 207.7, 'ft_p95_ms': 249.2, 'ft_p99_ms': 262.5, 'conn_p50_ms': 81.0, 'conn_p99_ms': 93.5, 'drift': {'early_ms': -356.0, 'late_ms': -358.3, 'drift_ms': -2.2}, 'gpu': {'streams_peak': 30, 'streams_avg': 18.0, 'polls': 12}}

== benchmark_results/scaling_law_v10_20260209.json
created_utc 2026-02-09T22:38:52.981391Z
gpu_observed Vast serverless workers (vram_total ~102GB class GPUs)

== benchmark_results/gpu_ceiling_honest.json
<class 'list'> 13

== benchmark_results/gpu_ceiling_test.json
<class 'list'> 11


Output:
Chunk ID: ce74a8
Wall time: 0.0506 seconds
Process exited with code 0
Original token count: 339
Output:
exists benchmark_results/full_v10_fleurs_c1_100_r2.csv
rows 14
{'mode': 'http', 'concurrency': '1', 'ok': '2', 'errors': '0', 'queue_full_errors': '0', 'throughput_x': '6.912', 'total_ms_p95': '1184.58', 'first_token_ms_p95': '1184.58', 'wer_pct': '21.228', 'vram_used_gb_max': '10.43', 'worker_cur_load_max': '20.0'}
{'mode': 'http', 'concurrency': '5', 'ok': '10', 'errors': '0', 'queue_full_errors': '0', 'throughput_x': '28.914', 'total_ms_p95': '2175.04', 'first_token_ms_p95': '2175.04', 'wer_pct': '21.03', 'vram_used_gb_max': '10.43', 'worker_cur_load_max': '70.0'}
{'mode': 'http', 'concurrency': '10', 'ok': '20', 'errors': '0', 'queue_full_errors': '0', 'throughput_x': '33.17', 'total_ms_p95': '4142.99', 'first_token_ms_p95': '4142.99', 'wer_pct': '21.923', 'vram_used_gb_max': '10.43', 'worker_cur_load_max': '110.0'}
{'mode': 'http', 'concurrency': '25', 'ok': '50', 'errors': '0', 'queue_full_errors': '0', 'throughput_x': '36.518', 'total_ms_p95': '8452.92', 'first_token_ms_p95': '8452.92', 'wer_pct': '26.106', 'vram_used_gb_max': '10.43', 'worker_cur_load_max': '270.0'}
{'mode': 'http', 'concurrency': '50', 'ok': '100', 'errors': '0', 'queue_full_errors': '0', 'throughput_x': '37.484', 'total_ms_p95': '15370.32', 'first_token_ms_p95': '15370.32', 'wer_pct': '25.122', 'vram_used_gb_max': '10.43', 'worker_cur_load_max': '610.0'}


Output:
Chunk ID: 5d41e6
Wall time: 0.0516 seconds
Process exited with code 0
Original token count: 287
Output:
rows 1064 ok 1023 err 41
errors [("route failed: {'__request_id': '8e407ec5-940c-4f27-bb36-8ffd681b6217', 'endpoint': 'hindi-asr-prod', 'request_idx': 2241, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}", 1), ("route failed: {'__request_id': '1527c60e-bfca-4cf3-a1d7-0459c9e3e5e4', 'endpoint': 'hindi-asr-prod', 'request_idx': 2242, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}", 1), ("route failed: {'__request_id': '964922ea-bb13-4c38-a62b-1031930d73fd', 'endpoint': 'hindi-asr-prod', 'request_idx': 2243, 'status': 'total workers: 2 loading workers: 0 standby workers: 0 error workers: 1'}", 1), ("route failed: {'__request_id': 'f3df8d4d-3fa9-4a18-8a99-31ab2fb163f8', 'endpoint': 'hindi-asr-prod', 'request_idx': 2244, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}", 1), ("route failed: {'__request_id': '19604311-72d6-448b-956e-062f5aa43f53', 'endpoint': 'hindi-asr-prod', 'request_idx': 2245, 'status': 'total workers: 3 loading workers: 0 standby workers: 0 error workers: 1'}", 1)]
status Counter({'200': 1023, '0': 41})


**Planning capacity sweep review**


python3 - << 'PY'
import json
p='benchmark_results/capacity_sweep_a100_80gb.json'
d=json.load(open(p))
res=d['results']
print('levels', [r['concurrency'] for r in res])
# print key metrics each
for r in res:
    print(r['concurrency'], 'valid',r['valid_streams'],'err',r['errored_streams'],'wer',r['wer_pct'],'ft_p95',r['ft_p95_ms'],'throughput_rtx',r['throughput_rtx'],'drift',r.get('drift',{}).get('drift_ms'))
PY


Output:
Chunk ID: b5b35a
Wall time: 0.0511 seconds
Process exited with code 1
Original token count: 268
Output:
levels [1, 10, 30, 50, 80, 100, 110, 120, 130, 140, 150, 175, 200]
1 valid 1 err 0 wer 37.5 ft_p95 137.7 throughput_rtx 0.78 drift None
10 valid 10 err 0 wer 22.85 ft_p95 199.3 throughput_rtx 5.79 drift -1.5
30 valid 30 err 0 wer 29.85 ft_p95 249.2 throughput_rtx 14.76 drift -2.2
50 valid 50 err 0 wer 31.19 ft_p95 289.7 throughput_rtx 24.3 drift -41.2
80 valid 80 err 0 wer 33.54 ft_p95 2393.3 throughput_rtx 36.31 drift -111.5
100 valid 100 err 0 wer 33.24 ft_p95 3393.4 throughput_rtx 46.04 drift -116.9
110 valid 110 err 0 wer 32.39 ft_p95 401.7 throughput_rtx 53.84 drift 180.7
120 valid 120 err 0 wer 35.36 ft_p95 7463.3 throughput_rtx 54.38 drift -182.1
130 valid 130 err 0 wer 36.98 ft_p95 9645.4 throughput_rtx 58.61 drift -1947.3
140 valid 140 err 0 wer 33.32 ft_p95 478.0 throughput_rtx 68.69 drift 252.6
150 valid 150 err 0 wer 51.95 ft_p95 11388.6 throughput_rtx 65.27 drift 1863.8
175 valid 175 err 0 wer 96.93 ft_p95 1901.2 throughput_rtx 79.84 drift 6445.2
Traceback (most recent call last):
  File "<stdin>", line 8, in <module>
KeyError: 'ft_p95_ms'


**Investigating capacity sweep scripts**


rg -n "capacity_sweep|throughput_rtx|drift_ms|sustained_s|clean_disconnects|total_chunks" -S . | sed -n '1,240p'


Output:
Chunk ID: f65857
Wall time: 0.1395 seconds
Process exited with code 0
Original token count: 4428
Output:
./benchmark_results/capacity_c80_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c80_s60.json:7:  "clean_disconnects": 80,
./benchmark_results/capacity_c80_s60.json:12:  "throughput_rtx": 36.31,
./benchmark_results/capacity_c80_s60.json:13:  "total_chunks": 1512,
./benchmark_results/capacity_c80_s60.json:27:    "drift_ms": -111.5
./benchmark_results/capacity_c20_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c20_s60.json:7:  "clean_disconnects": 20,
./benchmark_results/capacity_c20_s60.json:12:  "throughput_rtx": 20.77,
./benchmark_results/capacity_c20_s60.json:13:  "total_chunks": 1862,
./benchmark_results/capacity_c20_s60.json:27:    "drift_ms": -35.2
./benchmark_results/capacity_c100_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c100_s60.json:7:  "clean_disconnects": 100,
./benchmark_results/capacity_c100_s60.json:12:  "throughput_rtx": 46.04,
./benchmark_results/capacity_c100_s60.json:13:  "total_chunks": 1889,
./benchmark_results/capacity_c100_s60.json:27:    "drift_ms": -116.9
./benchmark_results/stress_sweep_s2.0.json:7:    "clean_disconnects": 1,
./benchmark_results/stress_sweep_s2.0.json:12:    "throughput_rtx": 1.3,
./benchmark_results/stress_sweep_s2.0.json:13:    "total_chunks_analyzed": 16,
./benchmark_results/stress_sweep_s2.0.json:34:    "clean_disconnects": 10,
./benchmark_results/stress_sweep_s2.0.json:39:    "throughput_rtx": 10.5,
./benchmark_results/stress_sweep_s2.0.json:40:    "total_chunks_analyzed": 185,
./benchmark_results/stress_sweep_s2.0.json:52:      "drift_ms": -261.1
./benchmark_results/stress_sweep_s2.0.json:65:    "clean_disconnects": 50,
./benchmark_results/stress_sweep_s2.0.json:70:    "throughput_rtx": 34.2,
./benchmark_results/stress_sweep_s2.0.json:71:    "total_chunks_analyzed": 942,
./benchmark_results/stress_sweep_s2.0.json:83:      "drift_ms": -65.2
./benchmark_results/stress_sweep_s2.0.json:96:    "clean_disconnects": 100,
./benchmark_results/stress_sweep_s2.0.json:101:    "throughput_rtx": 42.8,
./benchmark_results/stress_sweep_s2.0.json:102:    "total_chunks_analyzed": 1884,
./benchmark_results/stress_sweep_s2.0.json:114:      "drift_ms": 141.2
./benchmark_results/stress_sweep_s2.0.json:127:    "clean_disconnects": 250,
./benchmark_results/stress_sweep_s2.0.json:132:    "throughput_rtx": 57.5,
./benchmark_results/stress_sweep_s2.0.json:133:    "total_chunks_analyzed": 4706,
./benchmark_results/stress_sweep_s2.0.json:145:      "drift_ms": -99.8
./benchmark_results/stress_sweep_s2.0.json:158:    "clean_disconnects": 500,
./benchmark_results/stress_sweep_s2.0.json:163:    "throughput_rtx": 69.8,
./benchmark_results/stress_sweep_s2.0.json:164:    "total_chunks_analyzed": 9414,
./benchmark_results/stress_sweep_s2.0.json:176:      "drift_ms": -2852.1
./benchmark_results/stress_sweep_s2.0.json:185:    "clean_disconnects": 1000,
./benchmark_results/stress_sweep_s2.0.json:190:    "throughput_rtx": 213.9,
./benchmark_results/stress_sweep_s2.0.json:191:    "total_chunks_analyzed": 100,
./benchmark_results/stress_sweep_s2.0.json:203:      "drift_ms": 14.4
./benchmark_results/capacity_c1_sustained15.json:3:  "sustained_s": 15.0,
./benchmark_results/capacity_c1_sustained15.json:7:  "clean_disconnects": 1,
./benchmark_results/capacity_c1_sustained15.json:12:  "throughput_rtx": 0.94,
./benchmark_results/capacity_c1_sustained15.json:13:  "total_chunks_analyzed": 25,
./benchmark_results/capacity_c1_sustained15.json:27:    "drift_ms": 635.9
./benchmark_results/capacity_c30_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c30_s60.json:7:  "clean_disconnects": 30,
./benchmark_results/capacity_c30_s60.json:12:  "throughput_rtx": 14.76,
./benchmark_results/capacity_c30_s60.json:13:  "total_chunks": 563,
./benchmark_results/capacity_c30_s60.json:27:    "drift_ms": -2.2
./benchmark_results/capacity_c150_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c150_s60.json:7:  "clean_disconnects": 150,
./benchmark_results/capacity_c150_s60.json:12:  "throughput_rtx": 65.27,
./benchmark_results/capacity_c150_s60.json:13:  "total_chunks": 2158,
./benchmark_results/capacity_c150_s60.json:27:    "drift_ms": 1863.8
./benchmark_results/capacity_c175_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c175_s60.json:7:  "clean_disconnects": 130,
./benchmark_results/capacity_c175_s60.json:12:  "throughput_rtx": 79.84,
./benchmark_results/capacity_c175_s60.json:13:  "total_chunks": 258,
./benchmark_results/capacity_c175_s60.json:27:    "drift_ms": 6445.2
./benchmark_results/capacity_c5_sustained20.json:3:  "sustained_s": 20.0,
./benchmark_results/capacity_c5_sustained20.json:7:  "clean_disconnects": 5,
./benchmark_results/capacity_c5_sustained20.json:12:  "throughput_rtx": 1.09,
./benchmark_results/capacity_c5_sustained20.json:13:  "total_chunks_analyzed": 151,
./benchmark_results/capacity_c5_sustained20.json:27:    "drift_ms": 21.7
./benchmark_results/loadtest_concurrent_5_summary.json:13:  "total_chunks": 89,
./benchmark_results/loadtest_concurrent_5_summary.json:16:  "throughput_rtx": 5.3
./benchmark_results/loadtest_concurrent_25_summary.json:13:  "total_chunks": 478,
./benchmark_results/loadtest_concurrent_25_summary.json:16:  "throughput_rtx": 23.3
./benchmark_results/local_batched_sweep.json:5:    "total_chunks": 114,
./benchmark_results/local_batched_sweep.json:14:    "throughput_rtx": 2.4
./benchmark_results/local_batched_sweep.json:19:    "total_chunks": 704,
./benchmark_results/local_batched_sweep.json:28:    "throughput_rtx": 11.2
./benchmark_results/local_batched_sweep.json:33:    "total_chunks": 1423,
./benchmark_results/local_batched_sweep.json:42:    "throughput_rtx": 21.2
./benchmark_results/local_batched_sweep.json:47:    "total_chunks": 3987,
./benchmark_results/local_batched_sweep.json:56:    "throughput_rtx": 48.1
./benchmark_results/local_batched_sweep.json:61:    "total_chunks": 7330,
./benchmark_results/local_batched_sweep.json:70:    "throughput_rtx": 86.5
./benchmark_results/stress_c250_s2.0.json:6:  "clean_disconnects": 250,
./benchmark_results/stress_c250_s2.0.json:11:  "throughput_rtx": 57.5,
./benchmark_results/stress_c250_s2.0.json:12:  "total_chunks_analyzed": 4706,
./benchmark_results/stress_c250_s2.0.json:24:    "drift_ms": -99.8
./benchmark_results/capacity_c110_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c110_s60.json:7:  "clean_disconnects": 110,
./benchmark_results/capacity_c110_s60.json:12:  "throughput_rtx": 53.84,
./benchmark_results/capacity_c110_s60.json:13:  "total_chunks": 2074,
./benchmark_results/capacity_c110_s60.json:27:    "drift_ms": 180.7
./benchmark_results/capacity_c140_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c140_s60.json:7:  "clean_disconnects": 140,
./benchmark_results/capacity_c140_s60.json:12:  "throughput_rtx": 68.69,
./benchmark_results/capacity_c140_s60.json:13:  "total_chunks": 2644,
./benchmark_results/capacity_c140_s60.json:27:    "drift_ms": 252.6
./benchmark_results/capacity_c1_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c1_s60.json:7:  "clean_disconnects": 1,
./benchmark_results/capacity_c1_s60.json:12:  "throughput_rtx": 0.78,
./benchmark_results/capacity_c1_s60.json:13:  "total_chunks": 16,
./benchmark_results/capacity_c50_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c50_s60.json:7:  "clean_disconnects": 50,
./benchmark_results/capacity_c50_s60.json:12:  "throughput_rtx": 24.3,
./benchmark_results/capacity_c50_s60.json:13:  "total_chunks": 940,
./benchmark_results/capacity_c50_s60.json:27:    "drift_ms": -41.2
./benchmark_results/capacity_c30_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c30_s45.json:7:  "clean_disconnects": 30,
./benchmark_results/capacity_c30_s45.json:12:  "throughput_rtx": 31.93,
./benchmark_results/capacity_c30_s45.json:13:  "total_chunks": 2118,
./benchmark_results/capacity_c30_s45.json:27:    "drift_ms": 14.4
./benchmark_results/stress_c50_s2.0.json:6:  "clean_disconnects": 50,
./benchmark_results/stress_c50_s2.0.json:11:  "throughput_rtx": 4.3,
./benchmark_results/stress_c50_s2.0.json:12:  "total_chunks_analyzed": 939,
./benchmark_results/stress_c50_s2.0.json:24:    "drift_ms": 7.7
./benchmark_results/loadtest_concurrent_10_summary.json:13:  "total_chunks": 180,
./benchmark_results/loadtest_concurrent_10_summary.json:16:  "throughput_rtx": 9.7
./benchmark_results/capacity_c5_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c5_s45.json:7:  "clean_disconnects": 5,
./benchmark_results/capacity_c5_s45.json:12:  "throughput_rtx": 5.09,
./benchmark_results/capacity_c5_s45.json:13:  "total_chunks": 344,
./benchmark_results/capacity_c5_s45.json:27:    "drift_ms": -245.8
./benchmark_results/stress_c1000_s2.0.json:6:  "clean_disconnects": 1000,
./benchmark_results/stress_c1000_s2.0.json:11:  "throughput_rtx": 213.9,
./benchmark_results/stress_c1000_s2.0.json:12:  "total_chunks_analyzed": 100,
./benchmark_results/stress_c1000_s2.0.json:24:    "drift_ms": 14.4
./benchmark_results/loadtest_concurrent_1_summary.json:13:  "total_chunks": 16,
./benchmark_results/loadtest_concurrent_1_summary.json:16:  "throughput_rtx": 1.2
./benchmark_results/capacity_c80_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c80_s45.json:7:  "clean_disconnects": 80,
./benchmark_results/capacity_c80_s45.json:12:  "throughput_rtx": 83.54,
./benchmark_results/capacity_c80_s45.json:13:  "total_chunks": 5580,
./benchmark_results/capacity_c80_s45.json:27:    "drift_ms": 46.6
./benchmark_results/capacity_c250_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c250_s60.json:7:  "clean_disconnects": 0,
./benchmark_results/capacity_c250_s60.json:12:  "throughput_rtx": 173.78,
./benchmark_results/capacity_c250_s60.json:13:  "total_chunks": 9601,
./benchmark_results/capacity_c250_s60.json:27:    "drift_ms": 31.2
./benchmark_results/local_concurrent_proper.json:5:    "total_chunks": 114,
./benchmark_results/local_concurrent_proper.json:13:    "throughput_rtx": 45.2,
./benchmark_results/local_concurrent_proper.json:19:    "total_chunks": 704,
./benchmark_results/local_concurrent_proper.json:27:    "throughput_rtx": 107.6,
./benchmark_results/local_concurrent_proper.json:33:    "total_chunks": 1423,
./benchmark_results/local_concurrent_proper.json:41:    "throughput_rtx": 129.0,
./benchmark_results/local_concurrent_proper.json:47:    "total_chunks": 3987,
./benchmark_results/local_concurrent_proper.json:55:    "throughput_rtx": 169.1,
./benchmark_results/local_concurrent_proper.json:61:    "total_chunks": 7330,
./benchmark_results/local_concurrent_proper.json:69:    "throughput_rtx": 168.8,
./benchmark_results/stress_c500_s2.0.json:6:  "clean_disconnects": 500,
./benchmark_results/stress_c500_s2.0.json:11:  "throughput_rtx": 69.8,
./benchmark_results/stress_c500_s2.0.json:12:  "total_chunks_analyzed": 9414,
./benchmark_results/stress_c500_s2.0.json:24:    "drift_ms": -2852.1
./benchmark_results/capacity_c50_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c50_s45.json:7:  "clean_disconnects": 50,
./benchmark_results/capacity_c50_s45.json:12:  "throughput_rtx": 52.9,
./benchmark_results/capacity_c50_s45.json:13:  "total_chunks": 3523,
./benchmark_results/capacity_c50_s45.json:27:    "drift_ms": 35.4
./benchmark_results/local_direct_latency.json:7:  "total_chunks": 7330,
./benchmark_results/local_direct_latency.json:17:  "throughput_rtx": 110.7
./benchmark_results/capacity_c120_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c120_s60.json:7:  "clean_disconnects": 120,
./benchmark_results/capacity_c120_s60.json:12:  "throughput_rtx": 54.38,
./benchmark_results/capacity_c120_s60.json:13:  "total_chunks": 2210,
./benchmark_results/capacity_c120_s60.json:27:    "drift_ms": -182.1
./benchmark_results/capacity_c200_s60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c200_s60.json:7:  "clean_disconnects": 110,
./benchmark_results/capacity_c200_s60.json:12:  "throughput_rtx": 98.54,
./benchmark_results/capacity_c200_s60.json:13:  "total_chunks": 0,
./benchmark_results/gpu_ceiling_test.json:7:    "throughput_rtx": 54.6,
./benchmark_results/gpu_ceiling_test.json:16:    "throughput_rtx": 135.9,
./benchmark_results/gpu_ceiling_test.json:25:    "throughput_rtx": 183.4,
./benchmark_results/gpu_ceiling_test.json:34:    "throughput_rtx": 158.3,
./benchmark_results/gpu_ceiling_test.json:43:    "throughput_rtx": 148.6,
./benchmark_results/gpu_ceiling_test.json:52:    "throughput_rtx": 184.7,
./benchmark_results/gpu_ceiling_test.json:61:    "throughput_rtx": 224.4,
./benchmark_results/gpu_ceiling_test.json:70:    "throughput_rtx": 166.7,
./benchmark_results/gpu_ceiling_test.json:79:    "throughput_rtx": 209.6,
./benchmark_results/gpu_ceiling_test.json:88:    "throughput_rtx": 167.9,
./benchmark_results/gpu_ceiling_test.json:97:    "throughput_rtx": 211.1,
./benchmark_results/stress_c100_s2.0.json:6:  "clean_disconnects": 100,
./benchmark_results/stress_c100_s2.0.json:11:  "throughput_rtx": 4.3,
./benchmark_results/stress_c100_s2.0.json:12:  "total_chunks_analyzed": 1885,
./benchmark_results/stress_c100_s2.0.json:24:    "drift_ms": -4.8
./benchmark_results/capacity_c100_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c100_s45.json:7:  "clean_disconnects": 100,
./benchmark_results/capacity_c100_s45.json:12:  "throughput_rtx": 105.72,
./benchmark_results/capacity_c100_s45.json:13:  "total_chunks": 7007,
./benchmark_results/capacity_c100_s45.json:27:    "drift_ms": 94.8
./benchmark_results/overall_concurrent.json:8:    "throughput_rtx": 49.7,
./benchmark_results/overall_concurrent.json:17:    "throughput_rtx": 117.9,
./benchmark_results/overall_concurrent.json:26:    "throughput_rtx": 165.0,
./benchmark_results/overall_concurrent.json:35:    "throughput_rtx": 257.2,
./benchmark_results/overall_concurrent.json:44:    "throughput_rtx": 241.7,
./benchmark_results/capacity_c5_sustained60.json:3:  "sustained_s": 60.0,
./benchmark_results/capacity_c5_sustained60.json:7:  "clean_disconnects": 2,
./benchmark_results/capacity_c5_sustained60.json:12:  "throughput_rtx": 1.06,
./benchmark_results/capacity_c5_sustained60.json:13:  "total_chunks_analyzed": 178,
./benchmark_results/capacity_c5_sustained60.json:27:    "drift_ms": 1257.9
./benchmark_results/capacity_c10_s45.json:3:  "sustained_s": 45.0,
./benchmark_results/capacity_c10_s45.json:7:  "clean_disconnects": 10,
./benchmark_results/capacity_c10_s45.json:12:  "throughput_rtx": 10.41,
./benchmark_results/capacity_c10_s45.json:13:  "total_chunks": 711,
./benchmark_results/capacity_c10_s45.json:27:    "drift_ms": 32.6
./benchmark_results/stress_c10_s2.0.json:6:  "clean_disconnects": 10,
./benchmark_results/stress_c10_s2.0.json:11:  "throughput_rtx": 4.1,
./benchmark_results/stress_c10_s2.0.json:12:  "total_chunks_analyzed": 185,
./benchmark_results/stress_c10_s2.0.json:24:    "drift_ms": -234.1
./benchmark_results/stress_c1_s2.0.json:6:  "clean_disconnects": 1,
./benchmark_results/stress_c1_s2.0.json:11:  "throughput_rtx": 1.3,
./benchmark_results/stress_c1_s2.0.json:12:  "total_chunks_analyzed": 16,
./benchmark_results/realistic_full_pipeline.json:9:    "throughput_rtx": 53.7,
./benchmark_results/realistic_full_pipeline.json:20:    "throughput_rtx": 136.7,
./benchmark_results/realistic_full_pipeline.json:31:    "throughput_rtx": 250.5,
./benchmark_results/realistic_full_pipeline.json:42:    "throughput_rtx": 239.8,
./benchmark_results/realistic_full_pipeline.json:53:    "throughput_rtx": 213.8,
./benchmark_results/realistic_full_pipeline.json:64:    "throughput_rtx": 213.2,
./benchmark_results/realistic_full_pipeline.json:75:    "throughput_rtx": 202.4,
./benchmark_results/realistic_full_pipeline.json:86:    "throughput_rtx": 186.2,
./benchmark_results/realistic_full_pipeline.json:97:    "throughput_rtx": 160.7,
./benchmark_results/realistic_full_pipeline.json:108:    "throughput_rtx": 202.5,
./benchmark_results/stress_sweep_h100_s2.0.json:7:    "clean_disconnects": 1,
./benchmark_results/stress_sweep_h100_s2.0.json:12:    "throughput_rtx": 1.3,
./benchmark_results/stress_sweep_h100_s2.0.json:13:    "total_chunks_analyzed": 16,
./benchmark_results/stress_sweep_h100_s2.0.json:34:    "clean_disconnects": 10,
./benchmark_results/stress_sweep_h100_s2.0.json:39:    "throughput_rtx": 10.5,
./benchmark_results/stress_sweep_h100_s2.0.json:40:    "total_chunks_analyzed": 185,
./benchmark_results/stress_sweep_h100_s2.0.json:52:      "drift_ms": -261.1
./benchmark_results/stress_sweep_h100_s2.0.json:65:    "clean_disconnects": 50,
./benchmark_results/stress_sweep_h100_s2.0.json:70:    "throughput_rtx": 34.2,
./benchmark_results/stress_sweep_h100_s2.0.json:71:    "total_chunks_analyzed": 942,
./benchmark_results/stress_sweep_h100_s2.0.json:83:      "drift_ms": -65.2
./benchmark_results/stress_sweep_h100_s2.0.json:96:    "clean_disconnects": 100,
./benchmark_results/stress_sweep_h100_s2.0.json:101:    "throughput_rtx": 42.8,
./benchmark_results/stress_sweep_h100_s2.0.json:102:    "total_chunks_analyzed": 1884,
./benchmark_results/stress_sweep_h100_s2.0.json:114:      "drift_ms": 141.2
./benchmark_results/stress_sweep_h100_s2.0.json:127:    "clean_disconnects": 250,
./benchmark_results/stress_sweep_h100_s2.0.json:132:    "throughput_rtx": 57.5,
./benchmark_results/stress_sweep_h100_s2.0.json:133:    "total_chunks_analyzed": 4706,
./benchmark_results/stress_sweep_h100_s2.0.json:145:      "drift_ms": -99.8
./benchmark_results/stress_sweep_h100_s2.0.json:158:    "clean_disconnects": 500,
./benchmark_results/stress_sweep_h100_s2.0.json:163:    "throughput_rtx": 69.8,
./benchmark_results/stress_sweep_h100_s2.0.json:164:    "total_chunks_analyzed": 9414,


**Exploring benchmark data structure**


rg -n "throughput_rtx|clean_disconnects|total_chunks_analyzed|drift_ms|capacity" nemotron_asr nemotron production | sed -n '1,280p'


Output:
Chunk ID: d5ab26
Wall time: 0.0502 seconds
Process exited with code 0
Original token count: 438
Output:
nemotron_asr/nemotron_asr.py:94:    only ~300 are speaking at any moment. VAD effectively 3x GPU capacity.
nemotron_asr/nemotron_asr.py:476:        # VAD skips those chunks entirely, effectively 3x GPU capacity.
nemotron_asr/nemotron_asr.py:528:            Rejects connections when container is at capacity.
nemotron_asr/nemotron_asr.py:697:                            # listening (not speaking), so this 3x effective GPU capacity.
production/server.py:9:  - CPU-based Silero VAD gating (filters silence before GPU, ~3x effective capacity)
production/server.py:11:  - Backpressure: rejects new connections when at capacity (fast-fail > silent degrade)
production/server.py:841:    Backpressure: Rejects with 1013 when at capacity.
nemotron_asr/vad.py:3:WHY: Filters silence BEFORE GPU pipeline. 3x GPU capacity for free.
nemotron/README.md:315:2. **VAD gating** — Added CPU-based Silero VAD (`nemotron_asr/vad.py`) that filters silence before GPU pipeline. With voice-agent traffic (~30% speech duty cycle), effectively 3x GPU capacity.
nemotron/README.md:328:**Production capacity (A100-80GB, per local benchmarks):**
nemotron/README.md:332:- With VAD: effective capacity ~1500-2000 connected sessions per GPU
nemotron/README.md:355:- VAD was disabled for this test; enabling it would effectively 3x the "speaking" capacity
nemotron/README.md:392:Migrated to Vast.ai for capacity testing. Key advantage: Vast.ai's two-step routing (`/route/` for worker discovery, then direct TCP to GPU) eliminates all proxy overhead. No LB in the WebSocket data path.
nemotron/README.md:501:5. **Per-worker capacity test** (single-pass, accurate WER):
nemotron/README.md:537:4. Autoscaling config based on measured capacity (scale at 80, cap at 140 per A100-80GB)


**Reviewing README for capacity details**


I found historical capacity notes and benchmark methodology in the repo’s `nemotron/README.md`. I’m reading those sections next to compare your current serverless setup against the original streaming benchmark assumptions.


sed -n '1,220p' nemotron_asr/nemotron_asr.py


Output:
Chunk ID: 53cf2d
Wall time: 0.0509 seconds
Process exited with code 0
Original token count: 2343
Output:
### 2026-01-25 - Lhotse CutSet + SHAR Pipeline
- Added `scripts/prep_lhotse_cuts.py` to convert NeMo manifests into Lhotse CutSet JSONL(.gz) with language tags
- Extended `scripts/finetune_indic_v3.py` with Lhotse/SHAR CLI inputs (`--train_cuts`, `--train_shar_dir`, etc.)
- Started full train CutSet export to `/workspace/asropendata/ASR_DATASET/nemo_prepared/lhotse_cuts/train_cuts.jsonl.gz` (log: `lhotse_cuts_train_full.log`)
- Added multiprocessing support to `scripts/prep_lhotse_cuts.py` and restarted full export with max CPU workers + unbuffered logging
- Restarted full train SHAR export to `/workspace/asropendata/ASR_DATASET/nemo_prepared/lhotse_shar/train/` with WAV conversion (log: `lhotse_shar_train_full.log`)
- WAV SHAR export hit audio duration mismatches; restarted with `LHOTSE_AUDIO_DURATION_MISMATCH_TOLERANCE=10` + `--fault-tolerant` to continue
- Next: after SHAR completes, restart training with `--train_shar_dir` + `--val_shar_dir` (val SHAR already in `/workspace/asropendata/ASR_DATASET/nemo_prepared/lhotse_shar/val/`)

### 2026-02-08 - Production Hardening for Modal ASR Deployment

**Context**: Preparing Modal deployment (`nemotron_asr/`) for 1000+ concurrent users in voice-agent mobile app.

**Changes to `nemotron_asr/nemotron_asr.py`:**
1. **Event-driven batching** — Replaced fixed 300ms `asyncio.sleep()` with `asyncio.Event` + 50ms max_wait. Inference loop fires immediately when data arrives (high load) or after 50ms timeout (low load). 6x latency improvement over fixed sleep.
2. **VAD gating** — Added CPU-based Silero VAD (`nemotron_asr/vad.py`) that filters silence before GPU pipeline. With voice-agent traffic (~30% speech duty cycle), effectively 3x GPU capacity.
3. **Backpressure** — WebSocket endpoint rejects connections with HTTP 1013 when container exceeds `max_active_streams` (default 500). Fast-fail prevents silent degradation.
4. **Health endpoint** — `GET /health` returns container status, active streams, VRAM usage. Returns 503 when overloaded for load balancer integration.
5. **Pinned NeMo** — Changed `@main` to specific commit hash `@2d0ce15` to prevent upstream breakage.
6. **Proper secrets** — HF token now uses named Modal Secret `hf-token` with dev fallback.
7. **Scaling config** — `min_containers=1` (no cold starts), `buffer_containers=1` (burst absorption), `scaledown_window=300` (5min, not 1hr).
8. **Fixed 1s sleep** — Replaced `await asyncio.sleep(1.0)` on stream end with event signal + 100ms yield.
9. **Structured metrics** — Rolling-window p50/p99 for GPU time, cycle time, batch sizes, VRAM usage.

**New file: `nemotron_asr/vad.py`**
- Silero VAD v5 wrapper with hysteresis (3 consecutive silence frames before suppressing)
- Per-stream silence tracking, safe fallback on errors

**Production capacity (A100-80GB, per local benchmarks):**
- 1000 concurrent streams @ 50.3GB VRAM (63% utilization)
- First token: 58ms median, 65ms P99 at 1000 streams
- WER stable at 21.9% from 50 to 1000 streams (no degradation)
- With VAD: effective capacity ~1500-2000 connected sessions per GPU

### 2026-02-08 - Modal Stress Test Results (Event-Driven Batching, No VAD)

Sweep: 1 to 1000 concurrent WebSocket streams, A100 on Modal, 2x real-time audio pace.

| Conc | OK | Err | WER% | CER% | 1stTok p50 | 1stTok p99 | Connect p50 | ChkP99 | Thruput |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 | 1 | 0 | 37.5 | 19.8 | 206ms | 206ms | 398ms | -397ms | 1.3x |
| 10 | 10 | 0 | 23.2 | 10.0 | 446ms | 485ms | 441ms | -91ms | 10.3x |
| 50 | 50 | 0 | 31.0 | 16.9 | 472ms | 964ms | 1.3s | 170ms | 37.0x |
| 100 | 100 | 0 | 31.2 | 16.9 | 626ms | 1.1s | 3.4s | 218ms | 53.9x |
| 250 | 250 | 0 | 31.2 | 16.9 | 577ms | 1.4s | 13.7s | 362ms | 71.3x |
| 500 | 500 | 0 | 45.3 | 33.7 | 789ms | 8.2s | 29.7s | 6.6s | 63.1x |
| 1000 | 998 | 2 | 91.1 | 89.2 | 1.8s | 6.9s | 20.0s | 5.9s | 38.9x |

**Key findings:**
- 0-250 streams: stable zone (WER 23-31%, first token <600ms, 0 errors)
- 250-500: degradation begins (WER jumps to 45%, connect times blow up to 30s)
- 1000: severe overload (WER 91%, 2 timeout errors, connect p50=20s)
- The breaking point is ~250 concurrent *all-speaking* streams on Modal A100
- Connection establishment is the bottleneck at scale (29s p50 at 500 streams)
- Streams peak at 183 despite 1000 requested — Modal's tunnel serializes connections
- VAD was disabled for this test; enabling it would effectively 3x the "speaking" capacity

### 2026-02-08 - RunPod Load Balancing Deployment + Benchmark

Deployed same ASR server to RunPod as Load Balancing endpoint (direct WebSocket, no tunnel).
Image: `ghcr.io/maya-ai-ml-trainings-cookbooks/hindi-nemotron-asr:v2-slim` on H100 (AMPERE_80 pool).

| Conc | OK | Err | WER% | 1stTok p50 | Drift | Platform |
|---:|---:|---:|---:|---:|---:|:---|
| 1 | 1 | 0 | 37.5 | **209ms** | - | RunPod |
| 10 | 10 | 0 | 22.9 | **209ms** | -234ms | RunPod |
| 50 | 50 | 0 | 31.2 | **215ms** | **7.7ms** | RunPod |
| 100 | 100 | 0 | 33.2 | **206ms** | **-4.8ms** | RunPod |
| 100 | 100 | 0 | 31.2 | 590ms | 141ms | Modal H100 |

**Key findings:**
- First-token latency: **206ms on RunPod vs 590ms on Modal** at 100 concurrent (2.9x faster)
- **Zero drift** on RunPod (7.7ms / -4.8ms) vs Modal's 141ms drift at 100 streams
- Zero errors across all RunPod levels
- WER consistent at 31-33% (streaming WER, same model)
- Connect time high (serialized by single worker) — would distribute with multi-worker LB
- RunPod direct WebSocket eliminates Modal's tunnel bottleneck entirely

### 2026-02-09 - RunPod LB Limitations for WebSocket (Critical Finding)

RunPod's Load Balancing endpoint is an HTTP Layer 7 proxy. Critical issues discovered:
1. **WS upgrade serialization**: 20-200s per WebSocket connection through the LB proxy, regardless of approach (sequential or concurrent)
2. **5.5-minute processing timeout**: LB terminates connections after 5.5 min (`524` error) — kills voice agent sessions
3. **`proxy.runpod.net` returns 403 for serverless**: Direct worker access not available for serverless workers
4. **Worker instability**: Workers crash/restart between tests; cold starts take 3-10 min

Added `/connect` endpoint to `runpod_deploy/server.py` that returns `RUNPOD_POD_ID`, GPU name, direct WS URL, and all `RUNPOD_*` env vars. Confirmed GPU is `NVIDIA A100-SXM4-80GB` via the endpoint's `RUNPOD_GPU_NAME` env var.

**Conclusion**: RunPod LB is not suitable for WebSocket streaming ASR at production scale. Their own docs recommend "Expose HTTP/TCP Ports" for persistent connections, but this feature doesn't work for serverless workers.

### 2026-02-09 - Vast.ai Direct-Access Per-Worker Capacity Test

Migrated to Vast.ai for capacity testing. Key advantage: Vast.ai's two-step routing (`/route/` for worker discovery, then direct TCP to GPU) eliminates all proxy overhead. No LB in the WebSocket data path.

**Setup**: A100-SXM4-80GB on Vast.ai ($0.656/hr), Docker image `bharathkumar192/hindi-nemotron-asr:v3-direct`, direct IP:port access, 1x real-time audio pace, 45s sustained load per level.

| Conc | OK | Err | WER% | FT p50 | FT p99 | Conn p50 | Drift | Peak | Thru |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 | 1 | 0 | 33.3% | 306ms | 306ms | 404ms | - | 1 | 1.0x |
| 5 | 5 | 0 | 81.3%* | 371ms | 374ms | 405ms | -246ms | 5 | 5.1x |
| 10 | 10 | 0 | 47.6% | 357ms | 434ms | 427ms | 33ms | 10 | 10.4x |
| 20 | 20 | 0 | 52.3% | 365ms | 441ms | 425ms | -10ms | 20 | 21.0x |
| 30 | 30 | 0 | 43.4% | 394ms | 504ms | 435ms | 14ms | 30 | 31.9x |
| 50 | 50 | 0 | 46.9% | 390ms | 496ms | 452ms | 35ms | 50 | 52.9x |
| 80 | 80 | 0 | 45.7% | 333ms | 525ms | 831ms | 47ms | 80 | 83.5x |
| 100 | 100 | 0 | 44.0% | 461ms | 638ms | 566ms | 95ms | 100 | 105.7x |

*c=5 WER anomaly is audio-looping artifact; WER stable at 43-52% from c=10+

**Key findings:**
- **Zero errors at all concurrency levels** (0 to 100)
- **Connect time: 400-830ms** (vs RunPod LB's 50,000-134,000ms)
- **First token p50 scales from 306ms (c=1) to 461ms (c=100)** — still under 500ms
- **First token p99 stays under 640ms** even at c=100
- **Linear throughput**: 105.7x real-time at c=100 (100 streams, each at 1x)
- **Drift under 100ms** at c=100 (stable sessions, no accumulating latency)
- **WER stable at 43-52%** from c=10 to c=100 (streaming WER with 1x pace + audio looping)
- The c=5 WER=81.3% is a test artifact (audio looping boundary); consistent WER across c=10-100 confirms no quality degradation

### 2026-02-09 - Definitive Per-Worker Saturation Test (Single-Pass, Accurate WER)

**Previous looped-audio sweep (c=1-300, 60s sustained, audio looping)** — showed zero connection errors at all levels but WER >100% due to looping artifacts. Proved connections work at scale but WER data was unreliable.

**Definitive single-pass sweep** — fixed VRAM monitoring (`total_mem` -> `total_memory`), added `--no-loop` mode. Each stream sends its audio ONCE at 1x real-time (no looping). 20 FLEURS Hindi samples, A100-SXM4-80GB on Vast.ai ($0.88/hr).

| Conc | OK | WER% | FT p50 | FT p99 | Ch/strm | VRAM | Status |
|---:|---:|---:|---:|---:|---:|---:|:---|
| 1 | 1 | 37.5% | 138ms | 138ms | 16.0 | 10.4GB | HEALTHY |
| 10 | 10 | 22.9% | 191ms | 201ms | 18.5 | 10.4GB | HEALTHY |
| 30 | 30 | 29.9% | 208ms | 263ms | 18.8 | 10.4GB | HEALTHY |
| 50 | 50 | 31.2% | 211ms | 365ms | 18.8 | 10.4GB | HEALTHY |


Output:
Chunk ID: c3505f
Wall time: 0.0507 seconds
Process exited with code 0
Original token count: 1561
Output:
| 50 | 50 | 31.2% | 211ms | 365ms | 18.8 | 10.4GB | HEALTHY |
| 80 | 80 | 33.5% | 402ms | 2.5s | 18.9 | 10.4GB | HEALTHY |
| 100 | 100 | 33.2% | 332ms | 3.6s | 18.9 | 10.4GB | HEALTHY |
| 110 | 110 | 32.4% | 364ms | 408ms | 18.9 | 10.4GB | HEALTHY |
| 120 | 120 | 35.4% | 210ms | 7.7s | 18.4 | 10.4GB | EDGE |
| 130 | 130 | 37.0% | 425ms | 9.8s | 17.8 | 10.4GB | EDGE |
| 140 | 140 | 33.3% | 443ms | 481ms | 18.9 | 10.4GB | EDGE |
| 150 | 150 | **52.0%** | 340ms | 11.7s | **14.4** | 10.4GB | DEGRADED |
| 175 | 175 | **96.9%** | 197ms | 8.3s | **1.5** | 10.4GB | CRITICAL |
| 200 | 200 | **100%** | - | - | **0.0** | 10.4GB | SATURATED |

**Key findings:**
- **Zero errors at all levels** — model degrades gracefully, never crashes
- **WER stable at 22-37% from c=1 to c=140** — no quality loss in healthy zone
- **WER jumps to 52% at c=150** — GPU falling behind
- **WER = 100% at c=200** — GPU fully saturated, zero output
- **VRAM constant at 10.4GB / 85GB (12%)** — memory is NOT the bottleneck, GPU compute is
- **Chunks/stream is the true health metric**: 18-19 when healthy, drops to 1.5 at c=175, 0 at c=200

**GPU Saturation Zones (A100-SXM4-80GB):**

| Zone | Streams | WER | FT p50 | Recommendation |
|:---|---:|---:|---:|:---|
| **Safe** | 1-50 | 22-31% | <211ms | Production default target |
| **Optimal** | 50-100 | 31-33% | <402ms | Max steady-state per worker |
| **Edge** | 100-140 | 33-37% | <443ms | Short burst tolerance only |
| **Degraded** | 150 | 52% | 340ms | Autoscale immediately |
| **Critical** | 175 | 97% | - | Reject new connections |
| **Dead** | 200+ | 100% | - | Hard 503, no processing |

**Autoscaling rules (production):**
```
target_streams_per_worker: 50-80
scale_up_at:   active_streams > 80   (hit 80% of safe=100)
scale_down_at: active_streams < 30   (drop below 60% of target=50)
hard_cap:      140                   (reject above, return 503)
```
With VAD gating (~30% speech): effective **150-420 connected sessions per GPU**
Cost: $0.66-0.88/hr per A100-80GB = **~$0.006-0.009/hr per active stream**

**Platform comparison for WebSocket ASR:**

| Feature | RunPod LB | Modal | Vast.ai Direct |
|:---|:---|:---|:---|
| WS connect time | 50-134 seconds | 0.4-3.4 seconds | **75-620ms** |
| First token @ c=100 | 206ms (2x speed) | 590ms | **332ms** |
| Saturation point | 100 (LB limit) | 250 (tunnel limit) | **140 (GPU limit)** |
| 5.5min timeout | Yes (kills sessions) | No | **No (direct TCP)** |
| VRAM at c=100 | Unknown | Unknown | **10.4GB/85GB (12%)** |
| Autoscaling | Built-in | Built-in | Serverless (PyWorker) |
| Cost (A100-80GB) | ~$2.49/hr | ~$3.95/hr | **~$0.66-0.88/hr** |
| Direct GPU access | No (proxy) | No (tunnel) | **Yes** |
| Reliability | Managed DC | Managed DC | Community (0.95+) |

### 2026-02-09 - Cleanup & Summary for Vast.ai Serverless Rewrite

**What was done in this session (Feb 8-9):**

1. **Modal stress test** (from prior session): Swept c=1 to c=1000 on Modal A100. Found Modal tunnel bottleneck at 250+ streams (WER degrades, connect times blow up to 30s). Best: 206ms first-token on RunPod vs 590ms on Modal at c=100.

2. **RunPod LB deployment**: Built Docker image (`ghcr.io/maya-ai-ml-trainings-cookbooks/hindi-nemotron-asr`), deployed as RunPod Load Balancing endpoint. Added `/connect` endpoint returning `RUNPOD_POD_ID`, GPU name, env vars. Confirmed A100-SXM4-80GB via `RUNPOD_GPU_NAME` env var.

3. **RunPod LB issues discovered** (critical):
   - LB serializes WebSocket upgrades: 20-200s per connection regardless of approach
   - 5.5-minute processing timeout kills voice agent sessions
   - `proxy.runpod.net` returns 403 for serverless workers (direct access blocked)
   - Worker instability (crashes between tests, 3-10 min cold starts)
   - Conclusion: **RunPod LB is not suitable for WebSocket streaming ASR**

4. **Migrated to Vast.ai**: Direct TCP access (no proxy). Connection time dropped from 50-134 seconds to 75-620ms (200x improvement).

5. **Per-worker capacity test** (single-pass, accurate WER):
   - Safe zone: **1-50 concurrent streams** (WER 22-31%, FT <211ms)
   - Optimal: **50-100 streams** (WER 31-33%, FT <402ms)
   - Edge: **100-140 streams** (WER 33-37%, FT p99 spikes)
   - Saturated: **150+ streams** (WER >52%, chunks dropping)
   - Dead: **200+ streams** (WER 100%, zero output)
   - VRAM constant at 10.4GB/85GB — GPU compute is the bottleneck, not memory

6. **Fixed bugs**: VRAM monitoring (`total_mem` → `total_memory`), jiwer tuple error, WER accuracy (added `--no-loop` mode).

**Issues NOT fixed / remaining for production:**
- Vast.ai serverless PyWorker integration (metrics reporting loop embedded but `/route/` never tested end-to-end)
- Vast.ai tests ran on bare-metal GPU Pods, NOT serverless workers (no autoscaling)
- No Triton Inference Server (current uses NeMo Python pipeline directly)
- No TLS termination for WebSocket (need Cloudflare or nginx in front)
- No client reconnection logic for session recovery
- No session >5.5 min test (RunPod timeout verified in docs but not tested on Vast)
- GPU monitoring shows `memory_allocated` only (doesn't capture CUDA reserved memory)

**Cleanup done:**
- Deleted: `runpod_deploy/`, `runpod_docs/`, `modal_docs/`, all test scripts, wav files
- Destroyed: All Vast.ai instances and serverless endpoints
- RunPod: Scaled to 0 workers (endpoint preserved at zero cost)
- Docker images on Docker Hub: `bharathkumar192/hindi-nemotron-asr:v5-vram` (latest)

**What remains (essential code):**
- `nemotron/` — Model training configs, scripts, tokenizers
- `nemotron_asr/` — Modal deployment (core ASR pipeline logic, reusable)
- `nemotron-asr-frontend/` — Web UI for testing
- `benchmark_results/` — All benchmark data (JSON + CSV)
- `.env` — API keys for Vast, RunPod, HF, Docker Hub, Modal

**Next steps for production rewrite:**
1. Triton Inference Server for the NeMo RNNT model (proper batching, GPU utilization)
2. Vast.ai serverless with full PyWorker integration (`/route/` for direct WS access)
3. Production-grade WebSocket server (session management, reconnection, TLS)
4. Autoscaling config based on measured capacity (scale at 80, cap at 140 per A100-80GB)
5. Multi-worker stress test through Vast.ai `/route/` endpoint