Output:
Exit code: 0
Wall time: 253.1 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
Modes: http
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- HTTP MODE ---
  [HTTP] c=1: Starting 1 requests...
  [HTTP] c=1: OK=1 Err=0 FT_p50=3226ms FT_p99=3226ms Thru=2.8x WER=33.3% CER=18.1% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=10: Starting 10 requests...
  [HTTP] c=10: OK=10 Err=0 FT_p50=4638ms FT_p99=4644ms Thru=24.3x WER=17.6% CER=8.0% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=50: Starting 50 requests...
  [HTTP] c=50: OK=50 Err=0 FT_p50=7456ms FT_p99=13393ms Thru=43.5x WER=90.1% CER=88.2% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=100: Starting 100 requests...
  [HTTP] c=100: OK=100 Err=0 FT_p50=8606ms FT_p99=15291ms Thru=76.4x WER=87.0% CER=85.0% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=200: Starting 200 requests...
  [HTTP] c=200: OK=200 Err=0 FT_p50=15458ms FT_p99=26006ms Thru=83.3x WER=98.2% CER=97.8% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=300: Starting 300 requests...
  [HTTP] c=300: OK=300 Err=0 FT_p50=21960ms FT_p99=41000ms Thru=84.8x WER=97.0% CER=96.5% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=400: Starting 400 requests...
  [HTTP] c=400: OK=400 Err=0 FT_p50=25170ms FT_p99=47805ms Thru=95.7x WER=97.4% CER=97.0% VRAM=10.5/85GB
  Cooling down 5s...
  [HTTP] c=500: Starting 500 requests...
  [HTTP] c=500: OK=500 Err=0 FT_p50=23910ms FT_p99=48310ms Thru=115.4x WER=98.5% CER=98.3% VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
  http |     1 |    1 |    0 |   33.3 |   18.1 |    3226ms |    3226ms |     3226ms |    3229ms |    2.8x |     - |    10.5/85GB
  http |    10 |   10 |    0 |   17.6 |    8.0 |    4638ms |    4644ms |     4638ms |    4682ms |   24.3x |     - |    10.5/85GB
  http |    50 |   50 |    0 |   90.1 |   88.2 |    7456ms |   13393ms |     7456ms |   13462ms |   43.5x |     - |    10.5/85GB
  http |   100 |  100 |    0 |   87.0 |   85.0 |    8606ms |   15291ms |     8606ms |   15323ms |   76.4x |     - |    10.5/85GB
  http |   200 |  200 |    0 |   98.2 |   97.8 |   15458ms |   26006ms |    15458ms |   28086ms |   83.3x |     - |    10.5/85GB
  http |   300 |  300 |    0 |   97.0 |   96.5 |   21960ms |   41000ms |    21960ms |   41390ms |   84.8x |     - |    10.5/85GB
  http |   400 |  400 |    0 |   97.4 |   97.0 |   25170ms |   47805ms |    25170ms |   48892ms |   95.7x |     - |    10.5/85GB
  http |   500 |  500 |    0 |   98.5 |   98.3 |   23910ms |   48310ms |    23910ms |   50712ms |  115.4x |     - |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.json

--- QUALITY DEGRADATION CURVE ---
  HTTP baseline WER (c=1): 33.3%
    c=  10: WER=17.6% (delta=-15.7%)
    c=  50: WER=90.1% (delta=+56.8%) << DEGRADED
    c= 100: WER=87.0% (delta=+53.7%) << DEGRADED
    c= 200: WER=98.2% (delta=+64.9%) << DEGRADED
    c= 300: WER=97.0% (delta=+63.7%) << DEGRADED
    c= 400: WER=97.4% (delta=+64.1%) << DEGRADED
    c= 500: WER=98.5% (delta=+65.2%) << DEGRADED

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Running HTTP test**


HTTP on compile completed; latency stayed bounded but transcription quality collapsed at higher HTTP fan-in, which is important and likely a scheduler/backpressure artifact rather than model accuracy. I’m running the same HTTP sweep on TRT now to confirm if behavior matches.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode http --concurrency 1,10,50,100,200,300,400,500 --cooldown 5 --prefix realworld_trt_http


Output:
Exit code: 0
Wall time: 263.4 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: http
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA L40S
  VRAM: 10.4/48.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- HTTP MODE ---
  [HTTP] c=1: Starting 1 requests...
  [HTTP] c=1: OK=1 Err=0 FT_p50=2855ms FT_p99=2855ms Thru=3.2x WER=37.5% CER=21.6% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=10: Starting 10 requests...
  [HTTP] c=10: OK=10 Err=0 FT_p50=4075ms FT_p99=4090ms Thru=27.6x WER=18.3% CER=8.2% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=50: Starting 50 requests...
  [HTTP] c=50: OK=50 Err=0 FT_p50=5907ms FT_p99=8569ms Thru=67.8x WER=72.2% CER=67.8% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=100: Starting 100 requests...
  [HTTP] c=100: OK=100 Err=0 FT_p50=9197ms FT_p99=15532ms Thru=75.0x WER=84.9% CER=82.2% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=200: Starting 200 requests...
  [HTTP] c=200: OK=200 Err=0 FT_p50=16166ms FT_p99=27728ms Thru=83.9x WER=93.4% CER=92.3% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=300: Starting 300 requests...
  [HTTP] c=300: OK=300 Err=0 FT_p50=20556ms FT_p99=38931ms Thru=90.0x WER=95.8% CER=94.9% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=400: Starting 400 requests...
  [HTTP] c=400: OK=400 Err=0 FT_p50=28525ms FT_p99=55949ms Thru=83.1x WER=97.4% CER=96.9% VRAM=10.4/48GB
  Cooling down 5s...
  [HTTP] c=500: Starting 500 requests...
  [HTTP] c=500: OK=459 Err=41 FT_p50=31105ms FT_p99=59716ms Thru=87.7x WER=99.8% CER=99.8% VRAM=10.4/48GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
  http |     1 |    1 |    0 |   37.5 |   21.6 |    2855ms |    2855ms |     2855ms |    2857ms |    3.2x |     - |    10.4/48GB
  http |    10 |   10 |    0 |   18.3 |    8.2 |    4075ms |    4090ms |     4075ms |    4122ms |   27.6x |     - |    10.4/48GB
  http |    50 |   50 |    0 |   72.2 |   67.8 |    5907ms |    8569ms |     5907ms |    8625ms |   67.8x |     - |    10.4/48GB
  http |   100 |  100 |    0 |   84.9 |   82.2 |    9197ms |   15532ms |     9197ms |   15599ms |   75.0x |     - |    10.4/48GB
  http |   200 |  200 |    0 |   93.4 |   92.3 |   16166ms |   27728ms |    16166ms |   27906ms |   83.9x |     - |    10.4/48GB
  http |   300 |  300 |    0 |   95.8 |   94.9 |   20556ms |   38931ms |    20556ms |   39024ms |   90.0x |     - |    10.4/48GB
  http |   400 |  400 |    0 |   97.4 |   96.9 |   28525ms |   55949ms |    28525ms |   56327ms |   83.1x |     - |    10.4/48GB
  http |   500 |  459 |   41 |   99.8 |   99.8 |   31105ms |   59716ms |    31105ms |   61124ms |   87.7x |     - |    10.4/48GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.json

--- QUALITY DEGRADATION CURVE ---
  HTTP baseline WER (c=1): 37.5%
    c=  10: WER=18.3% (delta=-19.2%)
    c=  50: WER=72.2% (delta=+34.7%) << DEGRADED
    c= 100: WER=84.9% (delta=+47.4%) << DEGRADED
    c= 200: WER=93.4% (delta=+55.9%) << DEGRADED
    c= 300: WER=95.8% (delta=+58.3%) << DEGRADED
    c= 400: WER=97.4% (delta=+59.9%) << DEGRADED
    c= 500: WER=99.8% (delta=+62.3%) << DEGRADED

--- VRAM ---
Peak: 10.4 / 48 GB (21.7%)


Summary: auto


**Analyzing error sources and planning reruns**


I’ve completed WS (real-time + burst) and HTTP sweeps to 500 on both deployments. Before I finalize recommendations, I’m running a high-concurrency recheck (`300,400,500`) to confirm that the key tail-latency/quality behavior isn’t a one-off sample.


**Verifying real-time compilation**


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run --mode ws --concurrency 300,400,500 --cooldown 5 --prefix recheck_compile_ws_rt_hi


Output:
Exit code: 0
Wall time: 166.9 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=1501ms FT_p99=3408ms Thru=85.5x WER=33.5% CER=19.0%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1270ms FT_p99=3160ms Thru=82.5x WER=33.5% CER=19.0%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1326ms FT_p99=2756ms Thru=107.6x WER=33.6% CER=19.1%  Msgs=19.9 VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |   300 |  300 |    0 |   33.5 |   19.0 |    1501ms |    3408ms |    23529ms |   41079ms |   85.5x |  20.0 |    10.5/85GB
    ws |   400 |  400 |    0 |   33.5 |   19.0 |    1270ms |    3160ms |    29313ms |   56718ms |   82.5x |  19.9 |    10.5/85GB
    ws |   500 |  500 |    0 |   33.6 |   19.1 |    1326ms |    2756ms |    31148ms |   54370ms |  107.6x |  19.9 |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.csv
  /home/ubuntu/nemo_hindi/benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=300): 33.5%
    c= 400: WER=33.5% (delta=+0.0%)
    c= 500: WER=33.6% (delta=+0.1%)

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode ws --concurrency 300,400,500 --cooldown 5 --prefix recheck_trt_ws_rt_hi


Output:
Exit code: 0
Wall time: 236.3 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA L40S
  VRAM: 10.4/48.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=983ms FT_p99=1713ms Thru=61.4x WER=33.6% CER=19.0%  Msgs=20.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1022ms FT_p99=1859ms Thru=60.7x WER=33.7% CER=19.1%  Msgs=20.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1132ms FT_p99=6737ms Thru=67.2x WER=35.4% CER=20.9%  Msgs=19.6 VRAM=10.4/48GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |   300 |  300 |    0 |   33.6 |   19.0 |     983ms |    1713ms |    24628ms |   57139ms |   61.4x |  20.0 |    10.4/48GB
    ws |   400 |  400 |    0 |   33.7 |   19.1 |    1022ms |    1859ms |    33267ms |   77164ms |   60.7x |  20.0 |    10.4/48GB
    ws |   500 |  500 |    0 |   35.4 |   20.9 |    1132ms |    6737ms |    42808ms |   87123ms |   67.2x |  19.6 |    10.4/48GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.csv
  /home/ubuntu/nemo_hindi/benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=300): 33.6%
    c= 400: WER=33.7% (delta=+0.1%)
    c= 500: WER=35.4% (delta=+1.8%)

--- VRAM ---
Peak: 10.4 / 48 GB (21.7%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Analyzing benchmark script and endpoint behavior**


nl -ba nemotron_asr/nemotron_asr.py | sed -n '560,760p'


nl -ba nemotron_asr/nemotron_asr_trt.py | sed -n '1060,1260p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
   560	            """
   561	            Feed audio through streaming pipeline and collect full transcription.
   562	            Creates a virtual stream, feeds all audio, waits for final result.
   563	            """
   564	            self.start_inference_loop_if_needed()
   565
   566	            stream_id = self.pipeline.open_streaming_session()
   567	            transcription_queue = await self.register_client(stream_id)
   568
   569	            try:
   570	                # WHY: Feed audio in paced batches, not all-at-once burst.
   571	                # Original: dump everything instantly -> overwhelms batcher at high concurrency.
   572	                # Fix: feed ~10 chunks per yield, giving inference loop time to drain.
   573	                chunk_size_bytes = (self.cfg.streaming.att_context_size[1] + 1) * 1280 * 2
   574	                FEED_BATCH = 10  # chunks per yield cycle
   575	                chunk_count = 0
   576	                for i in range(0, len(pcm_bytes), chunk_size_bytes):
   577	                    chunk = pcm_bytes[i:i + chunk_size_bytes]
   578	                    self.pipeline.append_streaming_audio(stream_id, chunk)
   579	                    chunk_count += 1
   580	                    if chunk_count % FEED_BATCH == 0:
   581	                        if self._batch_ready_event is not None:
   582	                            self._batch_ready_event.set()
   583	                        await asyncio.sleep(0.01)  # 10ms yield — lets inference loop process
   584
   585	                # Final signal
   586	                if self._batch_ready_event is not None:
   587	                    self._batch_ready_event.set()
   588	                await asyncio.sleep(0.05)  # Let batcher pick up remaining frames
   589
   590	                # Mark stream ended
   591	                self.pipeline._streaming_request_generator.streams[stream_id].mark_end()
   592	                if self._batch_ready_event is not None:
   593	                    self._batch_ready_event.set()
   594
   595	                # Collect transcription. Each partial message has `current_step_transcript`
   596	                # (incremental delta). Each final message has `final_transcript` (accumulated).
   597	                # WHY: For HTTP batch, collect the LAST non-empty text from any message.
   598	                # Partial `text` fields are incremental — the pipeline accumulates internally
   599	                # and the last partial/final before stream drains has the fullest text.
   600	                # For reliable full text: track all finals + last partial separately.
   601	                final_segments = []
   602	                last_partial = ""
   603	                audio_dur = len(pcm_bytes) / (16000 * 2)
   604	                deadline = time.perf_counter() + max(30.0, audio_dur * 3)
   605	                while time.perf_counter() < deadline:
   606	                    try:
   607	                        output_dict = await asyncio.wait_for(
   608	                            transcription_queue.get(), timeout=2.0
   609	                        )
   610	                        if output_dict is None:
   611	                            break
   612	                        if output_dict.get('is_marker'):
   613	                            break
   614	                        json_str = output_dict.get('json_str', '{}')
   615	                        data = json.loads(json_str)
   616	                        t = data.get('text', '')
   617	                        seg = data.get('segment_text', '')
   618	                        if data.get('is_final') and seg:
   619	                            final_segments.append(seg)
   620	                        elif t and t.strip() != '<s>':
   621	                            last_partial = t
   622	                    except asyncio.TimeoutError:
   623	                        if stream_id not in self.pipeline._streaming_request_generator.streams:
   624	                            await asyncio.sleep(0.1)
   625	                            # Drain any remaining
   626	                            while True:
   627	                                try:
   628	                                    od = transcription_queue.get_nowait()
   629	                                    if od is None or od.get('is_marker'):
   630	                                        break
   631	                                    js = od.get('json_str', '{}')
   632	                                    d = json.loads(js)
   633	                                    seg = d.get('segment_text', '')
   634	                                    if d.get('is_final') and seg:
   635	                                        final_segments.append(seg)
   636	                                    elif d.get('text', '').strip() and d.get('text', '').strip() != '<s>':
   637	                                        last_partial = d['text']
   638	                                except asyncio.QueueEmpty:
   639	                                    break
   640	                            break
   641
   642	                # Prefer final segments if available, else last partial
   643	                if final_segments:
   644	                    return " ".join(final_segments).strip()
   645	                return last_partial.strip()
   646	            finally:
   647	                await self.unregister_client(stream_id)
   648
   649	        @self.web_app.post("/transcribe")
   650	        async def http_transcribe(request: FastAPIRequest):
   651	            """HTTP batch transcription. Accepts multipart file or JSON base64."""
   652	            t_start = time.perf_counter()
   653	            content_type = request.headers.get("content-type", "")
   654
   655	            if content_type.startswith("application/json"):
   656	                body = await request.json()
   657	                audio_b64 = body.get("audio_base64") or body.get("audio")
   658	                if not audio_b64:
   659	                    from fastapi.responses import JSONResponse
   660	                    return JSONResponse(status_code=400, content={"error": "audio_base64 required"})
   661	                audio_data = base64.b64decode(audio_b64)
   662	            else:
   663	                form = await request.form()
   664	                file_obj = form.get("file")
   665	                if file_obj is None:
   666	                    from fastapi.responses import JSONResponse
   667	                    return JSONResponse(status_code=400, content={"error": "file required"})
   668	                audio_data = await file_obj.read() if hasattr(file_obj, "read") else file_obj
   669
   670	            pcm_bytes = preprocess_audio(audio_data, target_sample_rate=16000)
   671	            duration = len(pcm_bytes) / (16000 * 2)
   672
   673	            t_infer = time.perf_counter()
   674	            text = await _transcribe_via_scheduler(pcm_bytes)
   675	            t_done = time.perf_counter()
   676
   677	            try:
   678	                vram_used = torch.cuda.memory_allocated() / 1e9
   679	                vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
   680	                gpu_name = torch.cuda.get_device_name(0)
   681	            except Exception:
   682	                vram_used = vram_total = 0
   683	                gpu_name = "unknown"
   684
   685	            return {
   686	                "text": text,
   687	                "duration": round(duration, 2),
   688	                "processing_time": round(t_done - t_infer, 3),
   689	                "total_time": round(t_done - t_start, 3),
   690	                "gpu": gpu_name,
   691	                "vram_gb": round(vram_used, 1),
   692	                "vram_total_gb": round(vram_total, 0),
   693	            }
   694
   695	        # Register WebSocket handler
   696	        @self.web_app.websocket("/ws")
   697	        async def run_with_websocket(ws: WebSocket):
   698	            """
   699	            Production WebSocket endpoint with backpressure.
   700	            Rejects connections when container is at capacity.
   701	            """
   702
   703	            # === BACKPRESSURE: Fast-fail when overloaded ===
   704	            # WHY: Better to reject immediately than silently degrade everyone.
   705	            # Client gets clear signal to retry / route to another container.
   706	            active = len(self.client_queues)
   707	            max_streams = self.cfg.streaming.max_active_streams
   708	            if active >= max_streams:
   709	                await ws.accept()
   710	                await ws.close(code=1013, reason=f"Overloaded: {active}/{max_streams}")
   711	                return
   712
   713	            await ws.accept()
   714
   715	            stream_id = None  # Track for cleanup
   716	            try:
   717	                # Start centralized inference loop if not already running
   718	                self.start_inference_loop_if_needed()
   719
   720	                # Open streaming session for this client
   721	                stream_id = self.pipeline.open_streaming_session()
   722	                print(f"✅ Opened stream_id={stream_id}")
   723
   724	                transcription_queue = await self.register_client(stream_id)
   725	                elapsed = time.perf_counter() - self.start_time
   726	                print(f"[+{elapsed:7.3f}s] Client {stream_id} connected ({len(self.client_queues)} total)")
   727	            except Exception as e:
   728	                print(f"❌ Error during connection setup: {e}")
   729	                import traceback
   730	                traceback.print_exc()
   731	                if stream_id is not None:
   732	                    await self.unregister_client(stream_id)
   733	                if ws and ws.application_state is WebSocketState.CONNECTED:
   734	                    await ws.close()
   735	                raise
   736
   737	            async def recv_loop(ws, stream_id, transcription_queue):
   738	                """
   739	                Receive audio chunks and append to stream buffer
   740	                """
   741
   742	                chunk_size = self.cfg.streaming.att_context_size[1] + 1
   743	                audio_buffer = b""  # Accumulate chunks here as bytes
   744	                num_buffer_samples = 0
   745
   746	                elapsed = time.perf_counter() - self.start_time
   747	                print(f"[+{elapsed:7.3f}s] Stream {stream_id}: Micro-batching enabled (batch_size={chunk_size})")
   748
   749	                try:
   750	                    while True:
   751	                        # EAGER MESSAGE DRAINING: Collect all available messages before processing
   752	                        # This reduces per-message overhead by batching WebSocket receives
   753	                        messages = []
   754
   755	                        # [1] First message (blocking) - always wait for at least one
   756	                        first_message = await ws.receive()
   757	                        messages.append(first_message)
   758
   759	                        # [2] Drain any additional messages with timeout (non-blocking)
   760	                        # This grabs messages that arrived during processing of previous batch


Output:
Exit code: 0
Wall time: 0 seconds
Output:
  1060	            Feed audio through streaming pipeline and collect full transcription.
  1061	            Creates a virtual stream, feeds all audio, waits for final result.
  1062	            """
  1063	            self.start_inference_loop_if_needed()
  1064
  1065	            stream_id = self.pipeline.open_streaming_session()
  1066	            transcription_queue = await self.register_client(stream_id)
  1067
  1068	            try:
  1069	                # WHY: Feed audio in paced batches, not all-at-once burst.
  1070	                # Original: dump everything instantly -> overwhelms batcher at high concurrency.
  1071	                # Fix: feed ~10 chunks per yield, giving inference loop time to drain.
  1072	                chunk_size_bytes = (self.cfg.streaming.att_context_size[1] + 1) * 1280 * 2
  1073	                FEED_BATCH = 10  # chunks per yield cycle
  1074	                chunk_count = 0
  1075	                for i in range(0, len(pcm_bytes), chunk_size_bytes):
  1076	                    chunk = pcm_bytes[i:i + chunk_size_bytes]
  1077	                    self.pipeline.append_streaming_audio(stream_id, chunk)
  1078	                    chunk_count += 1
  1079	                    if chunk_count % FEED_BATCH == 0:
  1080	                        if self._batch_ready_event is not None:
  1081	                            self._batch_ready_event.set()
  1082	                        await asyncio.sleep(0.01)  # 10ms yield — lets inference loop process
  1083
  1084	                # Final signal
  1085	                if self._batch_ready_event is not None:
  1086	                    self._batch_ready_event.set()
  1087	                await asyncio.sleep(0.05)  # Let batcher pick up remaining frames
  1088
  1089	                # Mark stream ended
  1090	                self.pipeline._streaming_request_generator.streams[stream_id].mark_end()
  1091	                if self._batch_ready_event is not None:
  1092	                    self._batch_ready_event.set()
  1093
  1094	                # Collect transcription. Each partial message has `current_step_transcript`
  1095	                # (incremental delta). Each final message has `final_transcript` (accumulated).
  1096	                # WHY: For HTTP batch, collect the LAST non-empty text from any message.
  1097	                # Partial `text` fields are incremental — the pipeline accumulates internally
  1098	                # and the last partial/final before stream drains has the fullest text.
  1099	                # For reliable full text: track all finals + last partial separately.
  1100	                final_segments = []
  1101	                last_partial = ""
  1102	                audio_dur = len(pcm_bytes) / (16000 * 2)
  1103	                deadline = time.perf_counter() + max(30.0, audio_dur * 3)
  1104	                while time.perf_counter() < deadline:
  1105	                    try:
  1106	                        output_dict = await asyncio.wait_for(
  1107	                            transcription_queue.get(), timeout=2.0
  1108	                        )
  1109	                        if output_dict is None:
  1110	                            break
  1111	                        if output_dict.get('is_marker'):
  1112	                            break
  1113	                        json_str = output_dict.get('json_str', '{}')
  1114	                        data = json.loads(json_str)
  1115	                        t = data.get('text', '')
  1116	                        seg = data.get('segment_text', '')
  1117	                        if data.get('is_final') and seg:
  1118	                            final_segments.append(seg)
  1119	                        elif t and t.strip() != '<s>':
  1120	                            last_partial = t
  1121	                    except asyncio.TimeoutError:
  1122	                        if stream_id not in self.pipeline._streaming_request_generator.streams:
  1123	                            await asyncio.sleep(0.1)
  1124	                            # Drain any remaining
  1125	                            while True:
  1126	                                try:
  1127	                                    od = transcription_queue.get_nowait()
  1128	                                    if od is None or od.get('is_marker'):
  1129	                                        break
  1130	                                    js = od.get('json_str', '{}')
  1131	                                    d = json.loads(js)
  1132	                                    seg = d.get('segment_text', '')
  1133	                                    if d.get('is_final') and seg:
  1134	                                        final_segments.append(seg)
  1135	                                    elif d.get('text', '').strip() and d.get('text', '').strip() != '<s>':
  1136	                                        last_partial = d['text']
  1137	                                except asyncio.QueueEmpty:
  1138	                                    break
  1139	                            break
  1140
  1141	                # Prefer final segments if available, else last partial
  1142	                if final_segments:
  1143	                    return " ".join(final_segments).strip()
  1144	                return last_partial.strip()
  1145	            finally:
  1146	                await self.unregister_client(stream_id)
  1147
  1148	        @self.web_app.post("/transcribe")
  1149	        async def http_transcribe(request: FastAPIRequest):
  1150	            """HTTP batch transcription. Accepts multipart file or JSON base64."""
  1151	            t_start = time.perf_counter()
  1152	            content_type = request.headers.get("content-type", "")
  1153
  1154	            if content_type.startswith("application/json"):
  1155	                body = await request.json()
  1156	                audio_b64 = body.get("audio_base64") or body.get("audio")
  1157	                if not audio_b64:
  1158	                    from fastapi.responses import JSONResponse
  1159	                    return JSONResponse(status_code=400, content={"error": "audio_base64 required"})
  1160	                audio_data = base64.b64decode(audio_b64)
  1161	            else:
  1162	                form = await request.form()
  1163	                file_obj = form.get("file")
  1164	                if file_obj is None:
  1165	                    from fastapi.responses import JSONResponse
  1166	                    return JSONResponse(status_code=400, content={"error": "file required"})
  1167	                audio_data = await file_obj.read() if hasattr(file_obj, "read") else file_obj
  1168
  1169	            pcm_bytes = preprocess_audio(audio_data, target_sample_rate=16000)
  1170	            duration = len(pcm_bytes) / (16000 * 2)
  1171
  1172	            t_infer = time.perf_counter()
  1173	            text = await _transcribe_via_scheduler(pcm_bytes)
  1174	            t_done = time.perf_counter()
  1175
  1176	            try:
  1177	                vram_used = torch.cuda.memory_allocated() / 1e9
  1178	                vram_total = torch.cuda.get_device_properties(0).total_memory / 1e9
  1179	                gpu_name = torch.cuda.get_device_name(0)
  1180	            except Exception:
  1181	                vram_used = vram_total = 0
  1182	                gpu_name = "unknown"
  1183
  1184	            return {
  1185	                "text": text,
  1186	                "duration": round(duration, 2),
  1187	                "processing_time": round(t_done - t_infer, 3),
  1188	                "total_time": round(t_done - t_start, 3),
  1189	                "gpu": gpu_name,
  1190	                "vram_gb": round(vram_used, 1),
  1191	                "vram_total_gb": round(vram_total, 0),
  1192	            }
  1193
  1194	        # Register WebSocket handler
  1195	        @self.web_app.websocket("/ws")
  1196	        async def run_with_websocket(ws: WebSocket):
  1197	            """
  1198	            Production WebSocket endpoint with backpressure.
  1199	            Rejects connections when container is at capacity.
  1200	            """
  1201
  1202	            # === BACKPRESSURE: Fast-fail when overloaded ===
  1203	            # WHY: Better to reject immediately than silently degrade everyone.
  1204	            # Client gets clear signal to retry / route to another container.
  1205	            active = len(self.client_queues)
  1206	            max_streams = self.cfg.streaming.max_active_streams
  1207	            if active >= max_streams:
  1208	                await ws.accept()
  1209	                await ws.close(code=1013, reason=f"Overloaded: {active}/{max_streams}")
  1210	                return
  1211
  1212	            await ws.accept()
  1213
  1214	            stream_id = None  # Track for cleanup
  1215	            try:
  1216	                # Start centralized inference loop if not already running
  1217	                self.start_inference_loop_if_needed()
  1218
  1219	                # Open streaming session for this client
  1220	                stream_id = self.pipeline.open_streaming_session()
  1221	                print(f"✅ Opened stream_id={stream_id}")
  1222
  1223	                transcription_queue = await self.register_client(stream_id)
  1224	                elapsed = time.perf_counter() - self.start_time
  1225	                print(f"[+{elapsed:7.3f}s] Client {stream_id} connected ({len(self.client_queues)} total)")
  1226	            except Exception as e:
  1227	                print(f"❌ Error during connection setup: {e}")
  1228	                import traceback
  1229	                traceback.print_exc()
  1230	                if stream_id is not None:
  1231	                    await self.unregister_client(stream_id)
  1232	                if ws and ws.application_state is WebSocketState.CONNECTED:
  1233	                    await ws.close()
  1234	                raise
  1235
  1236	            async def recv_loop(ws, stream_id, transcription_queue):
  1237	                """
  1238	                Receive audio chunks and append to stream buffer
  1239	                """
  1240
  1241	                chunk_size = self.cfg.streaming.att_context_size[1] + 1
  1242	                audio_buffer = b""  # Accumulate chunks here as bytes
  1243	                num_buffer_samples = 0
  1244
  1245	                elapsed = time.perf_counter() - self.start_time
  1246	                print(f"[+{elapsed:7.3f}s] Stream {stream_id}: Micro-batching enabled (batch_size={chunk_size})")
  1247
  1248	                try:
  1249	                    while True:
  1250	                        # EAGER MESSAGE DRAINING: Collect all available messages before processing
  1251	                        # This reduces per-message overhead by batching WebSocket receives
  1252	                        messages = []
  1253
  1254	                        # [1] First message (blocking) - always wait for at least one
  1255	                        first_message = await ws.receive()
  1256	                        messages.append(first_message)
  1257
  1258	                        # [2] Drain any additional messages with timeout (non-blocking)
  1259	                        # This grabs messages that arrived during processing of previous batch
  1260	                        DRAIN_TIMEOUT_MS = 1  # 1ms timeout


**Planning aggregated benchmark report**


ls -1t benchmark_results/realworld_* benchmark_results/recheck_* | head -n 40


python - <<'PY'
import json, glob, os
paths=sorted(glob.glob('benchmark_results/realworld_*_*.json'))
print('\n'.join(paths))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.csv
benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.json
benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.csv
benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.json
benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.csv
benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.json
benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.json
benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.csv
benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.csv
benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.json
benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.json
benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.csv
benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.json
benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.csv
benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.csv
benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.json


Output:
Exit code: 127
Wall time: 0 seconds
Output:
/bin/bash: line 1: python: command not found


**Choosing python3 environment**


python3 - <<'PY'
import glob
for p in sorted(glob.glob('benchmark_results/realworld_*_*.json')):
    print(p)
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.json
benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.json
benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.json
benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.json
benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.json
benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.json


**Planning aggregation script**


I have all fresh result artifacts now. I’m generating a consolidated comparison (real-time WS, burst WS, HTTP, plus high-concurrency recheck) so we can separate true model/runtime behavior from benchmark-mode artifacts.


python3 - <<'PY'
import json
from pathlib import Path
from statistics import mean

files = {
    'compile_ws_rt': Path('benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.json'),
    'trt_ws_rt': Path('benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.json'),
    'compile_ws_burst': Path('benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.json'),
    'trt_ws_burst': Path('benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.json'),
    'compile_http': Path('benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.json'),
    'trt_http': Path('benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.json'),
    'recheck_compile_hi': Path('benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.json'),
    'recheck_trt_hi': Path('benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.json'),
}

def load(p):
    return {r['concurrency']: r for r in json.loads(Path(p).read_text())}

data = {k: load(v) for k,v in files.items()}

levels = [1,10,50,100,200,300,400,500]

def fmt(v):
    if isinstance(v, float):
        return f"{v:.1f}"
    return str(v)

print('=== WS REALTIME ===')
print('conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_err, trt_err')
for c in levels:
    a=data['compile_ws_rt'][c]; b=data['trt_ws_rt'][c]
    print(', '.join(map(fmt,[c,a['ft_p50_ms'],b['ft_p50_ms'],a['ft_p99_ms'],b['ft_p99_ms'],a['wer_pct'],b['wer_pct'],a['throughput_x'],b['throughput_x'],a['errors'],b['errors']])))

print('\n=== WS BURST ===')
print('conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_err, trt_err')
for c in levels:
    a=data['compile_ws_burst'][c]; b=data['trt_ws_burst'][c]
    print(', '.join(map(fmt,[c,a['ft_p50_ms'],b['ft_p50_ms'],a['ft_p99_ms'],b['ft_p99_ms'],a['wer_pct'],b['wer_pct'],a['throughput_x'],b['throughput_x'],a['errors'],b['errors']])))

print('\n=== HTTP ===')
print('conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_ok, trt_ok, compile_err, trt_err')
for c in levels:
    a=data['compile_http'][c]; b=data['trt_http'][c]
    print(', '.join(map(fmt,[c,a['ft_p50_ms'],b['ft_p50_ms'],a['ft_p99_ms'],b['ft_p99_ms'],a['wer_pct'],b['wer_pct'],a['throughput_x'],b['throughput_x'],a['ok'],b['ok'],a['errors'],b['errors']])))

print('\n=== RECHECK HI (WS RT) ===')
for c in [300,400,500]:
    a=data['recheck_compile_hi'][c]; b=data['recheck_trt_hi'][c]
    print(f"c={c} compile(ft50={a['ft_p50_ms']}, ft99={a['ft_p99_ms']}, wer={a['wer_pct']}, thru={a['throughput_x']}) | trt(ft50={b['ft_p50_ms']}, ft99={b['ft_p99_ms']}, wer={b['wer_pct']}, thru={b['throughput_x']})")
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
=== WS REALTIME ===
conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_err, trt_err
1, 644.4, 676.1, 644.4, 676.1, 33.3, 37.5, 0.9, 0.9, 0, 0
10, 1386.3, 990.3, 1467.0, 1015.7, 20.7, 19.9, 6.5, 6.4, 0, 0
50, 982.5, 600.6, 1101.8, 649.2, 33.4, 33.7, 23.8, 23.5, 0, 0
100, 845.3, 780.5, 985.7, 2351.1, 33.1, 34.3, 46.8, 40.8, 0, 0
200, 954.2, 866.9, 2064.9, 2760.8, 33.4, 34.5, 89.5, 54.0, 0, 0
300, 1130.2, 1216.0, 2763.0, 6671.0, 33.6, 35.8, 85.2, 61.3, 0, 0
400, 1290.9, 1072.7, 2482.9, 3017.8, 33.4, 33.7, 83.8, 59.5, 0, 0
500, 1550.2, 1134.8, 2833.5, 3415.1, 33.5, 34.7, 107.7, 62.2, 0, 0

=== WS BURST ===
conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_err, trt_err
1, 171.7, 138.4, 171.7, 138.4, 79.2, 79.2, 8.7, 11.5, 0, 0
10, 447.6, 255.6, 620.7, 286.5, 28.1, 20.7, 16.9, 29.5, 0, 0
50, 470.3, 432.0, 931.0, 1563.6, 33.3, 33.7, 50.7, 56.2, 0, 0
100, 693.8, 624.4, 1513.8, 1362.7, 33.8, 34.0, 57.1, 65.3, 0, 0
200, 890.8, 802.6, 2243.8, 1474.4, 33.2, 33.7, 75.7, 73.2, 0, 0
300, 1270.5, 973.5, 2435.9, 1798.5, 33.5, 33.7, 83.7, 73.5, 0, 0
400, 1386.1, 952.0, 2755.2, 4096.4, 33.6, 34.8, 91.2, 75.0, 0, 0
500, 1344.0, 1058.8, 2753.8, 5307.9, 33.7, 37.3, 107.8, 75.7, 0, 0

=== HTTP ===
conc, compile_ft50, trt_ft50, compile_ft99, trt_ft99, compile_wer, trt_wer, compile_thru, trt_thru, compile_ok, trt_ok, compile_err, trt_err
1, 3225.8, 2855.2, 3225.8, 2855.2, 33.3, 37.5, 2.8, 3.2, 1, 1, 0, 0
10, 4637.7, 4074.7, 4644.4, 4090.2, 17.6, 18.3, 24.3, 27.6, 10, 10, 0, 0
50, 7455.9, 5906.6, 13392.6, 8569.1, 90.1, 72.2, 43.5, 67.8, 50, 50, 0, 0
100, 8606.0, 9197.1, 15290.7, 15531.6, 87.0, 84.9, 76.4, 75.0, 100, 100, 0, 0
200, 15457.9, 16166.0, 26005.6, 27727.6, 98.2, 93.4, 83.3, 83.9, 200, 200, 0, 0
300, 21960.3, 20555.9, 40999.6, 38930.8, 97.0, 95.8, 84.8, 90.0, 300, 300, 0, 0
400, 25169.9, 28524.9, 47805.1, 55949.2, 97.4, 97.4, 95.7, 83.1, 400, 400, 0, 0
500, 23909.7, 31105.3, 48310.4, 59715.7, 98.5, 99.8, 115.4, 87.7, 500, 459, 0, 41

=== RECHECK HI (WS RT) ===
c=300 compile(ft50=1501.1, ft99=3408.1, wer=33.5, thru=85.5) | trt(ft50=983.0, ft99=1713.2, wer=33.6, thru=61.4)
c=400 compile(ft50=1269.9, ft99=3160.0, wer=33.5, thru=82.5) | trt(ft50=1022.5, ft99=1858.8, wer=33.7, thru=60.7)
c=500 compile(ft50=1325.5, ft99=2755.7, wer=33.6, thru=107.6) | trt(ft50=1131.7, ft99=6736.8, wer=35.4, thru=67.2)


Summary: auto


**Planning HTTP error investigation**


source venv/bin/activate && python - <<'PY'
import asyncio, aiohttp, collections, time
from nemotron_asr.bench_modal import load_audio_samples, http_request

BASE='https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run'
CONC=500
samples=load_audio_samples(50)

async def run():
    connector=aiohttp.TCPConnector(limit=CONC+20)
    async with aiohttp.ClientSession(connector=connector) as sess:
        tasks=[http_request(BASE, samples[i%len(samples)], i, sess) for i in range(CONC)]
        t0=time.perf_counter()
        res=await asyncio.gather(*tasks)
        t1=time.perf_counter()
    ok=[r for r in res if r.ok]
    err=[r for r in res if not r.ok]
    ctr=collections.Counter(r.error.split(':')[0] if r.error else 'unknown' for r in err)
    print('wall_s', round(t1-t0,2), 'ok', len(ok), 'err', len(err))
    print('error_types', ctr.most_common(10))
    if err:
        print('sample_errors:')
        for e in err[:10]:
            print('-', e.error[:200])

asyncio.run(run())
PY


Output:
Exit code: 0
Wall time: 42.7 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)
wall_s 42.29 ok 499 err 1
error_types [('[Errno 104] Connection reset by peer', 1)]
sample_errors:
- [Errno 104] Connection reset by peer


**Validating transient error reduction**


I’m drilling into the TRT HTTP `c=500` failures to separate persistent service limits from transient network/container events. I just ran an independent 500-request replay and saw only a single connection reset, so I’m doing one more confirmation pass.


source venv/bin/activate && python - <<'PY'
import asyncio, aiohttp, collections, time
from nemotron_asr.bench_modal import load_audio_samples, http_request

BASE='https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run'
CONC=500
samples=load_audio_samples(50)

async def run():
    connector=aiohttp.TCPConnector(limit=CONC+20)
    async with aiohttp.ClientSession(connector=connector) as sess:
        tasks=[http_request(BASE, samples[i%len(samples)], i, sess) for i in range(CONC)]
        t0=time.perf_counter()
        res=await asyncio.gather(*tasks)
        t1=time.perf_counter()
    ok=[r for r in res if r.ok]
    err=[r for r in res if not r.ok]
    ctr=collections.Counter(r.error.split(':')[0] if r.error else 'unknown' for r in err)
    print('wall_s', round(t1-t0,2), 'ok', len(ok), 'err', len(err))
    print('error_types', ctr.most_common(10))

asyncio.run(run())
PY


Output:
Exit code: 0
Wall time: 62 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)
wall_s 61.64 ok 489 err 11
error_types [('timeout', 11)]


**Analyzing benchmark timeouts and safety zones**


The c=500 TRT HTTP failures are not deterministic; follow-up replays show mixed `timeout` and occasional connection reset, which points to saturation/timeout behavior rather than a hard crash. I’ll do a final health check on both services and then deliver a consolidated production-readiness assessment.


curl -sS https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run/health | jq .


curl -sS https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run/health | jq .


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "status": "healthy",
  "active_streams": 0,
  "max_streams": 400,
  "gpu": "NVIDIA H100 80GB HBM3",
  "vram_gb": 10.5,
  "vram_total_gb": 85,
  "inference_running": true,
  "uptime_s": 22898
}


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "status": "healthy",
  "active_streams": 0,
  "max_streams": 400,
  "gpu": "NVIDIA L40S",
  "vram_gb": 10.4,
  "vram_total_gb": 48,
  "encoder": "tensorrt",
  "inference_running": true,
  "uptime_s": 5779
}


**Planning plan status update**


Audit benchmark harness correctness against server output semantics and metric calculations

Identify active Modal endpoints for torch.compile and TensorRT deployments and confirm health/encoder type

Run controlled WS real-time benchmark sweep (1..500 concurrency) on both deployments

Run additional production-like stress scenarios (burst WS and HTTP) for both deployments

Aggregate results, compare latency/throughput/quality/error behavior, and provide production recommendations

Explanation: Completed fresh benchmark sweeps to 500 for compile and TRT across realtime WS, burst WS, and HTTP; also ran high-concurrency rechecks and replayed TRT HTTP c=500 failure mode for error characterization.


Output: Plan updated


**Structuring thorough benchmark response**


I completed a fresh end-to-end benchmark on **February 11, 2026** for both deployed Modal apps:

- `https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run` (torch.compile path, H100)
- `https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run` (TensorRT path, L40S, `/health` reports `encoder: "tensorrt"`)

**1) Benchmark script audit (`nemotron_asr/bench_modal.py`)**
- It is genuine for your WS production path:
  - Uses real FLEURS Hindi audio + references (`nemotron_asr/bench_modal.py:109`).
  - Computes WER/CER with `jiwer` (`nemotron_asr/bench_modal.py:86`).
  - Measures FT p50/p99, total p50/p99, throughput, errors.
- Important caveats:
  - `c=1` WER is single-sample sensitive (always sample 0).
  - `--no-realtime` is stress-only (not customer-realistic for live voice).
  - HTTP mode is valid as tested, but endpoint behavior makes high-concurrency quality collapse (see item 4).

**2) Implementation audit**
- torch.compile app: encoder compiled in `nemotron_asr/nemotron_asr.py:396`; deployed on H100 (`nemotron_asr/nemotron_asr.py:303`).
- TRT app: wrapper override exists at `nemotron_asr/nemotron_asr_trt.py:183`, TRT execute path at `nemotron_asr/nemotron_asr_trt.py:271`, deployed on L40S (`nemotron_asr/nemotron_asr_trt.py:736`).

**3) Benchmarks I ran (all to 500 concurrency)**
- WS real-time (`1,10,50,100,200,300,400,500`)
- WS burst (`--no-realtime`, same levels)
- HTTP batch (`/transcribe`, same levels)
- High-concurrency WS recheck (`300,400,500`) for stability

**4) Key results**
- WS real-time:
  - Both had **0 errors through c=500**.
  - Compile(H100): WER stayed ~33–34% through c=500; c=500 FT p50/p99 `1550/2834 ms`, throughput `107.7x`.
  - TRT(L40S): WER mostly ~34%, slight rise at top end in recheck (c=500 ~35.4%); c=500 FT p50/p99 `1135/3415 ms` (recheck p99 spike to `6737 ms`), throughput `62.2x` (recheck `67.2x`).
- WS burst:
  - Both stayed operational with 0 errors through c=500.
  - c=500 compile: WER `33.7%`, FT p99 `2754 ms`, throughput `107.8x`.
  - c=500 TRT: WER `37.3%`, FT p99 `5308 ms`, throughput `75.7x`.
- HTTP:
  - **Quality degrades hard from c>=50** on both apps.
  - Compile c=500: WER `98.5%` (no request failures).
  - TRT c=500: WER `99.8%`, plus failures in one sweep (`41` errors), with replay variability (1 and 11 errors in two independent reruns).
  - This is consistent with endpoint logic returning partial text under load (`deadline = max(30s, audio*3)` and fallback to `last_partial` in `nemotron_asr/nemotron_asr.py:604` and `nemotron_asr/nemotron_asr_trt.py:1103`).

**5) Situation summary**
- Your **streaming WS path is production-healthy** up to 500 stress concurrency on both apps.
- Current **compile vs TRT comparison is not apples-to-apples**, because hardware differs (H100 vs L40S). You can compare deployment behavior, not pure encoder-runtime gain.
- The large degradation you see in HTTP is mainly service-path behavior under fan-in, not a WS model-quality collapse.

**6) Artifacts**
- `benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.json`
- `benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.json`
- `benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.json`
- `benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.json`
- `benchmark_results/realworld_compile_http_NVIDIA_H100_80GB_HBM3_20260211_230701.json`
- `benchmark_results/realworld_trt_http_NVIDIA_L40S_20260211_231132.json`
- Rechecks: `benchmark_results/recheck_compile_ws_rt_hi_NVIDIA_H100_80GB_HBM3_20260211_231442.json`, `benchmark_results/recheck_trt_ws_rt_hi_NVIDIA_L40S_20260211_231843.json`

1. If you want strict TRT-vs-compile runtime truth, I can deploy both variants on the same GPU class (both L40S or both H100) and rerun the exact matrix.
2. I can patch `/transcribe` to fail-fast on truncation (instead of returning partial text) so HTTP quality metrics become production-meaningful under load.


just deploy TRT version hindi-nemotron-asr-1 on H100 and do a one last bench and report scores.


**Planning deployment and benchmark**


I’ll switch the TRT app config to `H100`, deploy `hindi-nemotron-asr-1`, then run one final WS benchmark sweep (`1..500`) and report the full scores. I’m editing only the TRT deployment config first.


rg -n "gpu=|L40S|H100" nemotron_asr/nemotron_asr_trt.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
727:    # WHY: L40S is the production target GPU ($1.60/hr, 48GB VRAM).
728:    # TRT gives 2.84x encoder speedup => ~500-600 concurrent streams per L40S.
729:    gpu="L40S",


Summary: auto


**Applying targeted comment patch**


I found the GPU pin in `nemotron_asr/nemotron_asr_trt.py`; I’m patching it to `H100` so this deploy is explicitly TRT-on-H100.


Input:
*** Begin Patch
*** Update File: /home/ubuntu/nemo_hindi/nemotron_asr/nemotron_asr_trt.py
@@
-    # WHY: L40S is the production target GPU ($1.60/hr, 48GB VRAM).
-    # TRT gives 2.84x encoder speedup => ~500-600 concurrent streams per L40S.
-    gpu="L40S",
+    # Benchmark pin: run TRT variant on H100 for apples-to-apples capacity checks.
+    gpu="H100",
*** End Patch


Output: {"output":"Success. Updated the following files:\nM /home/ubuntu/nemo_hindi/nemotron_asr/nemotron_asr_trt.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Checking syntax correctness**


python3 -m py_compile nemotron_asr/nemotron_asr_trt.py


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:


GPU pin is updated and syntax is clean. Next I’m deploying `nemotron_asr/nemotron_asr_trt.py` to Modal (`hindi-nemotron-asr-1`), then I’ll verify `/health` shows H100 before benchmarking.


source venv/bin/activate && modal deploy nemotron_asr/nemotron_asr_trt.py


Output:
Exit code: 0
Wall time: 35.3 seconds
Output:
✅ Hindi ASR TRT service (hindi-nemotron-asr-1) loaded!
Building image im-gjG0mmo2nzoemhblnxAth8

=> Step 0: FROM base

=> Step 1: COPY . /
Saving image...
Image saved, took 1.16s

Built image im-gjG0mmo2nzoemhblnxAth8 in 14.51s

Building image im-TSjaSjcXe0Qk7JfuLae0IJ

=> Step 0: FROM base

=> Step 1: RUN ls -la /root/asr_lib/ && cat /root/asr_lib/vad.py
total 283
drwxr-xr-x 3 root root  4096 Feb 11 23:55 .
drwxr-xr-x 3 root root    29 Feb 11 23:55 ..
-rw-rw-r-- 1 root root     0 Feb 11 23:55 __init__.py
drwxr-xr-x 2 root root  4096 Feb 11 23:55 __pycache__
-rw-rw-r-- 1 root root 11800 Feb 11 23:55 asr_utils.py
-rw-rw-r-- 1 root root 23702 Feb 11 23:55 bench_modal.py
-rw-rw-r-- 1 root root 24601 Feb 11 23:55 multi_stream_utils.py
-rw-rw-r-- 1 root root 73491 Feb 11 23:55 nemotron_asr.py
-rw-rw-r-- 1 root root 95659 Feb 11 23:55 nemotron_asr_trt.py
-rw-rw-r-- 1 root root 47741 Feb 11 23:55 pipelines.py
-rw-rw-r-- 1 root root  1642 Feb 11 23:55 vad.py
"""
CPU-based Voice Activity Detection using Silero VAD.
WHY: Filters silence BEFORE GPU pipeline. 3x GPU capacity for free.
"""
import numpy as np
import torch

class SileroVAD:
    def __init__(self, threshold=0.5, sample_rate=16000):
        self.threshold = threshold
        self.sample_rate = sample_rate
        self._model = None
        self._silence_counts = {}
        self.silence_threshold_frames = 3

    def _ensure_loaded(self):
        if self._model is None:
            self._model, _ = torch.hub.load(
                repo_or_dir="snakers4/silero-vad",
                model="silero_vad", force_reload=False, onnx=False,
            )
            self._model.eval()

    def is_speech(self, audio_bytes, stream_id=0):
        self._ensure_loaded()
        try:
            audio_np = np.frombuffer(audio_bytes, dtype=np.int16).astype(np.float32) / 32768.0
            audio_tensor = torch.from_numpy(audio_np)
            window_size = 512
            if len(audio_tensor) < window_size:
                return True
            chunk = audio_tensor[-window_size:]
            prob = self._model(chunk, self.sample_rate).item()
            is_speech = prob >= self.threshold
            if is_speech:
                self._silence_counts[stream_id] = 0
                return True
            else:
                count = self._silence_counts.get(stream_id, 0) + 1
                self._silence_counts[stream_id] = count
                return count < self.silence_threshold_frames
        except Exception:
            return True

    def reset_stream(self, stream_id):
        self._silence_counts.pop(stream_id, None)
Saving image...
Image saved, took 1.22s

Built image im-TSjaSjcXe0Qk7JfuLae0IJ in 8.51s

✓ Created objects.
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron_asr/nemotron_asr_trt.py
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron-asr-frontend
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron_asr
├── 🔨 Created function WebServer.*.
├── 🔨 Created web endpoint for WebServer.web =>
│   https://mayaresearch--hindi-nemotron-asr-1-webserver-web.modal.run
├── 🔨 Created function NemotronASR.*.
└── 🔨 Created web endpoint for NemotronASR.webapp =>
    https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
✓ App deployed in 26.495s! 🎉

View Deployment:
https://modal.com/apps/mayaresearch/main/deployed/hindi-nemotron-asr-1
[NeMo W 2026-02-11 23:54:56 <frozen importlib:241] Megatron num_microbatches_calculator not found, using Apex version.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.


**Planning health check with cold start**


Deployment succeeded. I’m now warming the new endpoint and polling `/health` until it reports `gpu: NVIDIA H100 ...` and `encoder: tensorrt`, then I’ll run the final benchmark sweep.


for i in $(seq 1 40); do echo "--- poll $i ---"; curl -sS https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run/health || true; echo; sleep 10; done


Output:
Exit code: 0
Wall time: 411 seconds
Output:
--- poll 1 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7813.0}
--- poll 2 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7823.0}
--- poll 3 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7834.0}
--- poll 4 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7844.0}
--- poll 5 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7854.0}
--- poll 6 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7865.0}
--- poll 7 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7875.0}
--- poll 8 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7885.0}
--- poll 9 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7895.0}
--- poll 10 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7906.0}
--- poll 11 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7916.0}
--- poll 12 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7926.0}
--- poll 13 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7936.0}
--- poll 14 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7947.0}
--- poll 15 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7957.0}
--- poll 16 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7967.0}
--- poll 17 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7978.0}
--- poll 18 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7988.0}
--- poll 19 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":7998.0}
--- poll 20 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8008.0}
--- poll 21 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8019.0}
--- poll 22 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8029.0}
--- poll 23 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8039.0}
--- poll 24 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8050.0}
--- poll 25 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8060.0}
--- poll 26 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8070.0}
--- poll 27 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8080.0}
--- poll 28 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8091.0}
--- poll 29 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8101.0}
--- poll 30 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8111.0}
--- poll 31 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8121.0}
--- poll 32 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8132.0}
--- poll 33 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8142.0}
--- poll 34 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8152.0}
--- poll 35 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8162.0}
--- poll 36 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8173.0}
--- poll 37 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8183.0}
--- poll 38 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8193.0}
--- poll 39 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8203.0}
--- poll 40 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA L40S","vram_gb":10.4,"vram_total_gb":48.0,"encoder":"tensorrt","inference_running":true,"uptime_s":8214.0}


**Investigating lingering old deployment**


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Version ┃ Time deployed        ┃ Client ┃ Deployed by ┃ Commit ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━┩
│ v3      │ 2026-02-11 23:55 UTC │ 1.3.2  │ bharath     │        │
│ v2      │ 2026-02-11 21:37 UTC │ 1.3.2  │ bharath     │        │
│ v1      │ 2026-02-11 21:29 UTC │ 1.3.2  │ bharath     │        │
└─────────┴──────────────────────┴────────┴─────────────┴────────┘


**Forcing app restart to update container**


The new version (`v3`) is deployed, but traffic is still landing on an old L40S container. I’m force-cycling `hindi-nemotron-asr-1` (stop + redeploy) so the app cold-starts on H100 and rebuilds the TRT engine for that GPU.


Output:
Exit code: 0
Wall time: 0.8 seconds
Output:


**Redeploying updated file**


Output:
Exit code: 0
Wall time: 10.1 seconds
Output:
✅ Hindi ASR TRT service (hindi-nemotron-asr-1) loaded!
✓ Created objects.
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron_asr/nemotron_asr_trt.py
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron-asr-frontend
├── 🔨 Created mount /home/ubuntu/nemo_hindi/nemotron_asr
├── 🔨 Created function WebServer.*.
├── 🔨 Created web endpoint for WebServer.web =>
│   https://mayaresearch--hindi-nemotron-asr-1-webserver-web.modal.run
├── 🔨 Created function NemotronASR.*.
└── 🔨 Created web endpoint for NemotronASR.webapp =>
    https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
✓ App deployed in 1.450s! 🎉

View Deployment:
https://modal.com/apps/mayaresearch/main/deployed/hindi-nemotron-asr-1
[NeMo W 2026-02-12 00:02:53 <frozen importlib:241] Megatron num_microbatches_calculator not found, using Apex version.
OneLogger: Setting error_handling_strategy to DISABLE_QUIETLY_AND_REPORT_METRIC_ERROR for rank (rank=0) with OneLogger disabled. To override: explicitly set error_handling_strategy parameter.
No exporters were provided. This means that no telemetry data will be collected.


**Planning health polling loop**


I’ve restarted and redeployed the app cleanly. I’m polling the endpoint until the new container comes up and confirms `gpu=H100` before I kick off the final benchmark run.


for i in $(seq 1 60); do echo "--- poll $i ---"; resp=$(curl -sS https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run/health || true); echo "$resp"; if echo "$resp" | grep -q "NVIDIA H100"; then break; fi; sleep 10; done


Output:
Exit code: 0
Wall time: 130.9 seconds
Output:
--- poll 1 ---
{"status":"healthy","active_streams":0,"max_streams":400,"gpu":"NVIDIA H100 80GB HBM3","vram_gb":10.5,"vram_total_gb":85.0,"encoder":"tensorrt","inference_running":false,"uptime_s":0.0}


Summary: auto


**Preparing final benchmark run**


Health now confirms the new deployment is **TRT on H100** (`gpu: NVIDIA H100 80GB HBM3`, `encoder: tensorrt`). I’m running the final WS real-time benchmark sweep to 500 concurrency now and will report the full score table.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode ws --concurrency 1,10,50,100,200,300,400,500 --cooldown 5 --prefix final_trt_h100_ws_rt


Output:
Exit code: 0
Wall time: 371.4 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=621ms FT_p99=621ms Thru=0.9x WER=37.5% CER=21.6%  Msgs=16.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=10: Starting 10 streams...
  [WS] c=10: OK=10 Err=0 FT_p50=629ms FT_p99=701ms Thru=6.3x WER=20.9% CER=10.6%  Msgs=19.4 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=50: Starting 50 streams...
  [WS] c=50: OK=50 Err=0 FT_p50=745ms FT_p99=799ms Thru=23.3x WER=33.5% CER=18.9%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=100: Starting 100 streams...
  [WS] c=100: OK=100 Err=0 FT_p50=797ms FT_p99=894ms Thru=46.6x WER=33.4% CER=18.8%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=200: Starting 200 streams...
  [WS] c=200: OK=200 Err=0 FT_p50=1095ms FT_p99=2072ms Thru=66.3x WER=33.8% CER=19.2%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=1174ms FT_p99=7425ms Thru=54.6x WER=38.9% CER=24.9%  Msgs=18.8 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1110ms FT_p99=2145ms Thru=65.2x WER=33.7% CER=19.1%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1081ms FT_p99=14802ms Thru=78.9x WER=44.4% CER=31.9%  Msgs=16.9 VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   37.5 |   21.6 |     621ms |     621ms |     9674ms |    9712ms |    0.9x |  16.0 |    10.5/85GB
    ws |    10 |   10 |    0 |   20.9 |   10.6 |     629ms |     701ms |    12616ms |   17933ms |    6.3x |  19.4 |    10.5/85GB
    ws |    50 |   50 |    0 |   33.5 |   18.9 |     745ms |     799ms |    12019ms |   25127ms |   23.3x |  20.0 |    10.5/85GB
    ws |   100 |  100 |    0 |   33.4 |   18.8 |     797ms |     894ms |    12376ms |   25131ms |   46.6x |  20.0 |    10.5/85GB
    ws |   200 |  200 |    0 |   33.8 |   19.2 |    1095ms |    2072ms |    20045ms |   35282ms |   66.3x |  19.9 |    10.5/85GB
    ws |   300 |  300 |    0 |   38.9 |   24.9 |    1174ms |    7425ms |    29992ms |   64281ms |   54.6x |  18.8 |    10.5/85GB
    ws |   400 |  400 |    0 |   33.7 |   19.1 |    1110ms |    2145ms |    32550ms |   71849ms |   65.2x |  20.0 |    10.5/85GB
    ws |   500 |  500 |    0 |   44.4 |   31.9 |    1081ms |   14802ms |    44064ms |   74148ms |   78.9x |  16.9 |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/final_trt_h100_ws_rt_NVIDIA_H100_80GB_HBM3_20260212_001133.csv
  /home/ubuntu/nemo_hindi/benchmark_results/final_trt_h100_ws_rt_NVIDIA_H100_80GB_HBM3_20260212_001133.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 37.5%
    c=  10: WER=20.9% (delta=-16.6%)
    c=  50: WER=33.5% (delta=-4.0%)
    c= 100: WER=33.4% (delta=-4.1%)
    c= 200: WER=33.8% (delta=-3.7%)
    c= 300: WER=38.9% (delta=+1.4%)
    c= 400: WER=33.7% (delta=-3.8%)
    c= 500: WER=44.4% (delta=+6.9%) << WARNING

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Verifying inference health and compatibility**


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "status": "healthy",
  "active_streams": 0,
  "max_streams": 400,
  "gpu": "NVIDIA H100 80GB HBM3",
  "vram_gb": 10.5,
  "vram_total_gb": 85,
  "encoder": "tensorrt",
  "inference_running": true,
  "uptime_s": 385
}


**Summarizing final deployment and benchmark**


TRT `hindi-nemotron-asr-1` is now deployed on **H100** (`nemotron_asr/nemotron_asr_trt.py:729` set to `gpu="H100"`), and `/health` confirms:

- `gpu`: `NVIDIA H100 80GB HBM3`
- `encoder`: `tensorrt`

Final benchmark run (WS realtime, conc `1,10,50,100,200,300,400,500`):

| Conc | OK | Err | FT p50 | FT p99 | Total p50 | Thru | WER | CER |
|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 1 | 1 | 0 | 621ms | 621ms | 9,674ms | 0.9x | 37.5% | 21.6% |
| 10 | 10 | 0 | 629ms | 701ms | 12,616ms | 6.3x | 20.9% | 10.6% |
| 50 | 50 | 0 | 745ms | 799ms | 12,019ms | 23.3x | 33.5% | 18.9% |
| 100 | 100 | 0 | 797ms | 894ms | 12,376ms | 46.6x | 33.4% | 18.8% |
| 200 | 200 | 0 | 1,095ms | 2,072ms | 20,045ms | 66.3x | 33.8% | 19.2% |
| 300 | 300 | 0 | 1,174ms | 7,425ms | 29,992ms | 54.6x | 38.9% | 24.9% |
| 400 | 400 | 0 | 1,110ms | 2,145ms | 32,550ms | 65.2x | 33.7% | 19.1% |
| 500 | 500 | 0 | 1,081ms | 14,802ms | 44,064ms | 78.9x | 44.4% | 31.9% |

Artifacts:
- `benchmark_results/final_trt_h100_ws_rt_NVIDIA_H100_80GB_HBM3_20260212_001133.csv`
- `benchmark_results/final_trt_h100_ws_rt_NVIDIA_H100_80GB_HBM3_20260212_001133.json`