Output:
Exit code: 124
Wall time: 7 seconds
Total output lines: 4658
Output:
command timed out after 7009 milliseconds
CONNECT /ws -> 101 Switching Protocols  (duration: 22.2 s, execution: 9.25 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.1 s, execution: 15.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.6 s, execution: 21.6 s)
[+127.338s] Client 799 disconnected, marked for cleanup (81 remaining)
[+127.339s] Stream 854: Detected format = RAW BYTES
[+402.554s] Stream 2105: Detected format = RAW BYTES
[+402.555s] Stream 2098: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846720.8500972, "audio_timestamp": 0.56, "is_final": false}
[+402.555s] Stream 2099: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846720.8500972, "audio_timestamp": 0.56, "is_final": false}
[+402.556s] Stream 2100: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846720.8500972, "audio_timestamp": 0.56, "is_final": false}
[+402.559s] Client 2036 disconnected, marked for cleanup (84 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.3 s, execution: 15.8 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.3 s, execution: 7.63 s)
[+127.397s] Client 773 disconnected, marked for cleanup (80 remaining)
✅ Opened stream_id=860
[+127.399s] Client 860 connected (81 total)
[+402.595s] Stream 2049: Received text Marker 'END'
✅ Opened stream_id=2110
[+402.596s] Client 2110 connected (85 total)
[+127.441s] Client 771 disconnected, marked for cleanup (80 remaining)
[+127.443s] Stream 860: Micro-batching enabled (batch_size=7)
[+402.642s] Stream 2049: Echoing Marker back to client (format=text)
[+402.642s] Stream 2110: Micro-batching enabled (batch_size=7)
✅ Opened stream_id=2111
[+402.642s] Client 2111 connected (86 total)
[+127.482s] Client 796 disconnected, marked for cleanup (79 remaining)
[+127.482s] Stream 766: Received text Marker 'END'
[+127.483s] Stream 855: Detected format = RAW BYTES
CONNECT /ws -> 101 Switching Protocols  (duration: 22.4 s, execution: 13.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 19.9 s)
[+402.701s] Stream 2111: Micro-batching enabled (batch_size=7)
[+402.713s] Stream 2101: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.008074, "audio_timestamp": 0.56, "is_final": false}
[+402.714s] Client 2049 disconnected, marked for cleanup (85 remaining)
[+402.715s] Stream 1996: Received text Marker 'END'
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 19.0 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 20.0 s)
[+127.552s] Stream 766: Echoing Marker back to client (format=text)
✅ Opened stream_id=861
[+127.554s] Client 861 connected (80 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 15.0 s)
[+402.768s] Stream 1996: Echoing Marker back to client (format=text)
[+402.768s] Stream 2018: Received text Marker 'END'
[+402.769s] Stream 2106: Detected format = RAW BYTES
[+402.769s] Stream 2107: Detected format = RAW BYTES
[+127.592s] Stream 861: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.5 s, execution: 12.6 s)
[+127.630s] Stream 856: Detected format = RAW BYTES
[+402.851s] Stream 2018: Echoing Marker back to client (format=text)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.5 s, execution: 16.2 s)
[+402.888s] Stream 2102: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.1830807, "audio_timestamp": 0.56, "is_final": false}
[+402.888s] Stream 2103: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.1830807, "audio_timestamp": 0.56, "is_final": false}
[+402.888s] Client 1996 disconnected, marked for cleanup (84 remaining)
[+402.892s] Client 2018 disconnected, marked for cleanup (83 remaining)
✅ Opened stream_id=2112
[+402.897s] Client 2112 connected (84 total)
[+127.673s] Stream 727: Received text Marker 'END'
✅ Opened stream_id=862
[+127.674s] Client 862 connected (81 total)
[+127.724s] Stream 727: Echoing Marker back to client (format=text)
[+127.726s] Stream 862: Micro-batching enabled (batch_size=7)
[+402.958s] Stream 2108: Detected format = RAW BYTES
[+402.958s] Stream 2109: Detected format = RAW BYTES
[+402.959s] Stream 2112: Micro-batching enabled (batch_size=7)
[+127.760s] Client 766 disconnected, marked for cleanup (80 remaining)
[+127.760s] Stream 857: Detected format = RAW BYTES
[+403.022s] Stream 2052: Received text Marker 'END'
[+403.022s] Stream 1983: Received text Marker 'END'
✅ Opened stream_id=2113
[+403.024s] Client 2113 connected (85 total)
✅ Opened stream_id=863
[+127.793s] Client 863 connected (81 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 15.3 s)
[+127.836s] Stream 791: Received text Marker 'END'
[+127.836s] Stream 863: Micro-batching enabled (batch_size=7)
[+403.030s] Stream 2052: Echoing Marker back to client (format=text)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 16.7 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.8 s, execution: 16.2 s)
[+127.896s] Stream 791: Echoing Marker back to client (format=text)
[+127.896s] Stream 858: Detected format = RAW BYTES
[+403.127s] Stream 1983: Echoing Marker back to client (format=text)
[+403.127s] Stream 2113: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.9 s, execution: 12.0 s)
✅ Opened stream_id=864
[+127.936s] Client 864 connected (82 total)
[+127.980s] Client 727 disconnected, marked for cleanup (81 remaining)
[+127.982s] Stream 864: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 22.9 s, execution: 12.7 s)
[+403.164s] Stream 2110: Detected format = RAW BYTES
[+403.164s] Stream 2111: Detected format = RAW BYTES
[+403.193s] Client 2052 disconnected, marked for cleanup (84 remaining)
[+403.193s] Client 1983 disconnected, marked for cleanup (83 remaining)
[+403.194s] Stream 2104: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.4900188, "audio_timestamp": 0.56, "is_final": false}
[+403.195s] Stream 2105: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.4900188, "audio_timestamp": 0.56, "is_final": false}
[+403.201s] Stream 2033: Received text Marker 'END'
[+403.202s] Stream 1995: Received text Marker 'END'
✅ Opened stream_id=2114
[+403.203s] Client 2114 connected (84 total)
✅ Opened stream_id=2115
[+403.203s] Client 2115 connected (85 total)
[+403.205s] Stream 2033: Echoing Marker back to client (format=text)
[+128.017s] Stream 859: Detected format = RAW BYTES
CONNECT /ws -> 101 Switching Protocols  (duration: 22.9 s, execution: 12.7 s)
✅ Opened stream_id=865
[+128.052s] Client 865 connected (82 total)
[+128.091s] Stream 865: Micro-batching enabled (batch_size=7)
[+403.265s] Stream 1995: Echoing Marker back to client (format=text)
[+403.266s] Stream 2114: Micro-batching enabled (batch_size=7)
[+403.266s] Stream 2115: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 22.8 s)
[+403.345s] Stream 2106: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.6349795, "audio_timestamp": 0.56, "is_final": false}
[+403.345s] Stream 2107: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.6349795, "audio_timestamp": 0.56, "is_final": false}
[+403.349s] Client 2033 disconnected, marked for cleanup (84 remaining)
[+403.349s] Client 1995 disconnected, marked for cleanup (83 remaining)
[+403.352s] Stream 2056: Received text Marker 'END'
[+403.359s] Stream 2056: Echoing Marker back to client (format=text)
[+128.120s] Client 791 disconnected, marked for cleanup (81 remaining)
[+128.121s] Stream 860: Detected format = RAW BYTES
[+128.153s] Stream 781: Received text Marker 'END'
✅ Opened stream_id=866
[+128.153s] Client 866 connected (82 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.1 s, execution: 8.82 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 20.3 s)
[+128.191s] Stream 781: Echoing Marker back to client (format=text)
[+128.192s] Stream 866: Micro-batching enabled (batch_size=7)
[+403.451s] Stream 2112: Detected format = RAW BYTES
[+403.494s] Stream 2108: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.7856262, "audio_timestamp": 0.56, "is_final": false}
[+403.494s] Stream 2109: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.7856262, "audio_timestamp": 0.56, "is_final": false}
[+403.498s] Client 2056 disconnected, marked for cleanup (82 remaining)
[+128.240s] Stream 861: Detected format = RAW BYTES
CONNECT /ws -> 101 Switching Protocols  (duration: 23.6 s, execution: 14.5 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.3 s, execution: 14.3 s)
✅ Opened stream_id=867
[+128.310s] Client 867 connected (83 total)
[+403.500s] Stream 2059: Received text Marker 'END'
[+403.503s] Stream 2113: Detected format = RAW BYTES
✅ Opened stream_id=2116
[+403.505s] Client 2116 connected (83 total)
[+403.509s] Stream 2059: Echoing Marker back to client (format=text)
[+128.350s] Stream 862: Detected format = RAW BYTES
[+128.350s] Stream 867: Micro-batching enabled (batch_size=7)
✅ Opened stream_id=868
[+128.380s] Client 868 connected (84 total)
[+403.589s] Stream 2116: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.3 s, execution: 11.4 s)
[+128.417s] Client 781 disconnected, marked for cleanup (83 remaining)
[+128.418s] Stream 868: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 15.4 s)
[+403.651s] Stream 2110: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.9399273, "audio_timestamp": 0.56, "is_final": false}
[+403.651s] Stream 2111: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846721.9399273, "audio_timestamp": 0.56, "is_final": false}
[+403.655s] Client 2059 disconnected, marked for cleanup (82 remaining)
[+403.660s] Stream 2114: Detected format = RAW BYTES
✅ Opened stream_id=2117
[+403.660s] Client 2117 connected (83 total)
[+128.459s] Stream 863: Detected format = RAW BYTES
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 11.6 s)
[+403.738s] Stream 2115: Detected format = RAW BYTES
[+403.738s] Stream 2117: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 12.0 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.4 s, execution: 15.7 s)
[+403.782s] Stream 2079: Received text Marker 'END'
✅ Opened stream_id=2118
[+403.782s] Client 2118 connected (84 total)
✅ Opened stream_id=869
[+128.498s] Client 869 connected (84 total)
[+128.536s] Stream 869: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.9 s, execution: 19.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.8 s, execution: 17.7 s)
[+128.577s] Stream 864: Detected format = RAW BYTES
✅ Opened stream_id=870
[+128.617s] Client 870 connected (85 total)
[+403.871s] Stream 2079: Echoing Marker back to client (format=text)
[+403.872s] Stream 2118: Micro-batching enabled (batch_size=7)
[+128.651s] Stream 870: Micro-batching enabled (batch_size=7)
[+128.687s] Stream 865: Detected format = RAW BYTES
[+403.938s] Client 2079 disconnected, marked for cleanup (83 remaining)
[+403.939s] Stream 1975: Received text Marker 'END'
[+403.941s] Stream 2024: Received text Marker 'END'
[+403.942s] Stream 2116: Detected format = RAW BYTES
✅ Opened stream_id=2119
[+403.943s] Client 2119 connected (84 total)
✅ Opened stream_id=871
[+128.726s] Client 871 connected (86 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.7 s, execution: 16.4 s)
[+404.016s] Stream 1975: Echoing Marker back to client (format=text)
[+404.016s] Stream 2024: Echoing Marker back to client (format=text)
[+404.017s] Stream 2119: Micro-batching enabled (batch_size=7)
[+128.770s] Stream 871: Micro-batching enabled (batch_size=7)
[+128.808s] Stream 866: Detected format = RAW BYTES
✅ Opened stream_id=872
[+128.844s] Client 872 connected (87 total)
[+404.125s] Stream 2112: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846722.4160528, "audio_timestamp": 0.56, "is_final": false}
[+404.125s] Stream 2113: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846722.4160528, "audio_timestamp": 0.56, "is_final": false}
[+404.126s] Client 1975 disconnected, marked for cleanup (83 remaining)
[+404.127s] Client 2024 disconnected, marked for cleanup (82 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.2 s, execution: 21.1 s)
[+128.890s] Stream 872: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.2 s, execution: 21.1 s)
✅ Opened stream_id=2120
[+404.131s] Client 2120 connected (83 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 23.9 s, execution: 11.6 s)
✅ Opened stream_id=873
[+128.964s] Client 873 connected (88 total)
[+404.188s] Stream 2117: Detected format = RAW BYTES
[+404.188s] Stream 2120: Micro-batching enabled (batch_size=7)
[+129.002s] Stream 867: Detected format = RAW BYTES
[+129.003s] Stream 873: Micro-batching enabled (batch_size=7)
[+404.254s] Stream 2114: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846722.544979, "audio_timestamp": 0.56, "is_final": false}
[+404.254s] Stream 2115: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846722.544979, "audio_timestamp": 0.56, "is_final": false}
CONNECT /ws -> 101 Switching Protocols  (duration: 24.3 s, execution: 15.0 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.3 s, execution: 20.5 s)
[+404.261s] Stream 2060: Received text Marker 'END'
[+404.261s] Stream 2074: Received text Marker 'END'
✅ Opened stream_id=2121
[+404.262s] Client 2121 connected (84 total)
✅ Opened stream_id=2122
[+404.264s] Client 2122 connected (85 total)
[+404.271s] Stream 2060: Echoing Marker back to client (format=text)
[+129.044s] Stream 868: Detected format = RAW BYTES
✅ Opened stream_id=874
[+129.081s] Client 874 connected (89 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.0 s, execution: 13.7 s)
[+129.120s] Stream 874: Micro-batching enabled (batch_size=7)
[+404.363s] Stream 2074: Echoing Marker back to client (format=text)
[+404.364s] Stream 2121: Micro-batching enabled (batch_size=7)
[+404.364s] Stream 2122: Micro-batching enabled (batch_size=7)
[+129.156s] Stream 869: Detected format = RAW BYTES
[+404.415s] Stream 2118: Detected format = RAW BYTES
[+129.203s] Stream 787: Received text Marker 'END'
✅ Opened stream_id=875
[+129.203s] Client 875 connected (90 total)
[+404.457s] Client 2060 disconnected, marked for cleanup (84 remaining)
[+404.458s] Client 2074 disconnected, marked for cleanup (83 remaining)
[+404.464s] Stream 2046: Received text Marker 'END'
[+404.465s] Stream 2051: Received text Marker 'END'
[+404.466s] Stream 2119: Detected format = RAW BYTES
✅ Opened stream_id=2123
[+404.468s] Client 2123 connected (84 total)
[+404.469s] Stream 2046: Echoing Marker back to client (format=text)
[+129.242s] Stream 787: Echoing Marker back to client (format=text)
[+129.243s] Stream 875: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.2 s, execution: 11.3 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.2 s, execution: 11.0 s)
[+404.549s] Stream 2051: Echoing Marker back to client (format=text)
[+404.550s] Stream 2123: Micro-batching enabled (batch_size=7)
[+129.293s] Stream 870: Detected format = RAW BYTES
✅ Opened stream_id=876
[+129.326s] Client 876 connected (91 total)
[+404.598s] Stream 2116: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846722.887805, "audio_timestamp": 0.56, "is_final": false}
[+404.600s] Client 2046 disconnected, marked for cleanup (83 remaining)
[+404.600s] Client 2051 disconnected, marked for cleanup (82 remaining)
[+404.601s] Stream 2025: Received text Marker 'END'
[+404.602s] Stream 1972: Received text Marker 'END'
[+404.603s] Stream 2000: Received text Marker 'END'
[+129.366s] Stream 876: Micro-batching enabled (batch_size=7)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.3 s, execution: 15.1 s)
[+129.405s] Stream 871: Detected format = RAW BYTES
[+404.701s] Stream 2025: Echoing Marker back to client (format=text)
[+404.701s] Stream 1972: Echoing Marker back to client (format=text)
[+404.701s] Stream 2000: Echoing Marker back to client (format=text)
[+404.701s] Stream 2120: Detected format = RAW BYTES
✅ Opened stream_id=877
[+129.443s] Client 877 connected (92 total)
[+129.479s] Client 787 disconnected, marked for cleanup (91 remaining)
[+129.480s] Stream 877: Micro-batching enabled (batch_size=7)
[+129.522s] Stream 872: Detected format = RAW BYTES
[+404.755s] Stream 2117: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846723.045678, "audio_timestamp": 0.56, "is_final": false}
[+404.757s] Client 2025 disconnected, marked for cleanup (81 remaining)
[+404.757s] Client 1972 disconnected, marked for cleanup (80 remaining)
[+404.757s] Client 2000 disconnected, marked for cleanup (79 remaining)
[+404.760s] Stream 2007: Received text Marker 'END'
✅ Opened stream_id=878
[+129.567s] Client 878 connected (92 total)
[+404.761s] Stream 2121: Detected format = RAW BYTES
[+404.761s] Stream 2122: Detected format = RAW BYTES
✅ Opened stream_id=2124
[+404.764s] Client 2124 connected (80 total)
✅ Opened stream_id=2125
[+404.764s] Client 2125 connected (81 total)
[+129.605s] Stream 878: Micro-batching enabled (batch_size=7)
[+129.642s] Stream 873: Detected format = RAW BYTES
[+404.875s] Stream 2007: Echoing Marker back to client (format=text)
[+404.876s] Stream 2124: Micro-batching enabled (batch_size=7)
[+404.876s] Stream 2125: Micro-batching enabled (batch_size=7)
✅ Opened stream_id=879
[+129.688s] Client 879 connected (93 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 24.6 s, execution: 7.89 s)
[+404.918s] Stream 2118: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846723.2067237, "audio_timestamp": 0.56, "is_final": false}
[+404.920s] Client 2007 disconnected, marked for cleanup (80 remaining)
[+404.926s] Stream 2066: Received text Marker 'END'
[+404.929s] Stream 2123: Detected format = RAW BYTES
[+404.933s] Stream 2066: Echoing Marker back to client (format=text)
[+129.731s] Stream 726: Received text Marker 'END'
[+129.732s] Stream 879: Micro-batching enabled (batch_size=7)
[+129.773s] Stream 726: Echoing Marker back to client (format=text)
[+129.773s] Stream 874: Detected format = RAW BYTES
[+405.042s] Stream 2119: Sending first JSON text message: {"text": "<s>", "timestamp": 1770846723.3315053, "audio_timestamp": 0.56, "is_final": false}
[+405.044s] Client 2066 disconnected, marked for cleanup (79 remaining)
✅ Opened stream_id=880
[+129.832s] Client 880 connected (94 total)
CONNECT /ws -> 101 Switching Protocols  (duration: 25.1 s, execution: 22.5 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 25.1 s, execution: 17.6 s)
[+405.051s] Stream 2037: Received text Marker 'END'
✅ Opened stream_id=2126
[+405.054s] Client 2126 connected (80 total)
[+129.871s] Stream 880: Micro-batching enabled (batch_size=7)
[+405.131s] Stream 2037: Echoing Marker back to client (format=text)
[+405.132s] Stream 2126: Micro-batching enabled (batch_size=7)
[+129.911s] Stream 875: Detected format = RAW BYTES
[+405.232s] Client 2037 disconnected, marked for cleanup (79 remaining)
[+405.234s] Stream 2029: Received text Marker 'END'
[+405.234s] Stream 2053: Received text Marker 'END'
[+405.235s] Stream 2040: Received text Marker 'END'
[+405.237s] Stream 2124: Detected format = RAW BYTES
[+405.237s] Stream 2125: Detected format = RAW BYTES
✅ Opened stream_id=2127
[+405.237s] Client 2127 connected (80 total)
✅ Opened stream_id=881
[+129.953s] Client 881 connected (95 total)
[+129.988s] Stream 881: Micro-batc…53309 tokens truncated…g)
CONNECT /ws -> 101 Switching Protocols  (duration: 43.9 s, execution: 19.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 43.9 s, execution: 12.4 s)
[+424.583s] Stream 2131: Received text Marker 'END'
[+424.583s] Stream 2131: Echoing Marker back to client (format=text)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.1 s, execution: 17.4 s)
[+  424.6s] METRICS | streams=24 batches=6700 outputs=46523 empty=0 vad_skip=0
         GPU: p50=28.5ms p99=65.9ms | Cycle: p50=142.6ms p99=657.3ms | Batch: avg=3 max=10
         VRAM: 10.4/48GB (22%)
[+424.696s] Client 2131 disconnected, marked for cleanup (23 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 43.8 s, execution: 14.3 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.1 s, execution: 14.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.1 s, execution: 12.7 s)
[+424.739s] Stream 2176: Received text Marker 'END'
[+424.739s] Stream 2176: Echoing Marker back to client (format=text)
[+424.782s] Stream 2174: Received text Marker 'END'
CONNECT /ws -> 101 Switching Protocols  (duration: 44.2 s, execution: 19.5 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.4 s, execution: 22.4 s)
[+424.805s] Stream 2174: Echoing Marker back to client (format=text)
[+424.841s] Client 2176 disconnected, marked for cleanup (22 remaining)
[+424.871s] Stream 2172: Received text Marker 'END'
[+424.872s] Stream 2172: Echoing Marker back to client (format=text)
[+424.929s] Client 2174 disconnected, marked for cleanup (21 remaining)
[+424.930s] Stream 2153: Received text Marker 'END'
[+424.975s] Stream 2153: Echoing Marker back to client (format=text)
[+424.976s] Client 2172 disconnected, marked for cleanup (20 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.4 s, execution: 13.6 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.4 s, execution: 13.9 s)
[+425.057s] Client 2153 disconnected, marked for cleanup (19 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.5 s, execution: 12.3 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.5 s, execution: 14.2 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.6 s, execution: 16.5 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.7 s, execution: 16.6 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.8 s, execution: 17.0 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 45.0 s, execution: 22.3 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 44.9 s, execution: 12.7 s)
[+425.665s] Stream 2143: Received text Marker 'END'
[+425.665s] Stream 2143: Echoing Marker back to client (format=text)
[+425.746s] Stream 2178: Received text Marker 'END'
[+425.747s] Stream 2178: Echoing Marker back to client (format=text)
[+425.781s] Client 2143 disconnected, marked for cleanup (18 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 45.3 s, execution: 18.4 s)
[+149.206s] Client 927 disconnected, marked for cleanup (22 remaining)
[+149.235s] Client 931 disconnected, marked for cleanup (21 remaining)
[+425.855s] Client 2178 disconnected, marked for cleanup (17 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 45.3 s, execution: 14.1 s)
[+149.255s] Stream 915: Received text Marker 'END'
[+149.255s] Stream 915: Echoing Marker back to client (format=text)
[+149.288s] Stream 930: Received text Marker 'END'
[+149.289s] Stream 930: Echoing Marker back to client (format=text)
[+149.382s] Client 915 disconnected, marked for cleanup (20 remaining)
[+149.409s] Client 930 disconnected, marked for cleanup (19 remaining)
[+149.503s] Stream 865: Received text Marker 'END'
[+149.503s] Stream 865: Echoing Marker back to client (format=text)
[+149.606s] Client 865 disconnected, marked for cleanup (18 remaining)
[+  149.6s] METRICS | streams=19 batches=2700 outputs=19930 empty=0 vad_skip=0
         GPU: p50=28.2ms p99=108.2ms | Cycle: p50=143.8ms p99=670.2ms | Batch: avg=3 max=14
         VRAM: 10.4/48GB (22%)
[+149.748s] Stream 936: Received text Marker 'END'
[+149.748s] Stream 936: Echoing Marker back to client (format=text)
[+149.851s] Client 936 disconnected, marked for cleanup (17 remaining)
[+149.970s] Stream 910: Received text Marker 'END'
[+149.970s] Stream 910: Echoing Marker back to client (format=text)
[+150.080s] Stream 908: Received text Marker 'END'
[+150.080s] Stream 908: Echoing Marker back to client (format=text)
[+150.101s] Client 910 disconnected, marked for cleanup (16 remaining)
[+150.102s] Stream 870: Received text Marker 'END'
[+150.103s] Stream 870: Echoing Marker back to client (format=text)
[+150.181s] Client 908 disconnected, marked for cleanup (15 remaining)
[+150.202s] Client 870 disconnected, marked for cleanup (14 remaining)
[+150.251s] Stream 937: Received text Marker 'END'
[+150.251s] Stream 937: Echoing Marker back to client (format=text)
[+150.368s] Client 937 disconnected, marked for cleanup (13 remaining)
[+426.155s] Stream 2169: Received text Marker 'END'
[+426.155s] Stream 2169: Echoing Marker back to client (format=text)
[+150.952s] Stream 933: Received text Marker 'END'
[+150.952s] Stream 933: Echoing Marker back to client (format=text)
[+426.256s] Client 2169 disconnected, marked for cleanup (16 remaining)
[+151.054s] Client 933 disconnected, marked for cleanup (12 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 45.6 s, execution: 13.9 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 45.7 s, execution: 15.7 s)
[+151.209s] Stream 894: Received text Marker 'END'
[+151.209s] Stream 894: Echoing Marker back to client (format=text)
[+151.335s] Client 894 disconnected, marked for cleanup (11 remaining)
[+426.596s] Stream 2184: Received text Marker 'END'
[+426.596s] Stream 2184: Echoing Marker back to client (format=text)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.1 s, execution: 20.4 s)
[+151.452s] Stream 920: Received text Marker 'END'
[+151.452s] Stream 920: Echoing Marker back to client (format=text)
[+426.707s] Client 2184 disconnected, marked for cleanup (15 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.1 s, execution: 14.3 s)
[+151.476s] Stream 938: Received text Marker 'END'
[+151.476s] Stream 938: Echoing Marker back to client (format=text)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.2 s, execution: 13.8 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.1 s, execution: 16.3 s)
[+151.552s] Client 920 disconnected, marked for cleanup (10 remaining)
[+151.577s] Client 938 disconnected, marked for cleanup (9 remaining)
[+151.581s] Stream 896: Received text Marker 'END'
[+151.582s] Stream 896: Echoing Marker back to client (format=text)
[+151.628s] Stream 898: Received text Marker 'END'
[+151.628s] Stream 898: Echoing Marker back to client (format=text)
[+151.683s] Client 896 disconnected, marked for cleanup (8 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.4 s, execution: 20.5 s)
[+151.730s] Client 898 disconnected, marked for cleanup (7 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.3 s, execution: 20.1 s)
[+151.762s] Stream 939: Received text Marker 'END'
[+151.762s] Stream 939: Echoing Marker back to client (format=text)
[+427.096s] Stream 2111: Received text Marker 'END'
[+427.096s] Stream 2111: Echoing Marker back to client (format=text)
[+151.862s] Client 939 disconnected, marked for cleanup (6 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.5 s, execution: 13.9 s)
[+427.122s] Stream 2182: Received text Marker 'END'
[+427.122s] Stream 2182: Echoing Marker back to client (format=text)
[+427.214s] Client 2111 disconnected, marked for cleanup (14 remaining)
[+427.223s] Client 2182 disconnected, marked for cleanup (13 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.7 s, execution: 25.2 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 46.6 s, execution: 15.1 s)
[+427.366s] Stream 2150: Received text Marker 'END'
[+427.383s] Stream 2150: Echoing Marker back to client (format=text)
[+427.496s] Client 2150 disconnected, marked for cleanup (12 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.0 s, execution: 19.1 s)
[+427.982s] Stream 2157: Received text Marker 'END'
[+428.004s] Stream 2157: Echoing Marker back to client (format=text)
[+428.068s] Stream 2133: Received text Marker 'END'
[+428.068s] Stream 2133: Echoing Marker back to client (format=text)
[+428.106s] Client 2157 disconnected, marked for cleanup (11 remaining)
[+152.930s] Stream 905: Received text Marker 'END'
[+152.931s] Stream 905: Echoing Marker back to client (format=text)
[+428.143s] Stream 2117: Received text Marker 'END'
[+428.143s] Stream 2117: Echoing Marker back to client (format=text)
[+428.169s] Client 2133 disconnected, marked for cleanup (10 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.7 s, execution: 19.1 s)
[+153.032s] Client 905 disconnected, marked for cleanup (5 remaining)
[+  428.3s] METRICS | streams=10 batches=6800 outputs=46645 empty=0 vad_skip=0
         GPU: p50=20.9ms p99=45.0ms | Cycle: p50=42.9ms p99=276.2ms | Batch: avg=1 max=7
         VRAM: 10.4/48GB (22%)
[+428.259s] Client 2117 disconnected, marked for cleanup (9 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.7 s, execution: 22.8 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.6 s, execution: 20.3 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.8 s, execution: 25.3 s)
[+428.318s] Stream 2179: Received text Marker 'END'
[+428.318s] Stream 2179: Echoing Marker back to client (format=text)
[+428.380s] Stream 2144: Received text Marker 'END'
[+428.380s] Stream 2144: Echoing Marker back to client (format=text)
[+428.419s] Client 2179 disconnected, marked for cleanup (8 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 47.8 s, execution: 16.7 s)
[+428.481s] Client 2144 disconnected, marked for cleanup (7 remaining)
[+428.622s] Stream 2167: Received text Marker 'END'
[+428.622s] Stream 2167: Echoing Marker back to client (format=text)
[+153.444s] Stream 913: Received text Marker 'END'
[+153.444s] Stream 913: Echoing Marker back to client (format=text)
[+428.681s] Stream 2146: Received text Marker 'END'
[+428.682s] Stream 2146: Echoing Marker back to client (format=text)
[+428.723s] Client 2167 disconnected, marked for cleanup (6 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.0 s, execution: 21.1 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.1 s, execution: 18.3 s)
[+153.544s] Client 913 disconnected, marked for cleanup (4 remaining)
[+428.796s] Client 2146 disconnected, marked for cleanup (5 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.2 s, execution: 19.6 s)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.3 s, execution: 21.1 s)
[+429.229s] Stream 2124: Received text Marker 'END'
[+429.229s] Stream 2124: Echoing Marker back to client (format=text)
[+429.330s] Client 2124 disconnected, marked for cleanup (4 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.9 s, execution: 25.2 s)
[+429.389s] Stream 2163: Received text Marker 'END'
[+429.389s] Stream 2163: Echoing Marker back to client (format=text)
[+429.497s] Client 2163 disconnected, marked for cleanup (3 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 48.9 s, execution: 19.6 s)
[+154.359s] Stream 919: Received text Marker 'END'
[+154.359s] Stream 919: Echoing Marker back to client (format=text)
[+154.460s] Client 919 disconnected, marked for cleanup (3 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 49.1 s, execution: 19.4 s)
[+154.634s] Stream 934: Received text Marker 'END'
[+154.634s] Stream 934: Echoing Marker back to client (format=text)
[+154.735s] Client 934 disconnected, marked for cleanup (2 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 49.3 s, execution: 17.5 s)
[+155.142s] Stream 922: Received text Marker 'END'
[+155.142s] Stream 922: Echoing Marker back to client (format=text)
[+155.242s] Client 922 disconnected, marked for cleanup (1 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 49.9 s, execution: 19.7 s)
[+432.731s] Stream 2170: Received text Marker 'END'
[+432.731s] Stream 2170: Echoing Marker back to client (format=text)
[+432.833s] Client 2170 disconnected, marked for cleanup (2 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 52.2 s, execution: 22.2 s)
[+433.503s] Stream 2181: Received text Marker 'END'
[+433.504s] Stream 2181: Echoing Marker back to client (format=text)
[+433.605s] Client 2181 disconnected, marked for cleanup (1 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 53.0 s, execution: 21.5 s)
[+433.985s] Stream 2155: Received text Marker 'END'
[+433.985s] Stream 2155: Echoing Marker back to client (format=text)
[+434.087s] Client 2155 disconnected, marked for cleanup (0 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 53.6 s, execution: 25.2 s)
[+162.027s] Stream 929: Received text Marker 'END'
[+162.027s] Stream 929: Echoing Marker back to client (format=text)
[+  162.0s] METRICS | streams=1 batches=2800 outputs=20039 empty=0 vad_skip=0
         GPU: p50=18.7ms p99=60.1ms | Cycle: p50=43.7ms p99=595.1ms | Batch: avg=1 max=6
         VRAM: 10.4/48GB (22%)
[+162.129s] Client 929 disconnected, marked for cleanup (0 remaining)
CONNECT /ws -> 101 Switching Protocols  (duration: 56.7 s, execution: 25.5 s)
    GET /health -> 200 OK  (duration: 142.9 ms, execution: 27.2 ms)
an error occurred during closing of asynchronous generator <async_generator object _ContainerIOManager.get_data_in at 0x2ad846faa140>
asyncgen: <async_generator object _ContainerIOManager.get_data_in at 0x2ad846faa140>
RuntimeError: aclose(): asynchronous generator is already running
an error occurred during closing of asynchronous generator <async_generator object _ContainerIOManager.get_data_in at 0x2ad846a22840>
asyncgen: <async_generator object _ContainerIOManager.get_data_in at 0x2ad846a22840>
RuntimeError: aclose(): asynchronous generator is already running
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-241668' coro=<Synchronizer._wrap_check_async_leakage.<locals>.coro_wrapped() done, defined at /pkg/synchronicity/synchronizer.py:263> exception=ClientClosed(47100944586496)>
Traceback (most recent call last):
  File "/pkg/synchronicity/synchronizer.py", line 425, in wrapper_coro
    return await inner_task
           ^^^^^^^^^^^^^^^^
  File "/pkg/synchronicity/synchronizer.py", line 264, in coro_wrapped
    value = await coro
            ^^^^^^^^^^
  File "/pkg/synchronicity/exceptions.py", line 31, in coro_wrapped
    return await coro
           ^^^^^^^^^^
  File "/pkg/modal/_runtime/container_io_manager.py", line 693, in get_data_in
    async for data in _stream_function_call_data(
  File "/pkg/modal/_utils/function_utils.py", line 427, in _stream_function_call_data
    async for chunk in stub_fn.unary_stream(req):
  File "/pkg/modal/_grpc_client.py", line 190, in unary_stream
    async for response in self.client._call_stream(self.wrapped_method, request, metadata=metadata):
  File "/pkg/modal/client.py", line 354, in _call_stream
    yield await self._call_safely(stream.__anext__(), f"{grpclib_method.name}.recv")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pkg/modal/client.py", line 301, in _call_safely
    raise ClientClosed(id(self)) from None
modal.exception.ClientClosed: 47100944586496
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-92832' coro=<Synchronizer._wrap_check_async_leakage.<locals>.coro_wrapped() done, defined at /pkg/synchronicity/synchronizer.py:263> exception=ClientClosed(47100944586496)>
Traceback (most recent call last):
  File "/pkg/synchronicity/synchronizer.py", line 425, in wrapper_coro
    return await inner_task
           ^^^^^^^^^^^^^^^^
  File "/pkg/synchronicity/synchronizer.py", line 264, in coro_wrapped
    value = await coro
            ^^^^^^^^^^
  File "/pkg/synchronicity/exceptions.py", line 31, in coro_wrapped
    return await coro
           ^^^^^^^^^^
  File "/pkg/modal/_runtime/container_io_manager.py", line 693, in get_data_in
    async for data in _stream_function_call_data(
  File "/pkg/modal/_utils/function_utils.py", line 427, in _stream_function_call_data
    async for chunk in stub_fn.unary_stream(req):
  File "/pkg/modal/_grpc_client.py", line 190, in unary_stream
    async for response in self.client._call_stream(self.wrapped_method, request, metadata=metadata):
  File "/pkg/modal/client.py", line 354, in _call_stream
    yield await self._call_safely(stream.__anext__(), f"{grpclib_method.name}.recv")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pkg/modal/client.py", line 301, in _call_safely
    raise ClientClosed(id(self)) from None
modal.exception.ClientClosed: 47100944586496
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-92823' coro=<Synchronizer._run_function_async.<locals>.wrapper_coro() done, defined at /pkg/synchronicity/synchronizer.py:422> exception=ClientClosed(47100944586496)>
Traceback (most recent call last):
  File "/pkg/synchronicity/synchronizer.py", line 425, in wrapper_coro
    return await inner_task
           ^^^^^^^^^^^^^^^^
  File "/pkg/synchronicity/synchronizer.py", line 264, in coro_wrapped
    value = await coro
            ^^^^^^^^^^
  File "/pkg/synchronicity/exceptions.py", line 31, in coro_wrapped
    return await coro
           ^^^^^^^^^^
  File "/pkg/modal/_runtime/container_io_manager.py", line 693, in get_data_in
    async for data in _stream_function_call_data(
  File "/pkg/modal/_utils/function_utils.py", line 427, in _stream_function_call_data
    async for chunk in stub_fn.unary_stream(req):
  File "/pkg/modal/_grpc_client.py", line 190, in unary_stream
    async for response in self.client._call_stream(self.wrapped_method, request, metadata=metadata):
  File "/pkg/modal/client.py", line 354, in _call_stream
    yield await self._call_safely(stream.__anext__(), f"{grpclib_method.name}.recv")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pkg/modal/client.py", line 301, in _call_safely
    raise ClientClosed(id(self)) from None
modal.exception.ClientClosed: 47100944586496
unhandled exception during asyncio.run() shutdown
task: <Task finished name='Task-241579' coro=<Synchronizer._run_function_async.<locals>.wrapper_coro() done, defined at /pkg/synchronicity/synchronizer.py:422> exception=ClientClosed(47100944586496)>
Traceback (most recent call last):
  File "/pkg/synchronicity/synchronizer.py", line 425, in wrapper_coro
    return await inner_task
           ^^^^^^^^^^^^^^^^
  File "/pkg/synchronicity/synchronizer.py", line 264, in coro_wrapped
    value = await coro
            ^^^^^^^^^^
  File "/pkg/synchronicity/exceptions.py", line 31, in coro_wrapped
    return await coro
           ^^^^^^^^^^
  File "/pkg/modal/_runtime/container_io_manager.py", line 693, in get_data_in
    async for data in _stream_function_call_data(
  File "/pkg/modal/_utils/function_utils.py", line 427, in _stream_function_call_data
    async for chunk in stub_fn.unary_stream(req):
  File "/pkg/modal/_grpc_client.py", line 190, in unary_stream
    async for response in self.client._call_stream(self.wrapped_method, request, metadata=metadata):
  File "/pkg/modal/client.py", line 354, in _call_stream
    yield await self._call_safely(stream.__anext__(), f"{grpclib_method.name}.recv")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pkg/modal/client.py", line 301, in _call_safely
    raise ClientClosed(id(self)) from None
modal.exception.ClientClosed: 47100944586496
Task was destroyed but it is pending!
task: <Task pending name='Task-240' coro=<NemotronASR.centralized_inference_loop() running at /root/nemotron_asr_trt.py:1713> wait_for=<Future pending cb=[Task.task_wakeup()]>>


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Inspecting function names for endpoints**


rg -n "@modal\.asgi_app|def web|def fastapi|@app\.function|@modal\.web_server|/health|/ws|/transcribe" nemotron_asr/nemotron_asr.py nemotron_asr/nemotron_asr_trt.py


nl -ba nemotron_asr/nemotron_asr.py | sed -n '1440,1525p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
nemotron_asr/nemotron_asr_trt.py:85:    # Original: import uvicorn, threading — removed: tunnel + uvicorn server replaced by @modal.asgi_app()
nemotron_asr/nemotron_asr_trt.py:1012:        @self.web_app.get("/health")
nemotron_asr/nemotron_asr_trt.py:1148:        @self.web_app.post("/transcribe")
nemotron_asr/nemotron_asr_trt.py:1195:        @self.web_app.websocket("/ws")
nemotron_asr/nemotron_asr_trt.py:1535:        # Everything is served through @modal.asgi_app() natively — no tunnel overhead.
nemotron_asr/nemotron_asr_trt.py:1923:    @modal.asgi_app()
nemotron_asr/nemotron_asr_trt.py:1924:    def webapp(self):
nemotron_asr/nemotron_asr_trt.py:2017:    @modal.asgi_app()
nemotron_asr/nemotron_asr_trt.py:2018:    def web(self):
nemotron_asr/nemotron_asr_trt.py:2034:                cls_instance.webapp.web_url.replace("http", "ws") + "/ws"
nemotron_asr/nemotron_asr.py:78:    # Original: import uvicorn, threading — removed: tunnel + uvicorn server replaced by @modal.asgi_app()
nemotron_asr/nemotron_asr.py:526:        @self.web_app.get("/health")
nemotron_asr/nemotron_asr.py:649:        @self.web_app.post("/transcribe")
nemotron_asr/nemotron_asr.py:696:        @self.web_app.websocket("/ws")
nemotron_asr/nemotron_asr.py:1036:        # Everything is served through @modal.asgi_app() natively — no tunnel overhead.
nemotron_asr/nemotron_asr.py:1424:    @modal.asgi_app()
nemotron_asr/nemotron_asr.py:1425:    def webapp(self):
nemotron_asr/nemotron_asr.py:1518:    @modal.asgi_app()
nemotron_asr/nemotron_asr.py:1519:    def web(self):
nemotron_asr/nemotron_asr.py:1535:                cls_instance.webapp.web_url.replace("http", "ws") + "/ws"


Output:
Exit code: 0
Wall time: 0 seconds
Output:
  1440
  1441	    @modal.method()
  1442	    def transcribe_file(self, audio_url: str) -> dict:
  1443	        """
  1444	        Transcribe an audio file from a URL or local path
  1445
  1446	        Args:
  1447	            audio_url: URL or path to audio file
  1448
  1449	        Returns:
  1450	            Dictionary with transcription results
  1451	        """
  1452	        print(f"Transcribing audio from: {audio_url}")
  1453
  1454	        # Preprocess audio to ensure it's 16kHz mono
  1455	        start_time = time.perf_counter()
  1456	        audio_bytes = preprocess_audio(audio_url, target_sample_rate=16000)
  1457	        preprocess_time = time.perf_counter() - start_time
  1458	        print(f"Audio preprocessing took {preprocess_time:.2f} seconds")
  1459
  1460	        # Write to temp file for pipeline
  1461	        import tempfile
  1462	        import wave
  1463	        import os
  1464
  1465	        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
  1466	            with wave.open(tmp, "wb") as wav_file:
  1467	                wav_file.setnchannels(1)
  1468	                wav_file.setsampwidth(2)
  1469	                wav_file.setframerate(16000)
  1470	                wav_file.writeframes(audio_bytes)
  1471	            tmp_path = tmp.name
  1472
  1473	        # Calculate audio duration before running pipeline
  1474	        data_dur = len(audio_bytes) / (16000 * 2)  # 16kHz, 16-bit (2 bytes per sample)
  1475
  1476	        # Run pipeline
  1477	        start_time = time.perf_counter()
  1478	        progress_bar = TQDMProgressBar()
  1479	        output = self.pipeline.run([tmp_path], progress_bar=progress_bar)
  1480	        inference_time = time.perf_counter() - start_time
  1481
  1482	        # Clean up temp file
  1483	        os.unlink(tmp_path)
  1484
  1485	        result = {
  1486	            "text": output[0]["text"],
  1487	            "inference_time": inference_time,
  1488	            "audio_duration": data_dur,
  1489	        }
  1490
  1491	        print(f"Transcription complete!")
  1492	        print(f"  Text: {result['text']}")
  1493	        print(f"  Audio duration: {data_dur:.2f}s")
  1494	        print(f"  Inference time: {inference_time:.2f}s")
  1495
  1496	        return result
  1497
  1498
  1499	# ## Frontend Service
  1500	#
  1501	# We serve a simple HTML/JS frontend to interact with the transcriber.
  1502	# The frontend captures microphone input and streams it to the WebSocket endpoint.
  1503
  1504	web_image = (
  1505	    modal.Image.debian_slim(python_version="3.12")
  1506	    .pip_install("fastapi")
  1507	    .add_local_dir(Path(__file__).parent.parent  / "nemotron-asr-frontend", "/root/frontend")
  1508	)
  1509
  1510	with web_image.imports():
  1511	    from fastapi import FastAPI
  1512	    from fastapi.responses import HTMLResponse, Response
  1513	    from fastapi.staticfiles import StaticFiles
  1514
  1515
  1516	@app.cls(image=web_image)
  1517	class WebServer:
  1518	    @modal.asgi_app()
  1519	    def web(self):
  1520	        web_app = FastAPI()
  1521	        web_app.mount("/static", StaticFiles(directory="/root/frontend"), name="static")
  1522
  1523	        @web_app.get("/status")
  1524	        async def status():
  1525	            return Response(status_code=200)


Summary: auto


**Inspecting method access for web URL**


nl -ba nemotron_asr/nemotron_asr.py | sed -n '1400,1465p'


nl -ba nemotron_asr/nemotron_asr_trt.py | sed -n '1898,2055p'


Output:
Exit code: 0
Wall time: 0 seconds
Output:
  1400	    def start_inference_loop_if_needed(self):
  1401	        """
  1402	        Start the centralized inference loop if it's not already running.
  1403	        Called on first client connection.
  1404	        """
  1405	        if self.inference_task is None:
  1406	            print("🔄 Starting centralized inference loop (first client)...")
  1407	            self.inference_task = asyncio.create_task(self.centralized_inference_loop())
  1408	            print("✅ Centralized inference loop started!")
  1409	        elif self.inference_task.done():
  1410	            print("⚠  Inference task was done! Restarting...")
  1411	            # Check if it had an exception
  1412	            try:
  1413	                exc = self.inference_task.exception()
  1414	                if exc:
  1415	                    print(f"❌ Previous task failed with: {exc}")
  1416	            except:
  1417	                pass
  1418	            self.inference_task = asyncio.create_task(self.centralized_inference_loop())
  1419	            print("✅ Centralized inference loop restarted!")
  1420	        else:
  1421	            pass  # Already running
  1422
  1423
  1424	    @modal.asgi_app()
  1425	    def webapp(self):
  1426	        """Expose the FastAPI WebSocket app"""
  1427	        return self.web_app
  1428
  1429	    @modal.method()
  1430	    def get_config_dict(self) -> dict:
  1431	        """
  1432	        Get a serializable dictionary representation of the pipeline configuration.
  1433
  1434	        Returns:
  1435	            Dictionary containing all configuration values
  1436	        """
  1437	        # Convert OmegaConf to regular dict for serialization
  1438	        config_dict = OmegaConf.to_container(self.cfg, resolve=True)
  1439	        return config_dict
  1440
  1441	    @modal.method()
  1442	    def transcribe_file(self, audio_url: str) -> dict:
  1443	        """
  1444	        Transcribe an audio file from a URL or local path
  1445
  1446	        Args:
  1447	            audio_url: URL or path to audio file
  1448
  1449	        Returns:
  1450	            Dictionary with transcription results
  1451	        """
  1452	        print(f"Transcribing audio from: {audio_url}")
  1453
  1454	        # Preprocess audio to ensure it's 16kHz mono
  1455	        start_time = time.perf_counter()
  1456	        audio_bytes = preprocess_audio(audio_url, target_sample_rate=16000)
  1457	        preprocess_time = time.perf_counter() - start_time
  1458	        print(f"Audio preprocessing took {preprocess_time:.2f} seconds")
  1459
  1460	        # Write to temp file for pipeline
  1461	        import tempfile
  1462	        import wave
  1463	        import os
  1464
  1465	        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:


Output:
Exit code: 0
Wall time: 0 seconds
Output:
  1898
  1899	    def start_inference_loop_if_needed(self):
  1900	        """
  1901	        Start the centralized inference loop if it's not already running.
  1902	        Called on first client connection.
  1903	        """
  1904	        if self.inference_task is None:
  1905	            print("🔄 Starting centralized inference loop (first client)...")
  1906	            self.inference_task = asyncio.create_task(self.centralized_inference_loop())
  1907	            print("✅ Centralized inference loop started!")
  1908	        elif self.inference_task.done():
  1909	            print("⚠  Inference task was done! Restarting...")
  1910	            # Check if it had an exception
  1911	            try:
  1912	                exc = self.inference_task.exception()
  1913	                if exc:
  1914	                    print(f"❌ Previous task failed with: {exc}")
  1915	            except:
  1916	                pass
  1917	            self.inference_task = asyncio.create_task(self.centralized_inference_loop())
  1918	            print("✅ Centralized inference loop restarted!")
  1919	        else:
  1920	            pass  # Already running
  1921
  1922
  1923	    @modal.asgi_app()
  1924	    def webapp(self):
  1925	        """Expose the FastAPI WebSocket app"""
  1926	        return self.web_app
  1927
  1928	    @modal.method()
  1929	    def get_config_dict(self) -> dict:
  1930	        """
  1931	        Get a serializable dictionary representation of the pipeline configuration.
  1932
  1933	        Returns:
  1934	            Dictionary containing all configuration values
  1935	        """
  1936	        # Convert OmegaConf to regular dict for serialization
  1937	        config_dict = OmegaConf.to_container(self.cfg, resolve=True)
  1938	        return config_dict
  1939
  1940	    @modal.method()
  1941	    def transcribe_file(self, audio_url: str) -> dict:
  1942	        """
  1943	        Transcribe an audio file from a URL or local path
  1944
  1945	        Args:
  1946	            audio_url: URL or path to audio file
  1947
  1948	        Returns:
  1949	            Dictionary with transcription results
  1950	        """
  1951	        print(f"Transcribing audio from: {audio_url}")
  1952
  1953	        # Preprocess audio to ensure it's 16kHz mono
  1954	        start_time = time.perf_counter()
  1955	        audio_bytes = preprocess_audio(audio_url, target_sample_rate=16000)
  1956	        preprocess_time = time.perf_counter() - start_time
  1957	        print(f"Audio preprocessing took {preprocess_time:.2f} seconds")
  1958
  1959	        # Write to temp file for pipeline
  1960	        import tempfile
  1961	        import wave
  1962	        import os
  1963
  1964	        with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
  1965	            with wave.open(tmp, "wb") as wav_file:
  1966	                wav_file.setnchannels(1)
  1967	                wav_file.setsampwidth(2)
  1968	                wav_file.setframerate(16000)
  1969	                wav_file.writeframes(audio_bytes)
  1970	            tmp_path = tmp.name
  1971
  1972	        # Calculate audio duration before running pipeline
  1973	        data_dur = len(audio_bytes) / (16000 * 2)  # 16kHz, 16-bit (2 bytes per sample)
  1974
  1975	        # Run pipeline
  1976	        start_time = time.perf_counter()
  1977	        progress_bar = TQDMProgressBar()
  1978	        output = self.pipeline.run([tmp_path], progress_bar=progress_bar)
  1979	        inference_time = time.perf_counter() - start_time
  1980
  1981	        # Clean up temp file
  1982	        os.unlink(tmp_path)
  1983
  1984	        result = {
  1985	            "text": output[0]["text"],
  1986	            "inference_time": inference_time,
  1987	            "audio_duration": data_dur,
  1988	        }
  1989
  1990	        print(f"Transcription complete!")
  1991	        print(f"  Text: {result['text']}")
  1992	        print(f"  Audio duration: {data_dur:.2f}s")
  1993	        print(f"  Inference time: {inference_time:.2f}s")
  1994
  1995	        return result
  1996
  1997
  1998	# ## Frontend Service
  1999	#
  2000	# We serve a simple HTML/JS frontend to interact with the transcriber.
  2001	# The frontend captures microphone input and streams it to the WebSocket endpoint.
  2002
  2003	web_image = (
  2004	    modal.Image.debian_slim(python_version="3.12")
  2005	    .pip_install("fastapi")
  2006	    .add_local_dir(Path(__file__).parent.parent  / "nemotron-asr-frontend", "/root/frontend")
  2007	)
  2008
  2009	with web_image.imports():
  2010	    from fastapi import FastAPI
  2011	    from fastapi.responses import HTMLResponse, Response
  2012	    from fastapi.staticfiles import StaticFiles
  2013
  2014
  2015	@app.cls(image=web_image)
  2016	class WebServer:
  2017	    @modal.asgi_app()
  2018	    def web(self):
  2019	        web_app = FastAPI()
  2020	        web_app.mount("/static", StaticFiles(directory="/root/frontend"), name="static")
  2021
  2022	        @web_app.get("/status")
  2023	        async def status():
  2024	            return Response(status_code=200)
  2025
  2026	        # Serve frontend
  2027	        @web_app.get("/")
  2028	        async def index():
  2029	            html_content = open("/root/frontend/index.html").read()
  2030
  2031	            # Get the WebSocket URL from the NemotronASR
  2032	            cls_instance = NemotronASR()
  2033	            ws_base_url = (
  2034	                cls_instance.webapp.web_url.replace("http", "ws") + "/ws"
  2035	            )
  2036	            script_tag = f'<script>window.WS_BASE_URL = "{ws_base_url}";</script>'
  2037	            html_content = html_content.replace(
  2038	                '<script src="/static/cache-aware-stt.js"></script>',
  2039	                f'{script_tag}\n    <script src="/static/cache-aware-stt.js"></script>',
  2040	            )
  2041	            return HTMLResponse(content=html_content)
  2042
  2043	        return web_app
  2044
  2045
  2046	@app.local_entrypoint()
  2047	def main():
  2048	    """Quick smoke test — deploy and check Hindi transcription with TRT works"""
  2049	    print("Starting Hindi Nemotron ASR (TRT) deployment test...")
  2050	    runner = NemotronASR()
  2051	    config = runner.get_config_dict.remote()
  2052	    print(f"\nDeployed config: model={config['asr']['model_name'][:60]}...")
  2053	    print(f"  streaming.att_context_size={config['streaming']['att_context_size']}")
  2054	    print(f"  streaming.batch_size={config['streaming']['batch_size']}")
  2055	    print("✅ Hindi ASR TRT service is ready!")


**Testing retrieval of deployed URL**


source venv/bin/activate && python - <<'PY'
import modal
for app in ["hindi-nemotron-asr","hindi-nemotron-asr-1"]:
    cls = modal.Cls.from_name(app, "NemotronASR")
    print(app, "webapp=", cls.webapp.web_url)
PY


Output:
Exit code: 1
Wall time: 0.4 seconds
Output:
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/synchronicity/synchronizer.py", line 715, in proxy_method
    return wrapped_method(instance, *args, **kwargs)
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/synchronicity/combined_types.py", line 33, in __call__
    raise uc_exc.exc
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/modal/_object.py", line 45, in wrapped
    await self.hydrate()
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/modal/_object.py", line 377, in hydrate
    await resolver.load(self, root_load_context)
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/modal/_resolver.py", line 157, in load
    return await cached_future
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/modal/_resolver.py", line 134, in loader
    await obj._load(obj, self, load_context, existing_object_id)
  File "/home/ubuntu/nemo_hindi/venv/lib/python3.10/site-packages/modal/cls.py", line 913, in error_loader
    raise AttributeError(
AttributeError: You can't access methods on a Cls directly - Did you forget to instantiate the class first?
e.g. instead of MyClass.method.remote(), do MyClass().method.remote()


**Testing class instantiation**


source venv/bin/activate && python - <<'PY'
import modal
for app in ["hindi-nemotron-asr","hindi-nemotron-asr-1"]:
    cls = modal.Cls.from_name(app, "NemotronASR")
    inst = cls()
    print(app, "webapp=", inst.webapp.web_url)
PY


Output:
Exit code: 0
Wall time: 0.7 seconds
Output:
hindi-nemotron-asr webapp= https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
hindi-nemotron-asr-1 webapp= https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
/usr/lib/python3.10/asyncio/events.py:80: PendingDeprecationError: 2025-05-06: The Function.web_url property will be removed in a future version of Modal.
Use the `Function.get_web_url()` method instead.

  self._context.run(self._callback, *self._args)
/usr/lib/python3.10/asyncio/events.py:80: PendingDeprecationError: 2025-05-06: The Function.web_url property will be removed in a future version of Modal.
Use the `Function.get_web_url()` method instead.

  self._context.run(self._callback, *self._args)


**Planning thorough benchmark runs**


Audit benchmark harness correctness against server output semantics and metric calculations

Identify active Modal endpoints for torch.compile and TensorRT deployments and confirm health/encoder type

Run controlled WS real-time benchmark sweep (1..500 concurrency) on both deployments

Run additional production-like stress scenarios (burst WS and HTTP) for both deployments

Aggregate results, compare latency/throughput/quality/error behavior, and provide production recommendations

Explanation: I validated the benchmark script logic and located the deployed app handles; now I’m confirming endpoint health/encoder identity before launching the full sweeps.


Output: Plan updated


I’ve located both live endpoints directly from deployed Modal class handles, so we can benchmark apples-to-apples now. Next I’m verifying `/health` on each endpoint (including encoder identity), then I’ll run a controlled WS sweep at `1,10,50,100,200,300,400,500` for both compile and TRT, followed by additional stress scenarios (burst WS and HTTP) and a full comparison report.


**Testing health endpoints concurrently**


curl -sS https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run/health | jq .


curl -sS https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run/health | jq .


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "status": "healthy",
  "active_streams": 0,
  "max_streams": 400,
  "gpu": "NVIDIA H100 80GB HBM3",
  "vram_gb": 10.5,
  "vram_total_gb": 85,
  "inference_running": false,
  "uptime_s": 20355
}


Output:
Exit code: 0
Wall time: 0.5 seconds
Output:
{
  "status": "healthy",
  "active_streams": 0,
  "max_streams": 400,
  "gpu": "NVIDIA L40S",
  "vram_gb": 10.4,
  "vram_total_gb": 48,
  "encoder": "tensorrt",
  "inference_running": true,
  "uptime_s": 3236
}


**Assessing GPU and implementation differences**


set -o pipefail; modal app logs hindi-nemotron-asr | rg -m 20 -n "torch\.compile|TRT|tensorrt|encoder active|fallback"


set -o pipefail; modal app logs hindi-nemotron-asr-1 | rg -m 20 -n "torch\.compile|TRT|tensorrt|encoder active|fallback|cache_aware_stream_step|warmup"


Output:
Exit code: 124
Wall time: 12 seconds
Output:
command timed out after 12009 milliseconds


Output:
Exit code: 124
Wall time: 12 seconds
Output:
command timed out after 12009 milliseconds


Summary: auto


**Analyzing transcript message structure**


I’m validating one subtle benchmark integrity point now: whether WS `text` packets are true deltas or cumulative finals, because that directly affects WER correctness. I’ll run a single debug stream and inspect raw message payloads before running the long sweeps.


source venv/bin/activate && python - <<'PY'
import asyncio, aiohttp, wave, io, json
from pathlib import Path

BASE='https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run'
ws_url=BASE.replace('https://','wss://')+'/ws'
audio=Path('benchmark_data/fleurs_hindi/sample_0000.wav').read_bytes()
with wave.open(io.BytesIO(audio),'rb') as wf:
    pcm=wf.readframes(wf.getnframes())
chunk=5120

async def run():
    async with aiohttp.ClientSession() as s:
        ws=await s.ws_connect(ws_url, timeout=30)
        msg=await ws.receive()
        print('ready', msg.type, msg.data[:120] if hasattr(msg,'data') else '')
        for i in range(0,len(pcm),chunk):
            await ws.send_bytes(pcm[i:i+chunk])
            await asyncio.sleep(0.16)
        await ws.send_str('END')
        texts=[]
        while True:
            m=await ws.receive()
            if m.type==aiohttp.WSMsgType.TEXT:
                if m.data=='END':
                    break
                d=json.loads(m.data)
                texts.append((d.get('is_final'), d.get('text',''), d.get('segment_text','')))
            elif m.type==aiohttp.WSMsgType.BINARY:
                pass
            elif m.type in (aiohttp.WSMsgType.CLOSE,aiohttp.WSMsgType.CLOSED,aiohttp.WSMsgType.CLOSING,aiohttp.WSMsgType.ERROR):
                break
        await ws.close()
        print('count',len(texts))
        for i,t in enumerate(texts[:8]):
            print('first',i,t)
        for i,t in enumerate(texts[-8:]):
            print('last',len(texts)-8+i,t)
        concat=''.join([t[1] for t in texts if t[1].strip()!='<s>']).strip()
        finals=' '.join([t[2] for t in texts if t[2]])
        print('concat:',concat)
        print('segjoin:',finals)

asyncio.run(run())
PY


Output:
Exit code: 0
Wall time: 9.8 seconds
Output:
ready WSMsgType.TEXT {"type": "Ready", "id": 2185}
count 16
first 0 (False, '<s>', '')
first 1 (False, ' बाइट', '')
first 2 (False, ' कुछ अर', '')
first 3 (False, 'ुणों में', '')
first 4 (False, ' अस्थिर', '')
first 5 (False, 'ता को', '')
first 6 (False, 'ताया जिसका', '')
first 7 (False, ' मतलब', '')
last 8 (False, ' यहां कि', '')
last 9 (False, ' उनमें थो', '')
last 10 (False, 'ड़े या बिना', '')
last 11 (False, ' किसी झ', '')
last 12 (False, 'टके से टूट', '')
last 13 (False, 'ने की प्रव', '')
last 14 (False, 'ृति होती', '')
last 15 (False, ' है', '')
concat: बाइट कुछ अरुणों में अस्थिरता कोताया जिसका मतलब यहां कि उनमें थोड़े या बिना किसी झटके से टूटने की प्रवृति होती है
segjoin:
<stdin>:13: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'


**Confirming message concatenation approach**


source venv/bin/activate && python - <<'PY'
import asyncio, aiohttp, wave, io, json
from pathlib import Path

BASE='https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run'
ws_url=BASE.replace('https://','wss://')+'/ws'
audio=Path('benchmark_data/fleurs_hindi/sample_0000.wav').read_bytes()
with wave.open(io.BytesIO(audio),'rb') as wf:
    pcm=wf.readframes(wf.getnframes())
chunk=5120

async def run():
    async with aiohttp.ClientSession() as s:
        ws=await s.ws_connect(ws_url, timeout=30)
        msg=await ws.receive()
        print('ready', msg.type, msg.data[:120] if hasattr(msg,'data') else '')
        for i in range(0,len(pcm),chunk):
            await ws.send_bytes(pcm[i:i+chunk])
            await asyncio.sleep(0.16)
        await ws.send_str('END')
        texts=[]
        while True:
            m=await ws.receive()
            if m.type==aiohttp.WSMsgType.TEXT:
                if m.data=='END':
                    break
                d=json.loads(m.data)
                texts.append((d.get('is_final'), d.get('text',''), d.get('segment_text','')))
            elif m.type==aiohttp.WSMsgType.BINARY:
                pass
            elif m.type in (aiohttp.WSMsgType.CLOSE,aiohttp.WSMsgType.CLOSED,aiohttp.WSMsgType.CLOSING,aiohttp.WSMsgType.ERROR):
                break
        await ws.close()
        print('count',len(texts))
        for i,t in enumerate(texts[:8]):
            print('first',i,t)
        for i,t in enumerate(texts[-8:]):
            print('last',len(texts)-8+i,t)
        concat=''.join([t[1] for t in texts if t[1].strip()!='<s>']).strip()
        finals=' '.join([t[2] for t in texts if t[2]])
        print('concat:',concat)
        print('segjoin:',finals)

asyncio.run(run())
PY


Output:
Exit code: 0
Wall time: 9.7 seconds
Output:
ready WSMsgType.TEXT {"type": "Ready", "id": 3}
count 16
first 0 (False, '<s>', '')
first 1 (False, ' बाइट', '')
first 2 (False, ' कुछ अर', '')
first 3 (False, 'ुणों में', '')
first 4 (False, ' अस्थिर', '')
first 5 (False, ' केन्द्र को', '')
first 6 (False, ' तय जिसका', '')
first 7 (False, ' मतलब', '')
last 8 (False, ' यहां कि', '')
last 9 (False, ' उनमें थो', '')
last 10 (False, 'ड़े या बिना', '')
last 11 (False, ' किसी झ', '')
last 12 (False, 'टके से टूट', '')
last 13 (False, 'ने की प्रव', '')
last 14 (False, 'ृति होती', '')
last 15 (False, ' है', '')
concat: बाइट कुछ अरुणों में अस्थिर केन्द्र को तय जिसका मतलब यहां कि उनमें थोड़े या बिना किसी झटके से टूटने की प्रवृति होती है
segjoin:
<stdin>:13: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'


**Assessing benchmark script options**


source venv/bin/activate && python production/test_stress_full.py -h


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
usage: test_stress_full.py [-h] [--url URL] [--endpoint ENDPOINT]
                           [--endpoint-key ENDPOINT_KEY]
                           [--endpoint-id ENDPOINT_ID]
                           [--route-cost ROUTE_COST] [--mode {http,sse,both}]
                           [--levels LEVELS]
                           [--requests-per-client REQUESTS_PER_CLIENT]
                           [--cooldown COOLDOWN]
                           [--metrics-interval METRICS_INTERVAL]
                           [--inter-request-delay INTER_REQUEST_DELAY]
                           [--audio-source {sine,fleurs}]
                           [--audio-duration AUDIO_DURATION]
                           [--audio-durations AUDIO_DURATIONS]
                           [--sample-pool SAMPLE_POOL]
                           [--min-duration MIN_DURATION]
                           [--max-duration MAX_DURATION] [--shuffle-samples]
                           [--seed SEED] [--output-prefix OUTPUT_PREFIX]

Full stress + quality tester for Hindi ASR

options:
  -h, --help            show this help message and exit
  --url URL             Direct URL, e.g. http://ip:8000
  --endpoint ENDPOINT   Vast endpoint name
  --endpoint-key ENDPOINT_KEY
                        Endpoint API key
  --endpoint-id ENDPOINT_ID
                        Endpoint id
  --route-cost ROUTE_COST
  --mode {http,sse,both}
  --levels LEVELS
  --requests-per-client REQUESTS_PER_CLIENT
  --cooldown COOLDOWN
  --metrics-interval METRICS_INTERVAL
  --inter-request-delay INTER_REQUEST_DELAY
  --audio-source {sine,fleurs}
  --audio-duration AUDIO_DURATION
                        Used for sine source
  --audio-durations AUDIO_DURATIONS
                        Comma-separated durations for sine source
  --sample-pool SAMPLE_POOL
                        Pool size for fleurs samples
  --min-duration MIN_DURATION
  --max-duration MAX_DURATION
  --shuffle-samples
  --seed SEED
  --output-prefix OUTPUT_PREFIX


**Planning comprehensive benchmarks**


I’ve confirmed the WS payload semantics, so the existing WER reconstruction is valid for this service. I’m starting with a quick one-level sanity run on both endpoints, then I’ll launch the full `1,10,50,100,200,300,400,500` sweeps.


**Running sanity tests concurrently**


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run --mode ws --concurrency 1 --cooldown 1 --prefix sanity_compile_ws


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode ws --concurrency 1 --cooldown 1 --prefix sanity_trt_ws


Output:
Exit code: 0
Wall time: 11.5 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=612ms FT_p99=612ms Thru=0.9x WER=33.3% CER=18.1%  Msgs=16.0 VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   33.3 |   18.1 |     612ms |     612ms |     9562ms |    9601ms |    0.9x |  16.0 |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/sanity_compile_ws_NVIDIA_H100_80GB_HBM3_20260211_224104.csv
  /home/ubuntu/nemo_hindi/benchmark_results/sanity_compile_ws_NVIDIA_H100_80GB_HBM3_20260211_224104.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 33.3%

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


Output:
Exit code: 0
Wall time: 11.4 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA L40S
  VRAM: 10.4/48.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=566ms FT_p99=566ms Thru=1.0x WER=37.5% CER=21.6%  Msgs=16.0 VRAM=10.4/48GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   37.5 |   21.6 |     566ms |     566ms |     9501ms |    9540ms |    1.0x |  16.0 |    10.4/48GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/sanity_trt_ws_NVIDIA_L40S_20260211_224104.csv
  /home/ubuntu/nemo_hindi/benchmark_results/sanity_trt_ws_NVIDIA_L40S_20260211_224104.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 37.5%

--- VRAM ---
Peak: 10.4 / 48 GB (21.7%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Planning sequential full sweeps**


Sanity checks passed on both endpoints, so I’m now running the full real-time WS sweep on the torch.compile deployment first (`1..500` concurrency). After that completes, I’ll run the same sweep on TRT for a direct scenario match.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run --mode ws --concurrency 1,10,50,100,200,300,400,500 --cooldown 5 --prefix realworld_compile_ws_rt


Output:
Exit code: 0
Wall time: 301.7 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=644ms FT_p99=644ms Thru=0.9x WER=33.3% CER=18.1%  Msgs=16.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=10: Starting 10 streams...
  [WS] c=10: OK=10 Err=0 FT_p50=1386ms FT_p99=1467ms Thru=6.5x WER=20.7% CER=10.1%  Msgs=19.4 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=50: Starting 50 streams...
  [WS] c=50: OK=50 Err=0 FT_p50=983ms FT_p99=1102ms Thru=23.8x WER=33.4% CER=18.8%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=100: Starting 100 streams...
  [WS] c=100: OK=100 Err=0 FT_p50=845ms FT_p99=986ms Thru=46.8x WER=33.1% CER=18.6%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=200: Starting 200 streams...
  [WS] c=200: OK=200 Err=0 FT_p50=954ms FT_p99=2065ms Thru=89.5x WER=33.4% CER=18.9%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=1130ms FT_p99=2763ms Thru=85.2x WER=33.6% CER=19.2%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1291ms FT_p99=2483ms Thru=83.8x WER=33.4% CER=18.8%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1550ms FT_p99=2833ms Thru=107.7x WER=33.5% CER=19.0%  Msgs=20.0 VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   33.3 |   18.1 |     644ms |     644ms |     9641ms |    9682ms |    0.9x |  16.0 |    10.5/85GB
    ws |    10 |   10 |    0 |   20.7 |   10.1 |    1386ms |    1467ms |    12540ms |   17606ms |    6.5x |  19.4 |    10.5/85GB
    ws |    50 |   50 |    0 |   33.4 |   18.8 |     982ms |    1102ms |    11775ms |   24588ms |   23.8x |  20.0 |    10.5/85GB
    ws |   100 |  100 |    0 |   33.1 |   18.6 |     845ms |     986ms |    12368ms |   25026ms |   46.8x |  20.0 |    10.5/85GB
    ws |   200 |  200 |    0 |   33.4 |   18.9 |     954ms |    2065ms |    17329ms |   26143ms |   89.5x |  20.0 |    10.5/85GB
    ws |   300 |  300 |    0 |   33.6 |   19.2 |    1130ms |    2763ms |    24111ms |   41208ms |   85.2x |  19.9 |    10.5/85GB
    ws |   400 |  400 |    0 |   33.4 |   18.8 |    1291ms |    2483ms |    29992ms |   55873ms |   83.8x |  20.0 |    10.5/85GB
    ws |   500 |  500 |    0 |   33.5 |   19.0 |    1550ms |    2834ms |    30710ms |   54307ms |  107.7x |  20.0 |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_ws_rt_NVIDIA_H100_80GB_HBM3_20260211_224615.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 33.3%
    c=  10: WER=20.7% (delta=-12.6%)
    c=  50: WER=33.4% (delta=+0.1%)
    c= 100: WER=33.1% (delta=-0.2%)
    c= 200: WER=33.4% (delta=+0.1%)
    c= 300: WER=33.6% (delta=+0.3%)
    c= 400: WER=33.4% (delta=+0.1%)
    c= 500: WER=33.5% (delta=+0.2%)

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Preparing to run TensorRT test**


The full real-time WS sweep on the torch.compile app completed cleanly through 500 concurrency with no errors. I’m now running the exact same sweep on the TRT app so we can compare quality, tail latency, and throughput under identical workload shape.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode ws --concurrency 1,10,50,100,200,300,400,500 --cooldown 5 --prefix realworld_trt_ws_rt


Output:
Exit code: 0
Wall time: 401.4 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): Yes (1x)

Health: healthy
  GPU: NVIDIA L40S
  VRAM: 10.4/48.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=676ms FT_p99=676ms Thru=0.9x WER=37.5% CER=21.6%  Msgs=16.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=10: Starting 10 streams...
  [WS] c=10: OK=10 Err=0 FT_p50=990ms FT_p99=1016ms Thru=6.4x WER=19.9% CER=9.5%  Msgs=19.5 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=50: Starting 50 streams...
  [WS] c=50: OK=50 Err=0 FT_p50=601ms FT_p99=649ms Thru=23.5x WER=33.7% CER=19.1%  Msgs=20.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=100: Starting 100 streams...
  [WS] c=100: OK=100 Err=0 FT_p50=780ms FT_p99=2351ms Thru=40.8x WER=34.3% CER=19.8%  Msgs=19.9 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=200: Starting 200 streams...
  [WS] c=200: OK=200 Err=0 FT_p50=867ms FT_p99=2761ms Thru=54.0x WER=34.5% CER=19.9%  Msgs=19.8 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=1216ms FT_p99=6671ms Thru=61.3x WER=35.8% CER=21.5%  Msgs=19.4 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1073ms FT_p99=3018ms Thru=59.5x WER=33.7% CER=19.1%  Msgs=20.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1135ms FT_p99=3415ms Thru=62.2x WER=34.7% CER=20.2%  Msgs=19.7 VRAM=10.4/48GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   37.5 |   21.6 |     676ms |     676ms |     9563ms |    9604ms |    0.9x |  16.0 |    10.4/48GB
    ws |    10 |   10 |    0 |   19.9 |    9.5 |     990ms |    1016ms |    12554ms |   17691ms |    6.4x |  19.5 |    10.4/48GB
    ws |    50 |   50 |    0 |   33.7 |   19.1 |     601ms |     649ms |    11684ms |   24934ms |   23.5x |  20.0 |    10.4/48GB
    ws |   100 |  100 |    0 |   34.3 |   19.8 |     780ms |    2351ms |    14055ms |   28703ms |   40.8x |  19.9 |    10.4/48GB
    ws |   200 |  200 |    0 |   34.5 |   19.9 |     867ms |    2761ms |    20107ms |   43325ms |   54.0x |  19.8 |    10.4/48GB
    ws |   300 |  300 |    0 |   35.8 |   21.5 |    1216ms |    6671ms |    25495ms |   57298ms |   61.3x |  19.4 |    10.4/48GB
    ws |   400 |  400 |    0 |   33.7 |   19.1 |    1073ms |    3018ms |    37986ms |   78691ms |   59.5x |  20.0 |    10.4/48GB
    ws |   500 |  500 |    0 |   34.7 |   20.2 |    1135ms |    3415ms |    41819ms |   94039ms |   62.2x |  19.7 |    10.4/48GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_ws_rt_NVIDIA_L40S_20260211_225303.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 37.5%
    c=  10: WER=19.9% (delta=-17.6%)
    c=  50: WER=33.7% (delta=-3.8%)
    c= 100: WER=34.3% (delta=-3.2%)
    c= 200: WER=34.5% (delta=-3.0%)
    c= 300: WER=35.8% (delta=-1.7%)
    c= 400: WER=33.7% (delta=-3.8%)
    c= 500: WER=34.7% (delta=-2.8%)

--- VRAM ---
Peak: 10.4 / 48 GB (21.7%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


Summary: auto


**Planning burst websocket tests**


The real-time WS sweeps are done for both deployments. I’m now running an additional production stress pattern: burst WS (`--no-realtime`) up to 500 concurrency on both apps, which simulates aggressive client buffering/network jitter where chunks arrive faster than wall-clock pace.


**Running compile burst**


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run --mode ws --concurrency 1,10,50,100,200,300,400,500 --no-realtime --cooldown 5 --prefix realworld_compile_ws_burst


Output:
Exit code: 0
Wall time: 265.4 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): No (burst)

Health: healthy
  GPU: NVIDIA H100 80GB HBM3
  VRAM: 10.5/85.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=172ms FT_p99=172ms Thru=8.7x WER=79.2% CER=67.2%  Msgs=9.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=10: Starting 10 streams...
  [WS] c=10: OK=10 Err=0 FT_p50=448ms FT_p99=621ms Thru=16.9x WER=28.1% CER=18.1%  Msgs=17.7 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=50: Starting 50 streams...
  [WS] c=50: OK=50 Err=0 FT_p50=470ms FT_p99=931ms Thru=50.7x WER=33.3% CER=18.9%  Msgs=19.8 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=100: Starting 100 streams...
  [WS] c=100: OK=100 Err=0 FT_p50=694ms FT_p99=1514ms Thru=57.1x WER=33.8% CER=19.1%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=200: Starting 200 streams...
  [WS] c=200: OK=200 Err=0 FT_p50=891ms FT_p99=2244ms Thru=75.7x WER=33.2% CER=18.8%  Msgs=20.0 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=1271ms FT_p99=2436ms Thru=83.7x WER=33.5% CER=19.0%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=1386ms FT_p99=2755ms Thru=91.2x WER=33.6% CER=19.2%  Msgs=19.9 VRAM=10.5/85GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1344ms FT_p99=2754ms Thru=107.8x WER=33.7% CER=19.2%  Msgs=19.9 VRAM=10.5/85GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   79.2 |   67.2 |     172ms |     172ms |     1008ms |    1048ms |    8.7x |   9.0 |    10.5/85GB
    ws |    10 |   10 |    0 |   28.1 |   18.1 |     448ms |     621ms |     4736ms |    6721ms |   16.9x |  17.7 |    10.5/85GB
    ws |    50 |   50 |    0 |   33.3 |   18.9 |     470ms |     931ms |     8335ms |   11542ms |   50.7x |  19.8 |    10.5/85GB
    ws |   100 |  100 |    0 |   33.8 |   19.1 |     694ms |    1514ms |    13682ms |   20477ms |   57.1x |  19.9 |    10.5/85GB
    ws |   200 |  200 |    0 |   33.2 |   18.8 |     891ms |    2244ms |    19740ms |   30917ms |   75.7x |  20.0 |    10.5/85GB
    ws |   300 |  300 |    0 |   33.5 |   19.0 |    1270ms |    2436ms |    26243ms |   41946ms |   83.7x |  19.9 |    10.5/85GB
    ws |   400 |  400 |    0 |   33.6 |   19.2 |    1386ms |    2755ms |    29672ms |   51329ms |   91.2x |  19.9 |    10.5/85GB
    ws |   500 |  500 |    0 |   33.7 |   19.2 |    1344ms |    2754ms |    33883ms |   54301ms |  107.8x |  19.9 |    10.5/85GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_compile_ws_burst_NVIDIA_H100_80GB_HBM3_20260211_225739.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 79.2%
    c=  10: WER=28.1% (delta=-51.1%)
    c=  50: WER=33.3% (delta=-45.9%)
    c= 100: WER=33.8% (delta=-45.4%)
    c= 200: WER=33.2% (delta=-46.0%)
    c= 300: WER=33.5% (delta=-45.7%)
    c= 400: WER=33.6% (delta=-45.6%)
    c= 500: WER=33.7% (delta=-45.5%)

--- VRAM ---
Peak: 10.5 / 85 GB (12.4%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Running trt burst**


Compile burst-mode is complete and stable up to 500 with zero request failures. I’m running the matching burst sweep on TRT now.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run --mode ws --concurrency 1,10,50,100,200,300,400,500 --no-realtime --cooldown 5 --prefix realworld_trt_ws_burst


Output:
Exit code: 0
Wall time: 299.7 seconds
Output:
Loaded 50 audio samples (585.1s total, 50 with ground truth)

================================================================================
Modal Nemotron ASR Benchmark (with WER scoring)
================================================================================
URL: https://mayaresearch--hindi-nemotron-asr-1-nemotronasr-webapp.modal.run
Modes: ws
Concurrency levels: [1, 10, 50, 100, 200, 300, 400, 500]
Audio samples: 50
Realtime pace (WS): No (burst)

Health: healthy
  GPU: NVIDIA L40S
  VRAM: 10.4/48.0 GB
  Active streams: 0

================================================================================
Starting benchmark...
================================================================================

--- WS MODE ---
  [WS] c=1: Starting 1 streams...
  [WS] c=1: OK=1 Err=0 FT_p50=138ms FT_p99=138ms Thru=11.5x WER=79.2% CER=63.8%  Msgs=10.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=10: Starting 10 streams...
  [WS] c=10: OK=10 Err=0 FT_p50=256ms FT_p99=286ms Thru=29.5x WER=20.7% CER=10.6%  Msgs=19.2 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=50: Starting 50 streams...
  [WS] c=50: OK=50 Err=0 FT_p50=432ms FT_p99=1564ms Thru=56.2x WER=33.7% CER=19.2%  Msgs=19.9 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=100: Starting 100 streams...
  [WS] c=100: OK=100 Err=0 FT_p50=624ms FT_p99=1363ms Thru=65.3x WER=34.0% CER=19.4%  Msgs=19.9 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=200: Starting 200 streams...
  [WS] c=200: OK=200 Err=0 FT_p50=803ms FT_p99=1474ms Thru=73.2x WER=33.7% CER=19.1%  Msgs=20.0 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=300: Starting 300 streams...
  [WS] c=300: OK=300 Err=0 FT_p50=973ms FT_p99=1799ms Thru=73.5x WER=33.7% CER=19.1%  Msgs=19.9 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=400: Starting 400 streams...
  [WS] c=400: OK=400 Err=0 FT_p50=952ms FT_p99=4096ms Thru=75.0x WER=34.8% CER=20.3%  Msgs=19.6 VRAM=10.4/48GB
  Cooling down 5s...
  [WS] c=500: Starting 500 streams...
  [WS] c=500: OK=500 Err=0 FT_p50=1059ms FT_p99=5308ms Thru=75.7x WER=37.3% CER=23.1%  Msgs=19.1 VRAM=10.4/48GB

============================================================================================================================================
  Mode |  Conc |   OK |  Err |   WER% |   CER% |   FT p50 |   FT p99 | Total p50 |     Wall |    Thru |  Msgs |         VRAM
--------------------------------------------------------------------------------------------------------------------------------------------
    ws |     1 |    1 |    0 |   79.2 |   63.8 |     138ms |     138ms |      754ms |     793ms |   11.5x |  10.0 |    10.4/48GB
    ws |    10 |   10 |    0 |   20.7 |   10.6 |     256ms |     286ms |     3126ms |    3848ms |   29.5x |  19.2 |    10.4/48GB
    ws |    50 |   50 |    0 |   33.7 |   19.2 |     432ms |    1564ms |     7896ms |   10417ms |   56.2x |  19.9 |    10.4/48GB
    ws |   100 |  100 |    0 |   34.0 |   19.4 |     624ms |    1363ms |    12138ms |   17918ms |   65.3x |  19.9 |    10.4/48GB
    ws |   200 |  200 |    0 |   33.7 |   19.1 |     803ms |    1474ms |    20056ms |   31988ms |   73.2x |  20.0 |    10.4/48GB
    ws |   300 |  300 |    0 |   33.7 |   19.1 |     974ms |    1798ms |    29544ms |   47784ms |   73.5x |  19.9 |    10.4/48GB
    ws |   400 |  400 |    0 |   34.8 |   20.3 |     952ms |    4096ms |    37723ms |   62395ms |   75.0x |  19.6 |    10.4/48GB
    ws |   500 |  500 |    0 |   37.3 |   23.1 |    1059ms |    5308ms |    43716ms |   77341ms |   75.7x |  19.1 |    10.4/48GB
============================================================================================================================================

Results saved to:
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.csv
  /home/ubuntu/nemo_hindi/benchmark_results/realworld_trt_ws_burst_NVIDIA_L40S_20260211_230244.json

--- QUALITY DEGRADATION CURVE ---
  WS baseline WER (c=1): 79.2%
    c=  10: WER=20.7% (delta=-58.5%)
    c=  50: WER=33.7% (delta=-45.5%)
    c= 100: WER=34.0% (delta=-45.2%)
    c= 200: WER=33.7% (delta=-45.5%)
    c= 300: WER=33.7% (delta=-45.5%)
    c= 400: WER=34.8% (delta=-44.4%)
    c= 500: WER=37.3% (delta=-41.9%)

--- VRAM ---
Peak: 10.4 / 48 GB (21.7%)
/home/ubuntu/nemo_hindi/nemotron_asr/bench_modal.py:185: DeprecationWarning: parameter 'timeout' of type 'float' is deprecated, please use 'timeout=ClientWSTimeout(ws_close=...)'
  ws = await session.ws_connect(ws_url, timeout=WS_CONNECT_TIMEOUT)


**Planning HTTP stress test**


Burst WS sweeps are complete for both. I’m now running HTTP `/transcribe` stress sweeps to 500 on each deployment to capture batch-request behavior in addition to streaming behavior.


source venv/bin/activate && python nemotron_asr/bench_modal.py --url https://mayaresearch--hindi-nemotron-asr-nemotronasr-webapp.modal.run --mode http --concurrency 1,10,50,100,200,300,400,500 --cooldown 5 --prefix realworld_compile_http