<permissions instructions>
Filesystem sandboxing defines which files can be read or written. `sandbox_mode` is `danger-full-access`: No filesystem sandboxing - all commands are permitted. Network access is enabled.
 Approvals are your mechanism to get user consent to run shell commands without the sandbox. `approval_policy` is `never`: This is a non-interactive mode where you may NEVER ask the user for approval to run commands. Instead, you must always persist and work around constraints to solve the task for the user. You MUST do your utmost best to finish the task and validate your work before yielding. If this mode is paired with `danger-full-access`, take advantage of it to deliver the best outcome for the user. Further, in this mode, your default testing philosophy is overridden: Even if you don't see local patterns for testing, you may add tests and scripts to validate your work. Just remove them before yielding.
</permissions instructions>

# AGENTS.md instructions for /home/ubuntu/nemo_hindi

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/nemo_hindi</cwd>
  <shell>bash</shell>
</environment_context>

<collaboration_mode># Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.
</collaboration_mode>

Here is what i asked my another coding agent.

finish revamp with clean architecture mode implementation, then deploy to vast(all keys are in .env) serverless workers. then do stress testing simulating like actual customers to understand the working production server behaviour mocking and capture all metrics, so we can set scaling laws and move our production OpenAI transcription paid service to self hosten custom finetuned hindi model and serve any customer base with proper scaling laws set. assuming somewhere at 500 concurrent batching requests per GPU of 80GB GPU. Also while deploying check if region based deployment is available and check for closer to india or asia or anything closer. Also while implementation or planning, plan for both HTTP based and websocket based requests handlers. If we have vCPUs that take requests and sends batches of chunks to GPU and gets data back, or triton server, or whatever man. i just need high throughput mode with both HTTP streaming and websocket streaming.

Read the vast's serverless docs, requirements
Plan the architecture that is failproof and high throughput mode with quality considered too.
start working on it
test it, deploy it to serverless
smoke test
stress test it and capture all major metrics(all) -> simulation tests that will be actual customers usage based
then we'll discuss on scaling laws.

and after a while of trying it failed but got somewhere. and below is a handover summary from that agent. Can you now take over and finish the proper deployment, testing, stress testing, and capture metrics. below is it's summary. Also, all secrets are in .env FYI. docs are also in this repo for vast's serverless stack. understand the flow. and see the current implementation, make changes, or revamp as needed.

## Summary

### What We Started With
- A **fine-tuned Hindi ASR model** (Nemotron-0.6B, 8.8% WER) deployed on Modal
- Prior benchmarks showing **A100-SXM4-80GB handles 50-100 concurrent streams** in the safe zone
- Existing code in `nemotron_asr/` — a Modal-coupled WebSocket streaming server with event-driven batching, VAD, and centralized inference loop
- The decision to move to **Vast.ai serverless** for production (cheaper, direct TCP, autoscaling)

### What We Want To Do
1. Build a **production-ready ASR service** with both HTTP and WebSocket endpoints
2. Deploy it on **Vast.ai serverless** with autoscaling (not bare metal instances)
3. Run **stress tests simulating real customers** to capture all metrics
4. Establish **scaling laws** so we can replace paid OpenAI transcription with our self-hosted model

### What We Built
- `production/server.py` — Standalone FastAPI server (HTTP batch, HTTP SSE streaming, WebSocket streaming)
- `production/worker.py` — Vast.ai PyWorker config (metrics reporting, benchmarks, autoscaling integration)
- `production/onstart.sh` — Startup script that launches both model server and PyWorker
- `production/Dockerfile.slim` — Slim image extending existing v5 image
- `production/deploy.py` — Deployment automation (create/destroy endpoint+workergroup)
- `production/test_smoke.py` — Smoke test (health, HTTP, WebSocket)
- `production/test_stress.py` — Full stress test with percentile metrics, CSV/JSON output
- PyWorker repo pushed to GitHub, Docker image on Docker Hub

### Vast.ai Serverless Deployment (Live)
- Endpoint: `hindi-asr-prod` (id=11967)
- Workergroup: 16773 (template hash `3ac272ae9dc0ae58d105a1d88145a325`)
- Worker provisioned and managed by the serverless engine

### Issues We Hit (and Fixed)

| Issue | Root Cause | Fix |
|:---|:---|:---|
| Created bare metal instance instead of serverless | My mistake — didn't follow docs | Destroyed instance, used `create endpoint` + `create workergroup` |
| `PYWORKER_REPO` env var empty on instance | Account-level env vars don't auto-propagate; Vast.ai doesn't auto-bootstrap PyWorker for custom images | Handle PyWorker clone+run in our own `onstart.sh` |
| `python-dateutil` downgraded to 2.6.1 | `vastai` pip package pulled old dateutil, broke matplotlib/NeMo | Install deps BEFORE starting server; pin `python-dateutil>=2.8.2` |
| `cannot import name 'Worker' from 'vastai'` | Wrong pip package — `vastai` is CLI tool, `vastai_sdk` has the Worker class | Changed requirements.txt to `vastai_sdk` |
| `KeyError: 'WORKER_PORT'` | `vastai_sdk` Worker expects env vars set by Vast.ai infra, which doesn't auto-set them for custom images | Manually set `WORKER_PORT=8000`, `VAST_TCP_PORT_8000` in onstart.sh; model server on 18000, PyWorker on 8000 |
| `Object of type bytes is not JSON serializable` | Benchmark generator returned file tuple with bytes; PyWorker sends benchmarks as JSON | Changed to base64-encoded audio, new `/v1/audio/transcriptions/json` endpoint |
| `'CacheAwareRNNTStreamingPipeline' has no attribute 'run'` | Streaming pipeline doesn't have `.run()` method (that's on the batch pipeline) | **Last fix** — replaced with `transcribe_pcm_bytes_streaming()` that uses the streaming session API |

### Where We Are Right Now
- Worker is **recycling** with the latest fix (streaming inference for benchmarks)
- Waiting to see if the PyWorker benchmark passes and the worker transitions to **"running"** (ready) status
- Once ready: smoke test, stress test, metrics capture

### What's Left After Worker Goes Ready
1. Smoke test (HTTP + WebSocket)
2. Stress test sweep (c=1,5,10,25,50,75,100) capturing all latency/throughput metrics
3. Analyze results, set scaling laws
4. Update README with production benchmarks

and it replied with

I made the server/pyworker JSON-compatible, rebuilt/pushed the image, updated the Vast template/workergroup, and wired the tests to run against `/route` with auth. The current serverless worker is still provisioning (status `loading`), so full stress runs are pending.

What changed (code + image)
- JSON+multipart compatibility for batch + SSE endpoints in `production/server.py` so PyWorker can call `/v1/audio/transcriptions` and `/v1/audio/transcriptions/stream` with JSON payloads.
- PyWorker now uses base64 JSON payloads, adds `/v1/audio/transcriptions/stream`, and uses a better workload estimator in `production/worker.py`.
- Onstart no longer clones a repo; it runs the bundled worker and only installs deps if missing in `production/onstart.sh`.
- Slim/full Dockerfiles now include `worker.py` and `vastai_sdk` deps in `production/Dockerfile.slim` and `production/Dockerfile`.
- Deployment + tests now understand endpoint auth via `/route` in `production/deploy.py`, `production/test_smoke.py`, and `production/test_stress.py`.

Image
- Built and pushed `bharathkumar192/hindi-nemotron-asr:v6-production` with the updated server/pyworker.

Current Vast state
- Endpoint: `hindi-asr-prod` (id `11967`)
- Workergroup: `16780` (template hash `3fc7205a26c3bd08fbd28f28ccca252a`)
- Template (id `341822`) now uses:
  - image `bharathkumar192/hindi-nemotron-asr:v6-production`
  - env `-p 8000:8000`
  - filters: `gpu_ram>=40 num_gpus=1 reliability>0.95 direct_port_count>=1 disk_space>=30 inet_down>200`
- Worker currently: `loading` (see `python3 production/deploy.py workers`).

Smoke results (last completed run on previous worker)
- Health: PASS
- HTTP batch: PASS (~1.1s for 3s test audio)
- SSE stream: PASS (~1.0s, 2 events)
- WebSocket: FAIL (timeout; no direct port mapping for 18000)

Why WebSocket failed in serverless
- The current template exposes only port 8000 (PyWorker). `/ws` lives on the model server port (18000), which is not mapped, so WS connections time out.
- To enable WS via serverless, the template must include `-p 18000:18000` and the search filters must require `direct_port_count>=2`. This is doable, but it caused unstable provisioning in this round.

How to proceed once the worker is idle
1) Smoke (serverless)
```
python3 production/test_smoke.py --endpoint hindi-asr-prod --route-cost 10
```

2) Stress (batch + SSE)
```
python3 production/test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10,25,50
python3 production/test_stress.py --endpoint hindi-asr-prod --mode sse  --levels 1,5,10,25,50
```
Outputs go to `benchmark_results/*.csv` and `benchmark_results/*.json`.

Region pinning (closer to India / Asia)
- Vast supports `geolocation` in search params. Example:
  - `geolocation in [IN,SG,JP,HK,AE]`
- You can apply this by updating the template search params:
```
vastai update template 3fc7205a26c3bd08fbd28f28ccca252a \
  --search_params "gpu_ram>=40 num_gpus=1 reliability>0.95 rented=False geolocation in [IN,SG,JP,HK,AE] direct_port_count>=1 disk_space>=30 inet_down>200"
```

If you want me to push WS support through serverless next, I’ll:
1) Update the template env to `-p 8000:8000 -p 18000:18000`.
2) Enforce `direct_port_count>=2`.
3) Recreate workergroup and re-run smoke/stress with `--ws-port 18000`.

Suggested next steps (pick one)
1) Wait for the current worker to hit `idle`, then I’ll run smoke + stress and report metrics.
2) Proceed with WebSocket-capable template/workergroup changes and retry provisioning.
3) Roll back to a fully open search (no geolocation) to get a worker fast, then collect metrics immediately.

and then i asked, check status

And it replied with
Status right now:
- Workers: 1
- Instance ID: `31133036`
- Status: `creating` (not ready yet)

Want me to keep watching until it reaches `idle` and then run smoke/stress?

and i asked,

what are the next steps. for our goal. here's what i asked.

Read the vast's serverless docs, requirements
Plan the architecture that is failproof and high throughput mode with quality considered too.
start working on it
test it, deploy it to serverless
smoke test
stress test it and capture all major metrics(all) -> simulation tests that will be actual customers usage based
then we'll discuss on scaling laws.

and it replied with

Here are the concrete next steps toward your stated goal, in order. I’ve kept them actionable and tied to what’s already in the repo.

Current state
- Serverless worker is still `creating` (not ready yet). We can’t run smoke/stress until it’s `idle`.

Next steps (exact order)
1) Wait for worker to become ready
   - I’ll keep polling `python3 production/deploy.py workers` until status is `idle`.

2) Smoke test (serverless)
   - Command:
     ```
     python3 production/test_smoke.py --endpoint hindi-asr-prod --route-cost 10
     ```
   - Confirms: auth flow, batch, SSE.
   - WebSocket is not reachable unless we expose port 18000 (see step 6).

3) Stress test (customer‑like load)
   - HTTP batch:
     ```
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode http --levels 1,5,10,25,50
     ```
   - SSE streaming:
     ```
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode sse --levels 1,5,10,25,50
     ```
   - Outputs go to `benchmark_results/*.csv` and `benchmark_results/*.json`.

4) Capture metrics and summarize
   - I’ll pull the CSV/JSON results and summarize p50/p95/p99 latency, throughput, error rate, and worker load snapshots.

5) Scaling laws draft
   - Use the stress results to propose concurrency limits per GPU and utilization targets.

6) WebSocket support (optional, but required if you want WS in prod)
   - We must expose port 18000 in the template and require `direct_port_count>=2`.
   - That requires template + workergroup update + new worker provisioning.
   - Once enabled, smoke/stress will include:
     ```
     python3 production/test_smoke.py --endpoint hindi-asr-prod --ws-port 18000
     python3 production/test_stress.py --endpoint hindi-asr-prod --mode ws --ws-port 18000 --levels 1,5,10,25,50
     ```

7) Region preference (Asia/India)
- I can set `geolocation in [IN,SG,JP,HK,AE]` in the template search params, but that narrows supply and can slow provisioning. We can toggle it once baseline metrics are done.

What I need from you
- Confirm whether you want WebSocket enabled now (requires template/workergroup change + new worker provisioning), or do HTTP+SSE first to get metrics quickly.

If you say “go ahead,” I’ll:
- keep polling until the worker is idle,
- run smoke and stress for HTTP+SSE immediately,
- then report metrics and draft scaling‑law estimates.

and i asked go, we can test teh websockets later.

and it said back,
Worker is up and idle now. The serverless template was failing to start because the template’s `onstart` was set to `"bash /app/onstart.sh"` (exec-form string). I updated the template to use `/app/onstart.sh` and re-pointed the workergroup to the new template hash. After that, the worker loaded and stayed healthy.

Smoke + stress tests (HTTP + SSE) are done (websockets skipped as requested). All runs were 0 errors.

Key results (10s audio per stream):
- HTTP p50 total latency: 0.7s (c=1), 1.1s (c=5), 1.9s (c=10), 3.7s (c=25), 7.0s (c=50)
- SSE p50 total latency: 0.64s (c=1), 1.48s (c=5), 2.58s (c=10), 5.87s (c=25), 10.92s (c=50)
- Throughput: ~38–42x realtime at c=50 for both HTTP and SSE

Artifacts saved:
- `benchmark_results/stress_results_20260209_195631.csv`
- `benchmark_results/stress_results_20260209_195631.json`
- `benchmark_results/stress_results_20260209_195730.csv`
- `benchmark_results/stress_results_20260209_195730.json`

Next steps (pick what you want me to do):
1) Add “full metrics” capture by exposing `/metrics` via PyWorker (or mapping port 18000) so we can log GPU ms, VRAM, batch stats during stress tests.
2) WebSockets later: update template to `-p 18000:18000` and add `direct_port_count>=2`, then run `test_smoke.py` + `test_stress.py --mode ws`.
3) Region tuning: add `geolocation in [IN,SG,JP,HK,AE]` (or your preferred list) to search filters and re-run workergroup; we’ll verify offers exist so it doesn’t churn.
4) Scaling laws: use the new JSON outputs to fit throughput/latency curves and pick target concurrency per GPU.

Tell me which option to execute next (or ask for a specific plot/summary from the JSON files).

and i asked,  yes. let's do a thorough full metrics and stress testing first now. let's finish them now.

and it replied with,

Full‑metrics instrumentation and stress tests are done. I updated the server/worker to expose `/metrics` through PyWorker, fixed the CUDA VRAM reporting (`total_memory`), rebuilt the image, redeployed, and ran full HTTP + SSE sweeps with metrics captured per level.

Where the metrics live (latest runs):
- HTTP: `benchmark_results/stress_results_20260209_202433.csv` and `benchmark_results/stress_results_20260209_202433.json`
- SSE: `benchmark_results/stress_results_20260209_202718.csv` and `benchmark_results/stress_results_20260209_202718.json`

Key results (RTX PRO 6000 WS, 10s audio per stream):
- HTTP p50 total latency: 1.48s (c=1), 7.04s (c=25), 13.00s (c=50), 24.25s (c=100)
- HTTP throughput: ~20–21x realtime at c=50–100; c=100 had 2/100 route failures (“All worker queues full”)
- SSE p50 total latency: 1.51s (c=1), 11.02s (c=25), 20.21s (c=50), 34.65s (c=100)
- SSE throughput: ~22–23x realtime at c=50–100; c=100 had 2/100 route failures
- VRAM: ~10.43GB used, ~12.97GB reserved, 102GB total (now correctly reported)
- `/metrics` GPU p50/p99 stays 0 for HTTP/SSE because those fields are tied to the WS batching loop; the JSON still captures all other server metrics + VRAM + HTTP counts

What changed (code + deploy):
- `/metrics` now supports POST and returns CUDA fields (`cuda_available`, `device_count`, `gpu_name`, VRAM used/reserved/total) from `production/server.py`
- PyWorker now proxies `/metrics` in `production/worker.py`
- `production/test_stress.py` now fetches metrics via serverless `/route`
- Deployed image: `bharathkumar192/hindi-nemotron-asr:v9-production`
- Workergroup template hash: `118364eef727b32735fa02ee2908b7a1`
- Current worker: `31135755` (idle and healthy)

If you want “GPU time per request” for HTTP/SSE (not just VRAM), I can add per‑request timing in the HTTP/SSE handlers and expose percentiles in `/metrics`.

Next steps (pick one):
1) Run longer‑duration stress (30s/60s) to simulate real customer sessions and capture drift.
2) Add per‑request GPU timing for HTTP/SSE and re‑run the sweeps.
3) Proceed to scaling‑law fitting using the new JSON outputs.

Can you now do a thorough stress testing ensurimg we capture max metrics while doing so and on the way we should probaly ensure the WER also stays sane. and check for any issues where the current behaviour might fail in our production-ready application. We can work on websockets once the above is finished. first finish end to end metrics testing and auto-scaling rules are set and everything is done with http sse based worker.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
so we've created a RTX 6000 pro-96GB gpu machine. and you're saying only 75 concurrent requests per GPU ? is that accurate ? cause there is a blog of NVIDIA nemotron asr team benchmarking the H100 GPU with 80GB and they achieved like 500 concurrent requests with minimal drift in latency. not sure if we are doing the right way of inference or throughput management. Can you double check, or explain what's happening. Here's the link to the blog and it's content. Or you can also read the contents by teh nemotron's ASR repo in huggingface.

Hugging Face's logo
Hugging Face
Models
Datasets
Spaces
Community
Docs
Pricing

nvidia
/
nemotron-speech-streaming-en-0.6b

like
468

Follow
NVIDIA
49.7k
Automatic Speech Recognition
NeMo
PyTorch

19 datasets
speech-recognition
cache-aware ASR
streaming-asr
speech
audio
FastConformer
RNNT
Parakeet
ASR
NeMo
Eval Results (legacy)

arxiv:
2312.17279

arxiv:
2305.05084

License:
nvidia-open-model-license
Model card
Files and versions
xet
Community
11
nemotron-speech-streaming-en-0.6b
/
README.md

kunaldhawan's picture
kunaldhawan
Add links to Multitalker Parakeet Streaming and Parakeet Realtime EOU
c0acae9
verified
9 days ago
preview
code
|
raw

Copy download link
history
blame
contribute
delete

16.5 kB
---
license: other
license_name: nvidia-open-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
library_name: nemo
datasets:
- nvidia/Granary
- YTC
- Yodas2
- LibriLight
- librispeech_asr
- fisher_corpus
- Switchboard-1
- WSJ-0
- WSJ-1
- National-Singapore-Corpus-Part-1
- National-Singapore-Corpus-Part-6
- vctk
- voxpopuli
- europarl
- multilingual_librispeech
- fleurs
- mozilla-foundation/common_voice_8_0
- MLCommons/peoples_speech
- google/speech_commands
thumbnail: null
tags:
- speech-recognition
- cache-aware ASR
- automatic-speech-recognition
- streaming-asr
- speech
- audio
- FastConformer
- RNNT
- Parakeet
- ASR
- pytorch
- NeMo
widget:
- example_title: Librispeech sample 1
  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
- example_title: Librispeech sample 2
  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
model-index:
- name: nemotron-speech-streaming-en-0.6b
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: AMI
      type: ami
      config: ihm
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.58
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Earnings22
      type: earnings22
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 12.48
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Gigaspeech
      type: gigaspeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.45
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-clean
      type: librispeech_asr
      config: clean
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.31
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-other
      type: librispeech_asr
      config: other
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.75
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: SPGI Speech
      type: spgispeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.62
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: TEDLIUM
      type: tedlium
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.5
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: VoxPopuli
      type: voxpopuli
      config: en
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 7.57
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---

# Nemotron Speech ASR

[![Model architecture](https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-lightgrey#model-badge)](#model-architecture)
| [![Model size](https://img.shields.io/badge/Params-600M-lightgrey#model-badge)](#model-architecture)
| [![Language](https://img.shields.io/badge/Language-English-lightgrey#model-badge)](#datasets)
Nemotron-Speech-Streaming-En-0.6b is the first unified model in the Nemotron Speech family, engineered to deliver high-quality English transcription across both low-latency streaming and high-throughput batch workloads. The model natively supports punctuation and capitalization and offers runtime flexibility with configurable chunk sizes, including 80ms, 160ms, 560ms, and 1120ms.
Why Choose nvidia/nemotron-speech-streaming-en-0.6b?
* **Native Streaming Architecture:** Cache-aware design enables efficient processing of continuous audio streams, designed and optimized for low-latency voice agent applications interaction.
* **Improved Operational Efficiency:** Delivers superior throughput compared to traditional buffered streaming approaches. This allows for a higher number of parallel streams within the same GPU memory constraints, directly reducing operational costs for production environments.
* **Dynamic Runtime Flexibility:** Enables you to choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
* **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text

    Nemotron-speech-streaming-en-0.6b allows users to choose the optimal operating point on the latency-accuracy pareto curve at inference time, without requiring any re-training. Further, the cache-aware streaming mechanism scales much better than buffered streaming approaches, consistently outperforming production models like  parakeet-ctc-1_1b-asr  across chunk sizes.

This model consists of a cache-aware streaming 🦜 Parakeet (FastConformer) encoder with an RNN-T decoder. It is designed for real-time speech-to-text applications where low latency is critical, such as voice assistants, live captioning, and conversational AI systems. Unlike traditional "buffered" streaming, the cache-aware architecture enables continuous transcription by processing only new audio chunks while reusing cached encoder context. This significantly improves computational efficiency and minimizes end-to-end delay without sacrificing accuracy.
🗣 **Experience `Nemotron-Speech-Streaming-En-0.6b` in action** here: [https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b)

This model is ready for commercial/non-commercial use.

Read more about the model in the [dev blog](https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents) and check out the [paper](https://arxiv.org/abs/2312.17279).

## Explore more from NVIDIA:
For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at [developer.nvidia.com](https://developer.nvidia.com/).
Join the community to access tools, support, and resources to accelerate your development with NVIDIA's NeMo, Riva, NIM, and foundation models.

What is [Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/)?

NVIDIA Developer [Nemotron](https://developer.nvidia.com/nemotron)

[NVIDIA Riva Speech](https://developer.nvidia.com/riva?sortBy=developer_learning_library%2Fsort%2Ffeatured_in.riva%3Adesc%2Ctitle%3Aasc#demos)

[NeMo Documentation](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/models.html)

Also, check out the following NVIDIA speech models that extend the capabilities of Nemotron-Speech-Streaming:
* Multitalker Parakeet Streaming - https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1
* Parakeet Realtime EOU - https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1
## Access Model Inference and Examples:
* Deploy the Nemotron Speech ASR endpoint on [Modal](https://github.com/modal-projects/modal-nvidia-asr)
* Build local voice agent using [Daily’s framework](https://github.com/pipecat-ai/nemotron-january-2026)
## Model Architecture
**Architecture Type:** FastConformer-CacheAware-RNNT
The model is based on the Cache-Aware [1] FastConformer [2] architecture with 24 encoder layers and an RNNT (Recurrent Neural Network Transducer) decoder. The cache-aware streaming design enables efficient processing of audio in chunks while maintaining context from previous frames. Unlike buffered inference, this model maintains caches for all encoder self-attention and convolution layers. This enables reuse of hidden states at every streaming step, where cached activations eliminate redundant computations. As a result, there are no overlapping computations; each processed frame is strictly non-overlapping.
The caching schema of self-attention and convolution layers for consecutive chunks is as follows. For more details, please refer to [1].

**Network Architecture:**
- Encoder: Cache-Aware FastConformer with 24 layers
- Decoder: RNNT (Recurrent Neural Network Transducer)
- Parameters: 600M
## NVIDIA NeMo
To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo)[4]. We recommend you install it after you've installed Cython and latest PyTorch version.
```bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]
```
## How to Use this Model
The model is available for use in the NeMo Framework, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
### Loading the Model
```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/nemotron-speech-streaming-en-0.6b")
```
### Streaming Inference
You can use the cache-aware streaming inference script from NeMo - [NeMo/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py)
``` bash
cd NeMo
python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
    model_path=<model_path> \
    dataset_manifest=<dataset_manifest> \
    batch_size=<batch_size> \
    att_context_size="[70,13]" \ #set the second value to the desired right context from {0,1,6,13}
    output_path=<output_folder>
```

You can also run streaming inference through the pipeline method, which uses [NeMo/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml) configuration file to build end‑to‑end workflows with punctuation and capitalization (PnC), inverse text normalization (ITN), and translation support.

```python
from nemo.collections.asr.inference.factory.pipeline_builder import PipelineBuilder
from omegaconf import OmegaConf
# Path to the cache aware config file downloaded from above link
cfg_path = 'cache_aware_rnnt.yaml'
cfg = OmegaConf.load(cfg_path)
# Pass the paths of all the audio files for inferencing
audios = ['/path/to/your/audio.wav']
# Create the pipeline object and run inference
pipeline = PipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audios)
# Print the output
for entry in output:
  print(entry['text'])
```

_______
### Setting up Streaming Configuration
Latency is defined by the `att_context_size` param, where  att_context_size = `{num_frames_left_context, num_frame_right_context}`, all measured in **80ms frames**:
* [70, 0]: Chunk size = 1 (1 × 80ms = 0.08s)
* [70, 1]: Chunk size = 2 (2 × 80ms = 0.16s)
* [70, 6]: Chunk size = 7 (7 × 80ms = 0.56s)
* [70, 13]: Chunk size = 14 (14 × 80ms = 1.12s)
Here, chunk size = current frame + right context; each chunk is processed in non-overlapping fashion.
### Input
This model accepts single-channel (mono) audio sampled at 16,000 Hz. At least 80ms duration is required.
### Output
The model outputs English text transcriptions with punctuation and capitalization. The output text might be empty if input audio doesn't contain any speech.
## Datasets
### Training Datasets
The majority of the training data comes from the English portion of the Granary dataset [3]:
- YouTube-Commons (YTC) (109.5k hours)
- YODAS2 (102k hours)
- Mosel (14k hours)
- LibriLight (49.5k hours)
In addition, the following datasets were used:
- Librispeech 960 hours
- Fisher Corpus
- Switchboard-1 Dataset
- WSJ-0 and WSJ-1
- National Speech Corpus (Part 1, Part 6)
- VCTK
- VoxPopuli (EN)
- Europarl-ASR (EN)
- Multilingual Librispeech (MLS EN)
- Mozilla Common Voice (v11.0)
- Mozilla Common Voice (v7.0)
- Mozilla Common Voice (v4.0)
- People Speech
- AMI
**Data Modality:** Audio and text
**Audio Training Data Size:** 285k hours
**Data Collection Method:** Human - All audios are human recorded
**Labeling Method:** Hybrid (Human, Synthetic) - Some transcripts are generated by ASR models, while some are manually labeled
### Evaluation Datasets
The model was evaluated on the HuggingFace ASR Leaderboard datasets:
- AMI
- Earnings22
- Gigaspeech
- LibriSpeech test-clean
- LibriSpeech test-other
- SPGI Speech
- TEDLIUM
- VoxPopuli
## Performance
## ASR Performance (w/o PnC)
ASR performance is measured using the Word Error Rate (WER). Both ground-truth and predicted texts are processed using [whisper-normalizer](https://pypi.org/project/whisper-normalizer/) version 0.1.12.
The following tables show the WER on the [HuggingFace OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) datasets:
### Word Error Rate (WER) for chunk size of 1.12s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.16** | 11.58 | 12.48 | 11.45 | 2.31 | 4.75 | 2.62 | 4.50 | 7.57 |
### WER for chunk size of 0.56s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.22** | 11.69 | 12.61 | 11.43 | 2.40 | 4.97 | 2.64 | 4.46 | 7.59 |
### WER for chunk size of 0.16s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.84** | 13.88 | 13.61 | 12.12 | 2.43 | 5.33 | 2.82 | 4.80 | 7.72 |
### WER for chunk size of 0.08s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **8.528** | 16.05 | 14.6 | 12.92 | 2.55 | 5.79 | 3.01 | 5.07 | 8.23 |
## Software Integration
**Runtime Engine:** NeMo 25.11
**Supported Hardware Microarchitecture Compatibility:**
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Hopper
- NVIDIA Volta
**Test Hardware:**
- NVIDIA V100
- NVIDIA A100
- NVIDIA A6000
- DGX Spark
**Preferred/Supported Operating System(s):** Linux
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
## References
[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)

link to the blog for benchmarks - https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents

Scaling Real-Time Voice Agents with Cache-Aware Streaming ASR
Community Article
Published January 5, 2026
Kunal Dhawan's avatar
Kunal Dhawan
kunaldhawan

Follow
nvidia
Adi- margolin's avatar
Adi- margolin
Amargolin

Follow
nvidia
Gordana Neskovic's avatar
Gordana Neskovic
nesgor

Follow
nvidia
Maryam Motamedi's avatar
Maryam Motamedi
maryameee

Follow
nvidia
Yasmina Benkhoui's avatar
Yasmina Benkhoui
YasminaBenk

Follow
nvidia

In voice AI interactions, we have long been trapped by a familiar trade-off: speed versus accuracy. Traditionally, real-time Automatic Speech Recognition (ASR) relies on buffered inference; a workaround where the system repeatedly re-processes overlapping audio windows to maintain context. It is the computational equivalent of re-reading the last few pages of a book every time you turn the page.
NVIDIA Nemotron Speech ASR, a new, open model built specifically for real-time voice agents, breaks this cycle. Built on the FastConformer architecture with 8x downsampling, it introduces cache-aware technology to process only new audio "deltas." By reusing past computations rather than re-calculating them, it achieves up to 3x higher efficiency than traditional buffered systems.

In this post, we’ll explore how cache-aware architecture redefines the limits of real-time voice agents, and show real-world results from Daily and Modal for high-concurrency, low-latency voice agent workloads.

The Challenge: Why Streaming ASR Breaks at Scale
Most systems labeled as “streaming ASR” were never designed for true real-time interaction at scale. The Nemotron Speech collection—part of the NVIDIA Nemotron family of open models—enables developers to plug speech into custom agentic workflows.

Buffered Inference is Not Real Streaming

In many production systems, streaming is implemented using buffered inference. Audio is processed in sliding windows, and each new window overlaps with the previous one to preserve context. While this produces correct transcripts, it is fundamentally inefficient.

The model repeatedly reprocesses audio it has already seen, sometimes several times over, just to maintain continuity.

Overlapping Windows Waste Compute

This overlap means redundant computation at every step:

The same audio frames are re-encoded
The same attention context is recomputed
GPU work scales faster than the actual audio stream
At low concurrency, this inefficiency may be tolerable. At scale, it becomes expensive and fragile.

Latency Drift Breaks Conversational Agents

As the number of concurrent streams increases, buffered systems often hit a scaling cliff. Latency begins to drift, and responses arrive later and later relative to spoken audio.

This drift is not a scheduling issue but a hardware resource problem. Because buffered inference repeatedly recomputes overlapping context, GPU memory fills with redundant activations and intermediate states. As memory pressure increases, the system becomes increasingly constrained, forcing slower execution, reduced batching efficiency, or outright throttling under load.

For conversational agents, this is fatal, even small delays disrupt downstream tasks such as turn-taking and interruption handling, making interactions feel unnatural. Over time, the system lags so far behind real speech that it can no longer support real-time dialogue—or it fails to scale at all when strict latency thresholds must be maintained.

This is the core limitation of legacy streaming ASR: it works in isolation, but breaks down under the computational and latency pressures of real-world, multi-user systems.

The Solution: Cache-Aware Streaming ASR for Lower Latency, Linear Scale, and Predictable Cost
Nemotron Speech ASR introduces a next-generation streaming architecture that replaces the buffered inference models of legacy systems. Its cache-aware design enables real-time, high concurrency voice agents with stable latency, linear scaling, and significantly higher GPU throughput—without compromising accuracy or robustness.

Key Benefits

Lower end-to-end latency: Reduces ASR processing time and redundant computation, minimizing end-to-end latency across voice agent pipelines with LLM reasoning and text-to-speech (TTS).
Efficient high concurrency: Maintains near flat latency, even as concurrency increases by up to 3x, avoiding the rapid degradation seen in buffered systems. In practice, latency grows sublinearly and does not meaningfully budge until concurrency is significantly higher.
Linear memory scaling: Cache-aware streaming prevents memory blow-ups enabling predictable performance and stable batching.
Higher GPU efficiency, lower cost: Maximizes parallel stream throughput per GPU, reducing overall cost per stream.
Inside Nemotron Speech ASR: FastConformer and 8x Downsampling
Nemotron Speech ASR is built on the FastConformer RNNT architecture, similar to previous NVIDIA Parakeet ASR models, optimized end-to-end for streaming inference.

A key innovation is 8x downsampling using depth-wise separable convolutional subsampling. Compared to traditional 4x systems, the encoder processes significantly fewer tokens per second, reducing VRAM footprint and increasing throughput across GPUs.

Key Engineering Specs

Architecture: FastConformer with 24 encoder layers and RNNT decoder
Parameters: 600M, optimized for high-throughput NVIDIA GPUs

Input: 16 kHz streaming audio

Output: Streaming English text with punctuation and capitalization

Dynamic, runtime-configurable latency modes: 80ms, 160ms, 560ms, 1.12s (no retraining required)

How Cache-Aware Streaming Works
Instead of re-encoding overlapping audio windows, Nemotron Speech ASR maintains an internal cache of encoder representations across all self-attention and convolution layers. When new audio arrives, the model updates this cached state rather than recomputing the previous context.

Each audio frame is processed exactly once without overlap or redundancy.

This design eliminates the two biggest problems of buffering inference:

Wasted computation from reprocessing the same audio
Latency drift as concurrent streams increase
The result is predictable end-to-end latency and linear scaling, even under heavy load

Cache Aware Streaming Pipeline
Figure 1.: Cache-Aware Streaming Pipeline: The streaming ASR architecture utilizing a Cache Aware Conformer Encoder and a context manager to maintain encoder states without redundant computation.

Prediction Chunking and Audio Buffer
Figure 2: Prediction Chunking and Audio Buffer: Detailed view of the audio buffer logic, demonstrating how prediction chunks and lookahead frames are processed in successive steps to ensure predictable memory behavior.

For more details, check out the paper.

Results: Throughput, Accuracy, and Speed at Scale
Throughput That Holds Under Load

The architectural efficiencies of cache-aware streaming translate directly into significant throughput gains. On the NVIDIA H100, Nemotron Speech ASR supports 560 concurrent streams at a 320ms chunk size, a 3x improvement over the baseline (180-streams). Similar gains are observed across the stack: the NVIDIA RTX A5000 delivers over 5x higher concurrency, while the NVIDIA DGX B200 provides up to a 2x throughput across 160ms and 320ms configurations.

Crucially, these benchmarks validate the system’s stability, maintaining zero latency drift even when pushed to peak capacity, enabled by bounded memory growth and cache reuse rather than repeated computation.

Nemotron Speech ASR 3x Improvement Concurrent Streams
Figure 3: Throughput expansion on NVIDIA H100, showing a 3x increase in concurrent supported streams at a 320ms chunk size compared to previous baselines.

These results highlight why demos and proofs of concept should also account for scale and cost, ensuring low latency holds at target workloads and supports a sustainable business case.

Accuracy Where It Counts: Latency-WER Tradeoffs

Most ASR leaderboards evaluate models in offline mode, which hides the real-world cost of low latency. In streaming ASR, accuracy and latency are inseparable.

Nemotron Speech ASR provides dynamic runtime flexibility, allowing developers to choose the right operating point at inference time—not during training.

A chunk latency increases from 0.16s to 0.56s, the model captures additional phonetic context, reducing WER from 7.84% to 7.22%, while maintaining real-time responsiveness.

Scaling Accuracy and Phonetic Context
Figure 4: Nemotron Speech ASR consistently outperforms CTC baseline (light green). Increased chunk latency improves WER by capturing richer phonetic context.

Fastest Time-To-FInal Transcription

Nemotron Speech ASR also delivers industry-leading time-to-final transcription across both local and API-based alternatives:

Nemotron Speech ASR: 24ms (median)
Alternative (local, NVIDIA L40 GPU): 90ms
Alternative models (API-based): 200ms+
Critically, finalization time remains stable even for long utterances—an essential property for real-time agents.

Real-World Validation
Watch the video

Modal: Validating Minimal Latency Drift at Scale
In collaboration with Modal, Nemotron Speech ASR was evaluated using asynchronous WebSocket streaming to measure latency stability at scale.

Evaluation setup:

ASR: Nemotron Speech ASR
Serving Configuration: 560ms latency mode on an NVIDIA H100 GPU
Load: 127 concurrent WebSocket clients
Duration: 3-minute continuous stream
Min Latency Modal
Figure 5: Aggregate ASR timing analysis while running 127 concurrent WebSocket clients demonstrating perfectly linear timestamp synchronization and a stable median delay of 182ms, validating minimal latency drift at scale.

At 127 simultaneous clients, Nemotron Speech ASR maintained stable end-to-end latency with minimal drift during a three-minute stream (Figure 5.). For voice agents, this difference is decisive. Even a few seconds a lag breaks turn-taking and renders interruption handling impossible.

As shown below, Nemotron Speech ASR achieves a level of efficiency previously thought impossible in real-time voice AI. The 160ms latency setting showcases the model's raw speed, delivering the fastest 'time-to-final' transcription available for high-stakes, real-time interactions.

What makes this architecture truly revolutionary is its intelligent resource management. When set to 160 ms latency, it pushes the absolute boundaries of hardware concurrency, but Nemotron has the unique flexibility to shift into high-capacity modes (560ms or 1.12s latency) that 'flatten the curve' entirely. This ensures that even at massive enterprise scale, users experience zero-drift, human-like responsiveness that proprietary APIs simply cannot sustain.

Scalability
Figure 6: Comparing real-world scaling of Nemotron ASR deployed on Modal to other open streaming ASR inference engines deployed on Modal and a proprietary API.

Daily: End-To-End Voice Agent Performance
Daily builds real-time audio and video infrastructure for developers creating voice-first and multimodal applications—from AI meeting assistants and customer support agents to real-time collaboration tools. For Daily’s users, predictable, low-latency speech pipelines are critical: delays or jitter directly translate into unnatural conversations and poor user experience.

To evaluate real-world performance, Daily integrated Nemotron Speech ASR into a full production-style voice agent pipeline consisting of:

ASR: Nemotron Speech ASR
Brain: Nemotron 3 Nano 30B
Voice: Magpie TTS (multilingual) with 7 languages and 5 voices.
Orchestration Library: Pipecat by Daily
Platforms: Modal, DGX Spark, RTX 5090
In this setup, Nemotron Speech ASR achieved a median time to final transcription of just 24ms, independent of utterance length. Long audio segments finalized as quickly as short ones—an essential property for interactive agents where users may speak unpredictably.

End-to-end, the full voice-to-voice loop completed in under 900ms for local deployment. This enables natural, turn-based conversations with stable, predictable latency, even under sustained interaction — exactly the behavior Daily’s developers need to build responsive, production-grade voice agents their users can trust.

Conclusion: A New Baseline for Real-Time Voice Agents
Most ASR systems were originally designed for offline transcription and later adopted for streaming use cases. As these legacy approaches are pushed into high-concurrency, their limitations become clear, surfacing latency drift, rising infrastructure costs, and degraded user experiences.

Voice agents place fundamentally different demands on speech recognition. Streaming and real-time interaction can no longer be afterthoughts; they must be treated as first-class design goals. Meeting the complexity of voice-first applications requires ASR architectures that are purpose-built for low latency, scalability, and sustained performance under load.

Cache-aware streaming changes this foundation.

With Nemotron Speech ASR, voice agents no longer need to trade speed for accuracy or scalability. By eliminating redundant computation and enabling predictable, linear scaling, the model delivers sub-100ms responsiveness, stable latency under high concurrency, and production-ready performance at scale.

Nemotron Speech ASR establishes a new baseline for real-time, voice-first AI.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
yes. If you want, next I’ll run that exact apples-to-apples benchmark on your setup (fixed one GPU worker, realtime WS clients, 160/320/560ms chunk modes, drift curves), then we’ll know exactly how far the gap is from NVIDIA’s reported conditions. techniques that are best with throughputs. We can create max number of supported websocket connections, since not everyone will be speaking at once right. or where can we do this filtering of VAD awareness. anyway, once we have certain amount of websocket connections, we can filter VAD aware chunks and do an insane batching throughput with it right ? what do you think about this plan. or how does these big labs approach this problem. and how do i get most out of single worker without compromising quality and having insanely high throughputs and GPU utlisation with enough headroom.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
can you debug into why it's returning 404 with /ws ? also note that you can SSH into the worker to debug more, so, if you need additional information on how things are set, and why it's not reachable, just ssh if you need, else debug with your plan. fix websockets and comapre and try to reproduce same numbers or atleast as close as those blogs and readme.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
were you able to reproduce same results as of that blog benchmarks of nvidia nemotron asr.

so we've created a RTX 6000 pro-96GB gpu machine. and you're saying only 75 concurrent requests per GPU ? is that accurate ? cause there is a blog of NVIDIA nemotron asr team benchmarking the H100 GPU with 80GB and they achieved like 500 concurrent requests with minimal drift in latency. not sure if we are doing the right way of inference or throughput management. Can you double check, or explain what's happening. Here's the link to the blog and it's content. Or you can also read the contents by teh nemotron's ASR repo in huggingface.

Hugging Face's logo
Hugging Face
Models
Datasets
Spaces
Community
Docs
Pricing

nvidia
/
nemotron-speech-streaming-en-0.6b

like
468

Follow
NVIDIA
49.7k
Automatic Speech Recognition
NeMo
PyTorch

19 datasets
speech-recognition
cache-aware ASR
streaming-asr
speech
audio
FastConformer
RNNT
Parakeet
ASR
NeMo
Eval Results (legacy)

arxiv:
2312.17279

arxiv:
2305.05084

License:
nvidia-open-model-license
Model card
Files and versions
xet
Community
11
nemotron-speech-streaming-en-0.6b
/
README.md

kunaldhawan's picture
kunaldhawan
Add links to Multitalker Parakeet Streaming and Parakeet Realtime EOU
c0acae9
verified
9 days ago
preview
code
|
raw

Copy download link
history
blame
contribute
delete

16.5 kB
---
license: other
license_name: nvidia-open-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
library_name: nemo
datasets:
- nvidia/Granary
- YTC
- Yodas2
- LibriLight
- librispeech_asr
- fisher_corpus
- Switchboard-1
- WSJ-0
- WSJ-1
- National-Singapore-Corpus-Part-1
- National-Singapore-Corpus-Part-6
- vctk
- voxpopuli
- europarl
- multilingual_librispeech
- fleurs
- mozilla-foundation/common_voice_8_0
- MLCommons/peoples_speech
- google/speech_commands
thumbnail: null
tags:
- speech-recognition
- cache-aware ASR
- automatic-speech-recognition
- streaming-asr
- speech
- audio
- FastConformer
- RNNT
- Parakeet
- ASR
- pytorch
- NeMo
widget:
- example_title: Librispeech sample 1
  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
- example_title: Librispeech sample 2
  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
model-index:
- name: nemotron-speech-streaming-en-0.6b
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: AMI
      type: ami
      config: ihm
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.58
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Earnings22
      type: earnings22
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 12.48
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Gigaspeech
      type: gigaspeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.45
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-clean
      type: librispeech_asr
      config: clean
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.31
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-other
      type: librispeech_asr
      config: other
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.75
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: SPGI Speech
      type: spgispeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.62
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: TEDLIUM
      type: tedlium
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.5
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: VoxPopuli
      type: voxpopuli
      config: en
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 7.57
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---

# Nemotron Speech ASR

[![Model architecture](https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-lightgrey#model-badge)](#model-architecture)
| [![Model size](https://img.shields.io/badge/Params-600M-lightgrey#model-badge)](#model-architecture)
| [![Language](https://img.shields.io/badge/Language-English-lightgrey#model-badge)](#datasets)
Nemotron-Speech-Streaming-En-0.6b is the first unified model in the Nemotron Speech family, engineered to deliver high-quality English transcription across both low-latency streaming and high-throughput batch workloads. The model natively supports punctuation and capitalization and offers runtime flexibility with configurable chunk sizes, including 80ms, 160ms, 560ms, and 1120ms.
Why Choose nvidia/nemotron-speech-streaming-en-0.6b?
* **Native Streaming Architecture:** Cache-aware design enables efficient processing of continuous audio streams, designed and optimized for low-latency voice agent applications interaction.
* **Improved Operational Efficiency:** Delivers superior throughput compared to traditional buffered streaming approaches. This allows for a higher number of parallel streams within the same GPU memory constraints, directly reducing operational costs for production environments.
* **Dynamic Runtime Flexibility:** Enables you to choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
* **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text

    Nemotron-speech-streaming-en-0.6b allows users to choose the optimal operating point on the latency-accuracy pareto curve at inference time, without requiring any re-training. Further, the cache-aware streaming mechanism scales much better than buffered streaming approaches, consistently outperforming production models like  parakeet-ctc-1_1b-asr  across chunk sizes.

This model consists of a cache-aware streaming 🦜 Parakeet (FastConformer) encoder with an RNN-T decoder. It is designed for real-time speech-to-text applications where low latency is critical, such as voice assistants, live captioning, and conversational AI systems. Unlike traditional "buffered" streaming, the cache-aware architecture enables continuous transcription by processing only new audio chunks while reusing cached encoder context. This significantly improves computational efficiency and minimizes end-to-end delay without sacrificing accuracy.
🗣 **Experience `Nemotron-Speech-Streaming-En-0.6b` in action** here: [https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b)

This model is ready for commercial/non-commercial use.

Read more about the model in the [dev blog](https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents) and check out the [paper](https://arxiv.org/abs/2312.17279).

## Explore more from NVIDIA:
For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at [developer.nvidia.com](https://developer.nvidia.com/).
Join the community to access tools, support, and resources to accelerate your development with NVIDIA's NeMo, Riva, NIM, and foundation models.

What is [Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/)?

NVIDIA Developer [Nemotron](https://developer.nvidia.com/nemotron)

[NVIDIA Riva Speech](https://developer.nvidia.com/riva?sortBy=developer_learning_library%2Fsort%2Ffeatured_in.riva%3Adesc%2Ctitle%3Aasc#demos)

[NeMo Documentation](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/models.html)

Also, check out the following NVIDIA speech models that extend the capabilities of Nemotron-Speech-Streaming:
* Multitalker Parakeet Streaming - https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1
* Parakeet Realtime EOU - https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1
## Access Model Inference and Examples:
* Deploy the Nemotron Speech ASR endpoint on [Modal](https://github.com/modal-projects/modal-nvidia-asr)
* Build local voice agent using [Daily’s framework](https://github.com/pipecat-ai/nemotron-january-2026)
## Model Architecture
**Architecture Type:** FastConformer-CacheAware-RNNT
The model is based on the Cache-Aware [1] FastConformer [2] architecture with 24 encoder layers and an RNNT (Recurrent Neural Network Transducer) decoder. The cache-aware streaming design enables efficient processing of audio in chunks while maintaining context from previous frames. Unlike buffered inference, this model maintains caches for all encoder self-attention and convolution layers. This enables reuse of hidden states at every streaming step, where cached activations eliminate redundant computations. As a result, there are no overlapping computations; each processed frame is strictly non-overlapping.
The caching schema of self-attention and convolution layers for consecutive chunks is as follows. For more details, please refer to [1].

**Network Architecture:**
- Encoder: Cache-Aware FastConformer with 24 layers
- Decoder: RNNT (Recurrent Neural Network Transducer)
- Parameters: 600M
## NVIDIA NeMo
To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo)[4]. We recommend you install it after you've installed Cython and latest PyTorch version.
```bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]
```
## How to Use this Model
The model is available for use in the NeMo Framework, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
### Loading the Model
```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/nemotron-speech-streaming-en-0.6b")
```
### Streaming Inference
You can use the cache-aware streaming inference script from NeMo - [NeMo/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py)
``` bash
cd NeMo
python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
    model_path=<model_path> \
    dataset_manifest=<dataset_manifest> \
    batch_size=<batch_size> \
    att_context_size="[70,13]" \ #set the second value to the desired right context from {0,1,6,13}
    output_path=<output_folder>
```

You can also run streaming inference through the pipeline method, which uses [NeMo/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml) configuration file to build end‑to‑end workflows with punctuation and capitalization (PnC), inverse text normalization (ITN), and translation support.

```python
from nemo.collections.asr.inference.factory.pipeline_builder import PipelineBuilder
from omegaconf import OmegaConf
# Path to the cache aware config file downloaded from above link
cfg_path = 'cache_aware_rnnt.yaml'
cfg = OmegaConf.load(cfg_path)
# Pass the paths of all the audio files for inferencing
audios = ['/path/to/your/audio.wav']
# Create the pipeline object and run inference
pipeline = PipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audios)
# Print the output
for entry in output:
  print(entry['text'])
```

_______
### Setting up Streaming Configuration
Latency is defined by the `att_context_size` param, where  att_context_size = `{num_frames_left_context, num_frame_right_context}`, all measured in **80ms frames**:
* [70, 0]: Chunk size = 1 (1 × 80ms = 0.08s)
* [70, 1]: Chunk size = 2 (2 × 80ms = 0.16s)
* [70, 6]: Chunk size = 7 (7 × 80ms = 0.56s)
* [70, 13]: Chunk size = 14 (14 × 80ms = 1.12s)
Here, chunk size = current frame + right context; each chunk is processed in non-overlapping fashion.
### Input
This model accepts single-channel (mono) audio sampled at 16,000 Hz. At least 80ms duration is required.
### Output
The model outputs English text transcriptions with punctuation and capitalization. The output text might be empty if input audio doesn't contain any speech.
## Datasets
### Training Datasets
The majority of the training data comes from the English portion of the Granary dataset [3]:
- YouTube-Commons (YTC) (109.5k hours)
- YODAS2 (102k hours)
- Mosel (14k hours)
- LibriLight (49.5k hours)
In addition, the following datasets were used:
- Librispeech 960 hours
- Fisher Corpus
- Switchboard-1 Dataset
- WSJ-0 and WSJ-1
- National Speech Corpus (Part 1, Part 6)
- VCTK
- VoxPopuli (EN)
- Europarl-ASR (EN)
- Multilingual Librispeech (MLS EN)
- Mozilla Common Voice (v11.0)
- Mozilla Common Voice (v7.0)
- Mozilla Common Voice (v4.0)
- People Speech
- AMI
**Data Modality:** Audio and text
**Audio Training Data Size:** 285k hours
**Data Collection Method:** Human - All audios are human recorded
**Labeling Method:** Hybrid (Human, Synthetic) - Some transcripts are generated by ASR models, while some are manually labeled
### Evaluation Datasets
The model was evaluated on the HuggingFace ASR Leaderboard datasets:
- AMI
- Earnings22
- Gigaspeech
- LibriSpeech test-clean
- LibriSpeech test-other
- SPGI Speech
- TEDLIUM
- VoxPopuli
## Performance
## ASR Performance (w/o PnC)
ASR performance is measured using the Word Error Rate (WER). Both ground-truth and predicted texts are processed using [whisper-normalizer](https://pypi.org/project/whisper-normalizer/) version 0.1.12.
The following tables show the WER on the [HuggingFace OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) datasets:
### Word Error Rate (WER) for chunk size of 1.12s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.16** | 11.58 | 12.48 | 11.45 | 2.31 | 4.75 | 2.62 | 4.50 | 7.57 |
### WER for chunk size of 0.56s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.22** | 11.69 | 12.61 | 11.43 | 2.40 | 4.97 | 2.64 | 4.46 | 7.59 |
### WER for chunk size of 0.16s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.84** | 13.88 | 13.61 | 12.12 | 2.43 | 5.33 | 2.82 | 4.80 | 7.72 |
### WER for chunk size of 0.08s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **8.528** | 16.05 | 14.6 | 12.92 | 2.55 | 5.79 | 3.01 | 5.07 | 8.23 |
## Software Integration
**Runtime Engine:** NeMo 25.11
**Supported Hardware Microarchitecture Compatibility:**
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Hopper
- NVIDIA Volta
**Test Hardware:**
- NVIDIA V100
- NVIDIA A100
- NVIDIA A6000
- DGX Spark
**Preferred/Supported Operating System(s):** Linux
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
## References
[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)

link to the blog for benchmarks - https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents

Scaling Real-Time Voice Agents with Cache-Aware Streaming ASR
Community Article
Published January 5, 2026
Kunal Dhawan's avatar
Kunal Dhawan
kunaldhawan

Follow
nvidia
Adi- margolin's avatar
Adi- margolin
Amargolin

Follow
nvidia
Gordana Neskovic's avatar
Gordana Neskovic
nesgor

Follow
nvidia
Maryam Motamedi's avatar
Maryam Motamedi
maryameee

Follow
nvidia
Yasmina Benkhoui's avatar
Yasmina Benkhoui
YasminaBenk

Follow
nvidia

In voice AI interactions, we have long been trapped by a familiar trade-off: speed versus accuracy. Traditionally, real-time Automatic Speech Recognition (ASR) relies on buffered inference; a workaround where the system repeatedly re-processes overlapping audio windows to maintain context. It is the computational equivalent of re-reading the last few pages of a book every time you turn the page.
NVIDIA Nemotron Speech ASR, a new, open model built specifically for real-time voice agents, breaks this cycle. Built on the FastConformer architecture with 8x downsampling, it introduces cache-aware technology to process only new audio "deltas." By reusing past computations rather than re-calculating them, it achieves up to 3x higher efficiency than traditional buffered systems.

In this post, we’ll explore how cache-aware architecture redefines the limits of real-time voice agents, and show real-world results from Daily and Modal for high-concurrency, low-latency voice agent workloads.

The Challenge: Why Streaming ASR Breaks at Scale
Most systems labeled as “streaming ASR” were never designed for true real-time interaction at scale. The Nemotron Speech collection—part of the NVIDIA Nemotron family of open models—enables developers to plug speech into custom agentic workflows.

Buffered Inference is Not Real Streaming

In many production systems, streaming is implemented using buffered inference. Audio is processed in sliding windows, and each new window overlaps with the previous one to preserve context. While this produces correct transcripts, it is fundamentally inefficient.

The model repeatedly reprocesses audio it has already seen, sometimes several times over, just to maintain continuity.

Overlapping Windows Waste Compute

This overlap means redundant computation at every step:

The same audio frames are re-encoded
The same attention context is recomputed
GPU work scales faster than the actual audio stream
At low concurrency, this inefficiency may be tolerable. At scale, it becomes expensive and fragile.

Latency Drift Breaks Conversational Agents

As the number of concurrent streams increases, buffered systems often hit a scaling cliff. Latency begins to drift, and responses arrive later and later relative to spoken audio.

This drift is not a scheduling issue but a hardware resource problem. Because buffered inference repeatedly recomputes overlapping context, GPU memory fills with redundant activations and intermediate states. As memory pressure increases, the system becomes increasingly constrained, forcing slower execution, reduced batching efficiency, or outright throttling under load.

For conversational agents, this is fatal, even small delays disrupt downstream tasks such as turn-taking and interruption handling, making interactions feel unnatural. Over time, the system lags so far behind real speech that it can no longer support real-time dialogue—or it fails to scale at all when strict latency thresholds must be maintained.

This is the core limitation of legacy streaming ASR: it works in isolation, but breaks down under the computational and latency pressures of real-world, multi-user systems.

The Solution: Cache-Aware Streaming ASR for Lower Latency, Linear Scale, and Predictable Cost
Nemotron Speech ASR introduces a next-generation streaming architecture that replaces the buffered inference models of legacy systems. Its cache-aware design enables real-time, high concurrency voice agents with stable latency, linear scaling, and significantly higher GPU throughput—without compromising accuracy or robustness.

Key Benefits

Lower end-to-end latency: Reduces ASR processing time and redundant computation, minimizing end-to-end latency across voice agent pipelines with LLM reasoning and text-to-speech (TTS).
Efficient high concurrency: Maintains near flat latency, even as concurrency increases by up to 3x, avoiding the rapid degradation seen in buffered systems. In practice, latency grows sublinearly and does not meaningfully budge until concurrency is significantly higher.
Linear memory scaling: Cache-aware streaming prevents memory blow-ups enabling predictable performance and stable batching.
Higher GPU efficiency, lower cost: Maximizes parallel stream throughput per GPU, reducing overall cost per stream.
Inside Nemotron Speech ASR: FastConformer and 8x Downsampling
Nemotron Speech ASR is built on the FastConformer RNNT architecture, similar to previous NVIDIA Parakeet ASR models, optimized end-to-end for streaming inference.

A key innovation is 8x downsampling using depth-wise separable convolutional subsampling. Compared to traditional 4x systems, the encoder processes significantly fewer tokens per second, reducing VRAM footprint and increasing throughput across GPUs.

Key Engineering Specs

Architecture: FastConformer with 24 encoder layers and RNNT decoder
Parameters: 600M, optimized for high-throughput NVIDIA GPUs

Input: 16 kHz streaming audio

Output: Streaming English text with punctuation and capitalization

Dynamic, runtime-configurable latency modes: 80ms, 160ms, 560ms, 1.12s (no retraining required)

How Cache-Aware Streaming Works
Instead of re-encoding overlapping audio windows, Nemotron Speech ASR maintains an internal cache of encoder representations across all self-attention and convolution layers. When new audio arrives, the model updates this cached state rather than recomputing the previous context.

Each audio frame is processed exactly once without overlap or redundancy.

This design eliminates the two biggest problems of buffering inference:

Wasted computation from reprocessing the same audio
Latency drift as concurrent streams increase
The result is predictable end-to-end latency and linear scaling, even under heavy load

Cache Aware Streaming Pipeline
Figure 1.: Cache-Aware Streaming Pipeline: The streaming ASR architecture utilizing a Cache Aware Conformer Encoder and a context manager to maintain encoder states without redundant computation.

Prediction Chunking and Audio Buffer
Figure 2: Prediction Chunking and Audio Buffer: Detailed view of the audio buffer logic, demonstrating how prediction chunks and lookahead frames are processed in successive steps to ensure predictable memory behavior.

For more details, check out the paper.

Results: Throughput, Accuracy, and Speed at Scale
Throughput That Holds Under Load

The architectural efficiencies of cache-aware streaming translate directly into significant throughput gains. On the NVIDIA H100, Nemotron Speech ASR supports 560 concurrent streams at a 320ms chunk size, a 3x improvement over the baseline (180-streams). Similar gains are observed across the stack: the NVIDIA RTX A5000 delivers over 5x higher concurrency, while the NVIDIA DGX B200 provides up to a 2x throughput across 160ms and 320ms configurations.

Crucially, these benchmarks validate the system’s stability, maintaining zero latency drift even when pushed to peak capacity, enabled by bounded memory growth and cache reuse rather than repeated computation.

Nemotron Speech ASR 3x Improvement Concurrent Streams
Figure 3: Throughput expansion on NVIDIA H100, showing a 3x increase in concurrent supported streams at a 320ms chunk size compared to previous baselines.

These results highlight why demos and proofs of concept should also account for scale and cost, ensuring low latency holds at target workloads and supports a sustainable business case.

Accuracy Where It Counts: Latency-WER Tradeoffs

Most ASR leaderboards evaluate models in offline mode, which hides the real-world cost of low latency. In streaming ASR, accuracy and latency are inseparable.

Nemotron Speech ASR provides dynamic runtime flexibility, allowing developers to choose the right operating point at inference time—not during training.

A chunk latency increases from 0.16s to 0.56s, the model captures additional phonetic context, reducing WER from 7.84% to 7.22%, while maintaining real-time responsiveness.

Scaling Accuracy and Phonetic Context
Figure 4: Nemotron Speech ASR consistently outperforms CTC baseline (light green). Increased chunk latency improves WER by capturing richer phonetic context.

Fastest Time-To-FInal Transcription

Nemotron Speech ASR also delivers industry-leading time-to-final transcription across both local and API-based alternatives:

Nemotron Speech ASR: 24ms (median)
Alternative (local, NVIDIA L40 GPU): 90ms
Alternative models (API-based): 200ms+
Critically, finalization time remains stable even for long utterances—an essential property for real-time agents.

Real-World Validation
Watch the video

Modal: Validating Minimal Latency Drift at Scale
In collaboration with Modal, Nemotron Speech ASR was evaluated using asynchronous WebSocket streaming to measure latency stability at scale.

Evaluation setup:

ASR: Nemotron Speech ASR
Serving Configuration: 560ms latency mode on an NVIDIA H100 GPU
Load: 127 concurrent WebSocket clients
Duration: 3-minute continuous stream
Min Latency Modal
Figure 5: Aggregate ASR timing analysis while running 127 concurrent WebSocket clients demonstrating perfectly linear timestamp synchronization and a stable median delay of 182ms, validating minimal latency drift at scale.

At 127 simultaneous clients, Nemotron Speech ASR maintained stable end-to-end latency with minimal drift during a three-minute stream (Figure 5.). For voice agents, this difference is decisive. Even a few seconds a lag breaks turn-taking and renders interruption handling impossible.

As shown below, Nemotron Speech ASR achieves a level of efficiency previously thought impossible in real-time voice AI. The 160ms latency setting showcases the model's raw speed, delivering the fastest 'time-to-final' transcription available for high-stakes, real-time interactions.

What makes this architecture truly revolutionary is its intelligent resource management. When set to 160 ms latency, it pushes the absolute boundaries of hardware concurrency, but Nemotron has the unique flexibility to shift into high-capacity modes (560ms or 1.12s latency) that 'flatten the curve' entirely. This ensures that even at massive enterprise scale, users experience zero-drift, human-like responsiveness that proprietary APIs simply cannot sustain.

Scalability
Figure 6: Comparing real-world scaling of Nemotron ASR deployed on Modal to other open streaming ASR inference engines deployed on Modal and a proprietary API.

Daily: End-To-End Voice Agent Performance
Daily builds real-time audio and video infrastructure for developers creating voice-first and multimodal applications—from AI meeting assistants and customer support agents to real-time collaboration tools. For Daily’s users, predictable, low-latency speech pipelines are critical: delays or jitter directly translate into unnatural conversations and poor user experience.

To evaluate real-world performance, Daily integrated Nemotron Speech ASR into a full production-style voice agent pipeline consisting of:

ASR: Nemotron Speech ASR
Brain: Nemotron 3 Nano 30B
Voice: Magpie TTS (multilingual) with 7 languages and 5 voices.
Orchestration Library: Pipecat by Daily
Platforms: Modal, DGX Spark, RTX 5090
In this setup, Nemotron Speech ASR achieved a median time to final transcription of just 24ms, independent of utterance length. Long audio segments finalized as quickly as short ones—an essential property for interactive agents where users may speak unpredictably.

End-to-end, the full voice-to-voice loop completed in under 900ms for local deployment. This enables natural, turn-based conversations with stable, predictable latency, even under sustained interaction — exactly the behavior Daily’s developers need to build responsive, production-grade voice agents their users can trust.

Conclusion: A New Baseline for Real-Time Voice Agents
Most ASR systems were originally designed for offline transcription and later adopted for streaming use cases. As these legacy approaches are pushed into high-concurrency, their limitations become clear, surfacing latency drift, rising infrastructure costs, and degraded user experiences.

Voice agents place fundamentally different demands on speech recognition. Streaming and real-time interaction can no longer be afterthoughts; they must be treated as first-class design goals. Meeting the complexity of voice-first applications requires ASR architectures that are purpose-built for low latency, scalability, and sustained performance under load.

Cache-aware streaming changes this foundation.

With Nemotron Speech ASR, voice agents no longer need to trade speed for accuracy or scalability. By eliminating redundant computation and enabling predictable, linear scaling, the model delivers sub-100ms responsiveness, stable latency under high concurrency, and production-ready performance at scale.

Nemotron Speech ASR establishes a new baseline for real-time, voice-first AI.

# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
I double-checked the stated benchmark conditions in NVIDIA’s January 5, 2026 post:

H100, 320ms mode: ~560 concurrent streams
H100, 560ms mode: 127 WS clients over 3 minutes with low drift
Source: https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents
Model card context: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/blob/main/README.md

are these scores reproducible on Modal's inference stack(as the benchmarks were done on H100 in modal right) ? i believe they have some limitations too regarding tunneling websocket connections. hence i wanted to run it here in serverless in vast.ai to avoid those issues. I need alteast like similar results being reproduced. what's like the best way to do it.


# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
yes. agreed to all points. But like for the 2nd point, why use english model. use our hindi model to compare the wer too. And also, for the 7th point, testing against single dedicated isntance for benchmark doesnt make any sense. ideal conditions to test are serverless as we need more instances being spinned when user concurrency demands. But before everything bench on a h100 in vast with current setup. debug more into why the current setup has bottlenecks.


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Starting repo inspection and benchmarking prep**


I’ll run an end-to-end diagnosis on your current Vast serverless setup with your Hindi model, then execute H100-focused benchmarks and extract bottlenecks from real metrics. First I’m going to inspect the current benchmark/deploy scripts and environment wiring so we can run the right tests without drifting from production conditions.


rg --files production benchmark_results vast_docs | sed -n '1,220p'


Output:
Chunk ID: e0a2ca
Wall time: 0.0514 seconds
Process exited with code 0
Original token count: 165
Output:
/home/ubuntu/nemo_hindi
total 60
drwxrwxr-x 10 ubuntu ubuntu  4096 Feb  9 08:14 .
drwxr-x--- 29 ubuntu ubuntu  4096 Feb  7 21:23 ..
-rw-rw-r--  1 ubuntu ubuntu   447 Feb  9 04:04 .env
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  8 03:59 __pycache__
drwxrwxr-x  2 ubuntu ubuntu 20480 Feb 10 00:54 benchmark_results
drwxrwxr-x  8 ubuntu ubuntu  4096 Feb  8 03:19 nemotron
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  7 21:23 nemotron-asr-frontend
drwxrwxr-x  3 ubuntu ubuntu  4096 Feb  8 20:33 nemotron_asr
drwxrwxr-x  3 ubuntu ubuntu  4096 Feb  9 22:06 production
drwxrwxr-x  2 ubuntu ubuntu  4096 Feb  9 07:47 vast_docs
drwxrwxr-x  6 ubuntu ubuntu  4096 Feb  7 20:14 venv