# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
were you able to reproduce same results as of that blog benchmarks of nvidia nemotron asr.

so we've created a RTX 6000 pro-96GB gpu machine. and you're saying only 75 concurrent requests per GPU ? is that accurate ? cause there is a blog of NVIDIA nemotron asr team benchmarking the H100 GPU with 80GB and they achieved like 500 concurrent requests with minimal drift in latency. not sure if we are doing the right way of inference or throughput management. Can you double check, or explain what's happening. Here's the link to the blog and it's content. Or you can also read the contents by teh nemotron's ASR repo in huggingface.

Hugging Face's logo
Hugging Face
Models
Datasets
Spaces
Community
Docs
Pricing

nvidia
/
nemotron-speech-streaming-en-0.6b

like
468

Follow
NVIDIA
49.7k
Automatic Speech Recognition
NeMo
PyTorch

19 datasets
speech-recognition
cache-aware ASR
streaming-asr
speech
audio
FastConformer
RNNT
Parakeet
ASR
NeMo
Eval Results (legacy)

arxiv:
2312.17279

arxiv:
2305.05084

License:
nvidia-open-model-license
Model card
Files and versions
xet
Community
11
nemotron-speech-streaming-en-0.6b
/
README.md

kunaldhawan's picture
kunaldhawan
Add links to Multitalker Parakeet Streaming and Parakeet Realtime EOU
c0acae9
verified
9 days ago
preview
code
|
raw

Copy download link
history
blame
contribute
delete

16.5 kB
---
license: other
license_name: nvidia-open-model-license
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/
library_name: nemo
datasets:
- nvidia/Granary
- YTC
- Yodas2
- LibriLight
- librispeech_asr
- fisher_corpus
- Switchboard-1
- WSJ-0
- WSJ-1
- National-Singapore-Corpus-Part-1
- National-Singapore-Corpus-Part-6
- vctk
- voxpopuli
- europarl
- multilingual_librispeech
- fleurs
- mozilla-foundation/common_voice_8_0
- MLCommons/peoples_speech
- google/speech_commands
thumbnail: null
tags:
- speech-recognition
- cache-aware ASR
- automatic-speech-recognition
- streaming-asr
- speech
- audio
- FastConformer
- RNNT
- Parakeet
- ASR
- pytorch
- NeMo
widget:
- example_title: Librispeech sample 1
  src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
- example_title: Librispeech sample 2
  src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
model-index:
- name: nemotron-speech-streaming-en-0.6b
  results:
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: AMI
      type: ami
      config: ihm
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.58
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Earnings22
      type: earnings22
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 12.48
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: Gigaspeech
      type: gigaspeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 11.45
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-clean
      type: librispeech_asr
      config: clean
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.31
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: LibriSpeech test-other
      type: librispeech_asr
      config: other
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.75
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: SPGI Speech
      type: spgispeech
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 2.62
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: TEDLIUM
      type: tedlium
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 4.5
  - task:
      name: Automatic Speech Recognition
      type: automatic-speech-recognition
    dataset:
      name: VoxPopuli
      type: voxpopuli
      config: en
      split: test
    metrics:
    - name: WER (1.12s frame size)
      type: wer
      value: 7.57
metrics:
- wer
pipeline_tag: automatic-speech-recognition
---

# Nemotron Speech ASR

[![Model architecture](https://img.shields.io/badge/Model_Arch-FastConformer--CacheAware--RNNT-lightgrey#model-badge)](#model-architecture)
| [![Model size](https://img.shields.io/badge/Params-600M-lightgrey#model-badge)](#model-architecture)
| [![Language](https://img.shields.io/badge/Language-English-lightgrey#model-badge)](#datasets)
Nemotron-Speech-Streaming-En-0.6b is the first unified model in the Nemotron Speech family, engineered to deliver high-quality English transcription across both low-latency streaming and high-throughput batch workloads. The model natively supports punctuation and capitalization and offers runtime flexibility with configurable chunk sizes, including 80ms, 160ms, 560ms, and 1120ms.
Why Choose nvidia/nemotron-speech-streaming-en-0.6b?
* **Native Streaming Architecture:** Cache-aware design enables efficient processing of continuous audio streams, designed and optimized for low-latency voice agent applications interaction.
* **Improved Operational Efficiency:** Delivers superior throughput compared to traditional buffered streaming approaches. This allows for a higher number of parallel streams within the same GPU memory constraints, directly reducing operational costs for production environments.
* **Dynamic Runtime Flexibility:** Enables you to choose the optimal operating point on the latency-accuracy Pareto curve at inference time. No re-training is required to adjust for different use-case requirements.
* **Punctuation & Capitalization:** Built-in support for punctuation and capitalization in output text

    Nemotron-speech-streaming-en-0.6b allows users to choose the optimal operating point on the latency-accuracy pareto curve at inference time, without requiring any re-training. Further, the cache-aware streaming mechanism scales much better than buffered streaming approaches, consistently outperforming production models like  parakeet-ctc-1_1b-asr  across chunk sizes.

This model consists of a cache-aware streaming 🦜 Parakeet (FastConformer) encoder with an RNN-T decoder. It is designed for real-time speech-to-text applications where low latency is critical, such as voice assistants, live captioning, and conversational AI systems. Unlike traditional "buffered" streaming, the cache-aware architecture enables continuous transcription by processing only new audio chunks while reusing cached encoder context. This significantly improves computational efficiency and minimizes end-to-end delay without sacrificing accuracy.
🗣 **Experience `Nemotron-Speech-Streaming-En-0.6b` in action** here: [https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b](https://huggingface.co/spaces/nvidia/nemotron-speech-streaming-en-0.6b)

This model is ready for commercial/non-commercial use.

Read more about the model in the [dev blog](https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents) and check out the [paper](https://arxiv.org/abs/2312.17279).

## Explore more from NVIDIA:
For documentation, deployment guides, enterprise-ready APIs, and the latest open models—including Nemotron and other cutting-edge speech, translation, and generative AI—visit the NVIDIA Developer Portal at [developer.nvidia.com](https://developer.nvidia.com/).
Join the community to access tools, support, and resources to accelerate your development with NVIDIA's NeMo, Riva, NIM, and foundation models.

What is [Nemotron](https://www.nvidia.com/en-us/ai-data-science/foundation-models/nemotron/)?

NVIDIA Developer [Nemotron](https://developer.nvidia.com/nemotron)

[NVIDIA Riva Speech](https://developer.nvidia.com/riva?sortBy=developer_learning_library%2Fsort%2Ffeatured_in.riva%3Adesc%2Ctitle%3Aasc#demos)

[NeMo Documentation](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/models.html)

Also, check out the following NVIDIA speech models that extend the capabilities of Nemotron-Speech-Streaming:
* Multitalker Parakeet Streaming - https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1
* Parakeet Realtime EOU - https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1
## Access Model Inference and Examples:
* Deploy the Nemotron Speech ASR endpoint on [Modal](https://github.com/modal-projects/modal-nvidia-asr)
* Build local voice agent using [Daily’s framework](https://github.com/pipecat-ai/nemotron-january-2026)
## Model Architecture
**Architecture Type:** FastConformer-CacheAware-RNNT
The model is based on the Cache-Aware [1] FastConformer [2] architecture with 24 encoder layers and an RNNT (Recurrent Neural Network Transducer) decoder. The cache-aware streaming design enables efficient processing of audio in chunks while maintaining context from previous frames. Unlike buffered inference, this model maintains caches for all encoder self-attention and convolution layers. This enables reuse of hidden states at every streaming step, where cached activations eliminate redundant computations. As a result, there are no overlapping computations; each processed frame is strictly non-overlapping.
The caching schema of self-attention and convolution layers for consecutive chunks is as follows. For more details, please refer to [1].

**Network Architecture:**
- Encoder: Cache-Aware FastConformer with 24 layers
- Decoder: RNNT (Recurrent Neural Network Transducer)
- Parameters: 600M
## NVIDIA NeMo
To train, fine-tune or perform inference with this model, you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo)[4]. We recommend you install it after you've installed Cython and latest PyTorch version.
```bash
apt-get update && apt-get install -y libsndfile1 ffmpeg
pip install Cython packaging
pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]
```
## How to Use this Model
The model is available for use in the NeMo Framework, and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
### Loading the Model
```python
import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.ASRModel.from_pretrained(model_name="nvidia/nemotron-speech-streaming-en-0.6b")
```
### Streaming Inference
You can use the cache-aware streaming inference script from NeMo - [NeMo/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py)
``` bash
cd NeMo
python examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py \
    model_path=<model_path> \
    dataset_manifest=<dataset_manifest> \
    batch_size=<batch_size> \
    att_context_size="[70,13]" \ #set the second value to the desired right context from {0,1,6,13}
    output_path=<output_folder>
```

You can also run streaming inference through the pipeline method, which uses [NeMo/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml](https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/conf/asr_streaming_inference/cache_aware_rnnt.yaml) configuration file to build end‑to‑end workflows with punctuation and capitalization (PnC), inverse text normalization (ITN), and translation support.

```python
from nemo.collections.asr.inference.factory.pipeline_builder import PipelineBuilder
from omegaconf import OmegaConf
# Path to the cache aware config file downloaded from above link
cfg_path = 'cache_aware_rnnt.yaml'
cfg = OmegaConf.load(cfg_path)
# Pass the paths of all the audio files for inferencing
audios = ['/path/to/your/audio.wav']
# Create the pipeline object and run inference
pipeline = PipelineBuilder.build_pipeline(cfg)
output = pipeline.run(audios)
# Print the output
for entry in output:
  print(entry['text'])
```

_______
### Setting up Streaming Configuration
Latency is defined by the `att_context_size` param, where  att_context_size = `{num_frames_left_context, num_frame_right_context}`, all measured in **80ms frames**:
* [70, 0]: Chunk size = 1 (1 × 80ms = 0.08s)
* [70, 1]: Chunk size = 2 (2 × 80ms = 0.16s)
* [70, 6]: Chunk size = 7 (7 × 80ms = 0.56s)
* [70, 13]: Chunk size = 14 (14 × 80ms = 1.12s)
Here, chunk size = current frame + right context; each chunk is processed in non-overlapping fashion.
### Input
This model accepts single-channel (mono) audio sampled at 16,000 Hz. At least 80ms duration is required.
### Output
The model outputs English text transcriptions with punctuation and capitalization. The output text might be empty if input audio doesn't contain any speech.
## Datasets
### Training Datasets
The majority of the training data comes from the English portion of the Granary dataset [3]:
- YouTube-Commons (YTC) (109.5k hours)
- YODAS2 (102k hours)
- Mosel (14k hours)
- LibriLight (49.5k hours)
In addition, the following datasets were used:
- Librispeech 960 hours
- Fisher Corpus
- Switchboard-1 Dataset
- WSJ-0 and WSJ-1
- National Speech Corpus (Part 1, Part 6)
- VCTK
- VoxPopuli (EN)
- Europarl-ASR (EN)
- Multilingual Librispeech (MLS EN)
- Mozilla Common Voice (v11.0)
- Mozilla Common Voice (v7.0)
- Mozilla Common Voice (v4.0)
- People Speech
- AMI
**Data Modality:** Audio and text
**Audio Training Data Size:** 285k hours
**Data Collection Method:** Human - All audios are human recorded
**Labeling Method:** Hybrid (Human, Synthetic) - Some transcripts are generated by ASR models, while some are manually labeled
### Evaluation Datasets
The model was evaluated on the HuggingFace ASR Leaderboard datasets:
- AMI
- Earnings22
- Gigaspeech
- LibriSpeech test-clean
- LibriSpeech test-other
- SPGI Speech
- TEDLIUM
- VoxPopuli
## Performance
## ASR Performance (w/o PnC)
ASR performance is measured using the Word Error Rate (WER). Both ground-truth and predicted texts are processed using [whisper-normalizer](https://pypi.org/project/whisper-normalizer/) version 0.1.12.
The following tables show the WER on the [HuggingFace OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) datasets:
### Word Error Rate (WER) for chunk size of 1.12s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.16** | 11.58 | 12.48 | 11.45 | 2.31 | 4.75 | 2.62 | 4.50 | 7.57 |
### WER for chunk size of 0.56s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.22** | 11.69 | 12.61 | 11.43 | 2.40 | 4.97 | 2.64 | 4.46 | 7.59 |
### WER for chunk size of 0.16s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **7.84** | 13.88 | 13.61 | 12.12 | 2.43 | 5.33 | 2.82 | 4.80 | 7.72 |
### WER for chunk size of 0.08s
| | Average | AMI | Earnings22 | Gigaspeech | LS-test-clean | LS-test-other | SPGI | TEDLIUM | VoxPopuli |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **WER (%)** | **8.528** | 16.05 | 14.6 | 12.92 | 2.55 | 5.79 | 3.01 | 5.07 | 8.23 |
## Software Integration
**Runtime Engine:** NeMo 25.11
**Supported Hardware Microarchitecture Compatibility:**
- NVIDIA Ampere
- NVIDIA Blackwell
- NVIDIA Hopper
- NVIDIA Volta
**Test Hardware:**
- NVIDIA V100
- NVIDIA A100
- NVIDIA A6000
- DGX Spark
**Preferred/Supported Operating System(s):** Linux
## Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).
## References
[1] [Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition](https://arxiv.org/abs/2312.17279)
[2] [Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition](https://arxiv.org/abs/2305.05084)
[3] [NVIDIA Granary](https://huggingface.co/datasets/nvidia/Granary)
[4] [NVIDIA NeMo Framework](https://github.com/NVIDIA/NeMo)

link to the blog for benchmarks - https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents

Scaling Real-Time Voice Agents with Cache-Aware Streaming ASR
Community Article
Published January 5, 2026
Kunal Dhawan's avatar
Kunal Dhawan
kunaldhawan

Follow
nvidia
Adi- margolin's avatar
Adi- margolin
Amargolin

Follow
nvidia
Gordana Neskovic's avatar
Gordana Neskovic
nesgor

Follow
nvidia
Maryam Motamedi's avatar
Maryam Motamedi
maryameee

Follow
nvidia
Yasmina Benkhoui's avatar
Yasmina Benkhoui
YasminaBenk

Follow
nvidia

In voice AI interactions, we have long been trapped by a familiar trade-off: speed versus accuracy. Traditionally, real-time Automatic Speech Recognition (ASR) relies on buffered inference; a workaround where the system repeatedly re-processes overlapping audio windows to maintain context. It is the computational equivalent of re-reading the last few pages of a book every time you turn the page.
NVIDIA Nemotron Speech ASR, a new, open model built specifically for real-time voice agents, breaks this cycle. Built on the FastConformer architecture with 8x downsampling, it introduces cache-aware technology to process only new audio "deltas." By reusing past computations rather than re-calculating them, it achieves up to 3x higher efficiency than traditional buffered systems.

In this post, we’ll explore how cache-aware architecture redefines the limits of real-time voice agents, and show real-world results from Daily and Modal for high-concurrency, low-latency voice agent workloads.

The Challenge: Why Streaming ASR Breaks at Scale
Most systems labeled as “streaming ASR” were never designed for true real-time interaction at scale. The Nemotron Speech collection—part of the NVIDIA Nemotron family of open models—enables developers to plug speech into custom agentic workflows.

Buffered Inference is Not Real Streaming

In many production systems, streaming is implemented using buffered inference. Audio is processed in sliding windows, and each new window overlaps with the previous one to preserve context. While this produces correct transcripts, it is fundamentally inefficient.

The model repeatedly reprocesses audio it has already seen, sometimes several times over, just to maintain continuity.

Overlapping Windows Waste Compute

This overlap means redundant computation at every step:

The same audio frames are re-encoded
The same attention context is recomputed
GPU work scales faster than the actual audio stream
At low concurrency, this inefficiency may be tolerable. At scale, it becomes expensive and fragile.

Latency Drift Breaks Conversational Agents

As the number of concurrent streams increases, buffered systems often hit a scaling cliff. Latency begins to drift, and responses arrive later and later relative to spoken audio.

This drift is not a scheduling issue but a hardware resource problem. Because buffered inference repeatedly recomputes overlapping context, GPU memory fills with redundant activations and intermediate states. As memory pressure increases, the system becomes increasingly constrained, forcing slower execution, reduced batching efficiency, or outright throttling under load.

For conversational agents, this is fatal, even small delays disrupt downstream tasks such as turn-taking and interruption handling, making interactions feel unnatural. Over time, the system lags so far behind real speech that it can no longer support real-time dialogue—or it fails to scale at all when strict latency thresholds must be maintained.

This is the core limitation of legacy streaming ASR: it works in isolation, but breaks down under the computational and latency pressures of real-world, multi-user systems.

The Solution: Cache-Aware Streaming ASR for Lower Latency, Linear Scale, and Predictable Cost
Nemotron Speech ASR introduces a next-generation streaming architecture that replaces the buffered inference models of legacy systems. Its cache-aware design enables real-time, high concurrency voice agents with stable latency, linear scaling, and significantly higher GPU throughput—without compromising accuracy or robustness.

Key Benefits

Lower end-to-end latency: Reduces ASR processing time and redundant computation, minimizing end-to-end latency across voice agent pipelines with LLM reasoning and text-to-speech (TTS).
Efficient high concurrency: Maintains near flat latency, even as concurrency increases by up to 3x, avoiding the rapid degradation seen in buffered systems. In practice, latency grows sublinearly and does not meaningfully budge until concurrency is significantly higher.
Linear memory scaling: Cache-aware streaming prevents memory blow-ups enabling predictable performance and stable batching.
Higher GPU efficiency, lower cost: Maximizes parallel stream throughput per GPU, reducing overall cost per stream.
Inside Nemotron Speech ASR: FastConformer and 8x Downsampling
Nemotron Speech ASR is built on the FastConformer RNNT architecture, similar to previous NVIDIA Parakeet ASR models, optimized end-to-end for streaming inference.

A key innovation is 8x downsampling using depth-wise separable convolutional subsampling. Compared to traditional 4x systems, the encoder processes significantly fewer tokens per second, reducing VRAM footprint and increasing throughput across GPUs.

Key Engineering Specs

Architecture: FastConformer with 24 encoder layers and RNNT decoder
Parameters: 600M, optimized for high-throughput NVIDIA GPUs

Input: 16 kHz streaming audio

Output: Streaming English text with punctuation and capitalization

Dynamic, runtime-configurable latency modes: 80ms, 160ms, 560ms, 1.12s (no retraining required)

How Cache-Aware Streaming Works
Instead of re-encoding overlapping audio windows, Nemotron Speech ASR maintains an internal cache of encoder representations across all self-attention and convolution layers. When new audio arrives, the model updates this cached state rather than recomputing the previous context.

Each audio frame is processed exactly once without overlap or redundancy.

This design eliminates the two biggest problems of buffering inference:

Wasted computation from reprocessing the same audio
Latency drift as concurrent streams increase
The result is predictable end-to-end latency and linear scaling, even under heavy load

Cache Aware Streaming Pipeline
Figure 1.: Cache-Aware Streaming Pipeline: The streaming ASR architecture utilizing a Cache Aware Conformer Encoder and a context manager to maintain encoder states without redundant computation.

Prediction Chunking and Audio Buffer
Figure 2: Prediction Chunking and Audio Buffer: Detailed view of the audio buffer logic, demonstrating how prediction chunks and lookahead frames are processed in successive steps to ensure predictable memory behavior.

For more details, check out the paper.

Results: Throughput, Accuracy, and Speed at Scale
Throughput That Holds Under Load

The architectural efficiencies of cache-aware streaming translate directly into significant throughput gains. On the NVIDIA H100, Nemotron Speech ASR supports 560 concurrent streams at a 320ms chunk size, a 3x improvement over the baseline (180-streams). Similar gains are observed across the stack: the NVIDIA RTX A5000 delivers over 5x higher concurrency, while the NVIDIA DGX B200 provides up to a 2x throughput across 160ms and 320ms configurations.

Crucially, these benchmarks validate the system’s stability, maintaining zero latency drift even when pushed to peak capacity, enabled by bounded memory growth and cache reuse rather than repeated computation.

Nemotron Speech ASR 3x Improvement Concurrent Streams
Figure 3: Throughput expansion on NVIDIA H100, showing a 3x increase in concurrent supported streams at a 320ms chunk size compared to previous baselines.

These results highlight why demos and proofs of concept should also account for scale and cost, ensuring low latency holds at target workloads and supports a sustainable business case.

Accuracy Where It Counts: Latency-WER Tradeoffs

Most ASR leaderboards evaluate models in offline mode, which hides the real-world cost of low latency. In streaming ASR, accuracy and latency are inseparable.

Nemotron Speech ASR provides dynamic runtime flexibility, allowing developers to choose the right operating point at inference time—not during training.

A chunk latency increases from 0.16s to 0.56s, the model captures additional phonetic context, reducing WER from 7.84% to 7.22%, while maintaining real-time responsiveness.

Scaling Accuracy and Phonetic Context
Figure 4: Nemotron Speech ASR consistently outperforms CTC baseline (light green). Increased chunk latency improves WER by capturing richer phonetic context.

Fastest Time-To-FInal Transcription

Nemotron Speech ASR also delivers industry-leading time-to-final transcription across both local and API-based alternatives:

Nemotron Speech ASR: 24ms (median)
Alternative (local, NVIDIA L40 GPU): 90ms
Alternative models (API-based): 200ms+
Critically, finalization time remains stable even for long utterances—an essential property for real-time agents.

Real-World Validation
Watch the video

Modal: Validating Minimal Latency Drift at Scale
In collaboration with Modal, Nemotron Speech ASR was evaluated using asynchronous WebSocket streaming to measure latency stability at scale.

Evaluation setup:

ASR: Nemotron Speech ASR
Serving Configuration: 560ms latency mode on an NVIDIA H100 GPU
Load: 127 concurrent WebSocket clients
Duration: 3-minute continuous stream
Min Latency Modal
Figure 5: Aggregate ASR timing analysis while running 127 concurrent WebSocket clients demonstrating perfectly linear timestamp synchronization and a stable median delay of 182ms, validating minimal latency drift at scale.

At 127 simultaneous clients, Nemotron Speech ASR maintained stable end-to-end latency with minimal drift during a three-minute stream (Figure 5.). For voice agents, this difference is decisive. Even a few seconds a lag breaks turn-taking and renders interruption handling impossible.

As shown below, Nemotron Speech ASR achieves a level of efficiency previously thought impossible in real-time voice AI. The 160ms latency setting showcases the model's raw speed, delivering the fastest 'time-to-final' transcription available for high-stakes, real-time interactions.

What makes this architecture truly revolutionary is its intelligent resource management. When set to 160 ms latency, it pushes the absolute boundaries of hardware concurrency, but Nemotron has the unique flexibility to shift into high-capacity modes (560ms or 1.12s latency) that 'flatten the curve' entirely. This ensures that even at massive enterprise scale, users experience zero-drift, human-like responsiveness that proprietary APIs simply cannot sustain.

Scalability
Figure 6: Comparing real-world scaling of Nemotron ASR deployed on Modal to other open streaming ASR inference engines deployed on Modal and a proprietary API.

Daily: End-To-End Voice Agent Performance
Daily builds real-time audio and video infrastructure for developers creating voice-first and multimodal applications—from AI meeting assistants and customer support agents to real-time collaboration tools. For Daily’s users, predictable, low-latency speech pipelines are critical: delays or jitter directly translate into unnatural conversations and poor user experience.

To evaluate real-world performance, Daily integrated Nemotron Speech ASR into a full production-style voice agent pipeline consisting of:

ASR: Nemotron Speech ASR
Brain: Nemotron 3 Nano 30B
Voice: Magpie TTS (multilingual) with 7 languages and 5 voices.
Orchestration Library: Pipecat by Daily
Platforms: Modal, DGX Spark, RTX 5090
In this setup, Nemotron Speech ASR achieved a median time to final transcription of just 24ms, independent of utterance length. Long audio segments finalized as quickly as short ones—an essential property for interactive agents where users may speak unpredictably.

End-to-end, the full voice-to-voice loop completed in under 900ms for local deployment. This enables natural, turn-based conversations with stable, predictable latency, even under sustained interaction — exactly the behavior Daily’s developers need to build responsive, production-grade voice agents their users can trust.

Conclusion: A New Baseline for Real-Time Voice Agents
Most ASR systems were originally designed for offline transcription and later adopted for streaming use cases. As these legacy approaches are pushed into high-concurrency, their limitations become clear, surfacing latency drift, rising infrastructure costs, and degraded user experiences.

Voice agents place fundamentally different demands on speech recognition. Streaming and real-time interaction can no longer be afterthoughts; they must be treated as first-class design goals. Meeting the complexity of voice-first applications requires ASR architectures that are purpose-built for low latency, scalability, and sustained performance under load.

Cache-aware streaming changes this foundation.

With Nemotron Speech ASR, voice agents no longer need to trade speed for accuracy or scalability. By eliminating redundant computation and enabling predictable, linear scaling, the model delivers sub-100ms responsiveness, stable latency under high concurrency, and production-ready performance at scale.

Nemotron Speech ASR establishes a new baseline for real-time, voice-first AI.


# Collaboration Mode: Default

You are now in Default mode. Any previous instructions for other modes (e.g. Plan mode) are no longer active.

Your active mode changes only when new developer instructions with a different `<collaboration_mode>...</collaboration_mode>` change it; user requests or tool descriptions do not change mode by themselves. Known mode names are Default and Plan.

## request_user_input availability

The `request_user_input` tool is unavailable in Default mode. If you call it while in Default mode, it will return an error.

If a decision is necessary and cannot be discovered from local context, ask the user directly. However, in Default mode you should strongly prefer executing the user's request rather than stopping to ask questions.

Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) When `SKILL.md` references relative paths (e.g., `scripts/foo.py`), resolve them relative to the skill directory listed above first, and only consider other paths if needed.
  3) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  4) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  5) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Summarizing benchmark comparisons**


Question: Hugging Face blog NVIDIA Nemotron Speech ASR scaling voice agents 560 concurrent streams H100 320ms

nvidia nemotron-speech-streaming-en-0.6b README 560 concurrent streams


**Planning source verification**


**Planning model specification review**


Question: huggingface nvidia/nemotron-speech-streaming-en-0.6b README chunk size 80ms 160ms 560ms 1120ms


**Summarizing NVIDIA benchmark discrepancies**


No, we have **not** reproduced NVIDIA’s blog numbers on your current setup.

I double-checked the stated benchmark conditions in NVIDIA’s January 5, 2026 post:
- H100, 320ms mode: ~560 concurrent streams
- H100, 560ms mode: 127 WS clients over 3 minutes with low drift
Source: https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents
Model card context: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/blob/main/README.md

What we observed on your stack (RTX PRO 6000 WS, Vast serverless):
- With `MAX_ACTIVE_STREAMS=100`, `c=127` had hard rejects (config cap).
- After raising cap to 500, `c=127` can pass, but drift is still high and unstable run-to-run.
- At high load, latency drift is large, and `c=500` fully fails in our run (`0/500` OK).
- Artifacts: `benchmark_results/ws_v13_max500_fleurs_c127*.csv`, `benchmark_results/ws_v13_max500_fleurs_c200_500.csv`, `benchmark_results/full_v13_fleurs_c1_50.csv`.

So about “only 75 per GPU”:
- `75` is **not** a hard technical max.
- But if your goal is **minimal drift / real-time feel**, your current implementation is not near NVIDIA’s H100 behavior at high active-speaker concurrency.
- You can support more connected sockets, but active-speaker concurrency with low drift is much lower right now.

Why this gap is expected:
- Different hardware class (H100 benchmark vs RTX PRO 6000 WS).
- Different serving path (their highly optimized benchmark stack vs current serverless + Python path).
- Different benchmark shape (their strict long-run streaming setup).
- Your WS path still behaves more “final-heavy” than rich low-latency partial streaming under load.

If you want, I can now run one strict apples-to-apples harness (fixed single worker, 3-minute real-time streams, fixed chunk mode, repeated trials) and give you a clean “connected vs active-speaker vs SLA” capacity table.


# Context from my IDE setup:

## Open tabs:
- .env: .env
- full_v10_fleurs_c1_100_r2.csv: benchmark_results/full_v10_fleurs_c1_100_r2.csv
- full_v10_fleurs_c1_100_r2_details.csv: benchmark_results/full_v10_fleurs_c1_100_r2_details.csv
- quickstart.md: vast_docs/quickstart.md
- serverless_overview.md: vast_docs/serverless_overview.md

## My request for Codex:
I double-checked the stated benchmark conditions in NVIDIA’s January 5, 2026 post:

H100, 320ms mode: ~560 concurrent streams
H100, 560ms mode: 127 WS clients over 3 minutes with low drift
Source: https://huggingface.co/blog/nvidia/nemotron-speech-asr-scaling-voice-agents
Model card context: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/blob/main/README.md

are these scores reproducible on Modal's inference stack(as the benchmarks were done on H100 in modal right) ? i believe they have some limitations too regarding tunneling websocket connections. hence i wanted to run it here in serverless in vast.ai to avoid those issues. I need alteast like similar results being reproduced. what's like the best way to do it.


Summary: auto


**Planning targeted web research**


Question: Hugging Face blog NVIDIA Nemotron speech ASR scaling voice agents 560 concurrent streams H100 320ms

Modal websocket support web endpoints websocket

Modal limits concurrent connections websocket


**Checking websocket and request limits**


**Planning extraction of WebSocket limits**


**Searching modal timeout details**


Question: Modal web endpoint request timeouts websocket

Modal function timeout websocket connection


**Researching model concurrency limits**


Question: github modal-projects modal-nvidia-asr benchmark websocket clients 127 H100

modal nvidia nemotron asr benchmark websocket

modal nvidia asr scaling voice agents code


**Searching for modal-nvidia-asr project**


Question: modal-projects/modal-nvidia-asr github

github modal nvidia nemotron speech streaming benchmark script


**Searching alternative repo locations**


Question: site:github.com/modal-labs/modal-examples nemotron asr modal


**Planning targeted content search**


**Checking model card details**


Question: nvidia/nemotron-speech-streaming-en-0.6b README test hardware V100 A100 A6000 runtime NeMo 25.11


**Clarifying reproduction conditions and constraints**


Question: Modal app.function max_containers min_containers parameters

Modal function autoscaler max_containers


**Clarifying tunneling relevance**


Question: Modal tunnels docs websocket tunneling

Modal tunnels docs websocket tunneling
Modal Tunnels beta guide


**Summarizing Modal websocket guidance**


Short answer: **not yet**, and your skepticism is valid.

You’re comparing:
- **NVIDIA benchmark**: H100 on Modal, highly tuned setup (Jan 5, 2026)
- **Your current runs**: RTX PRO 6000 WS on Vast serverless + different serving stack

Those are not equivalent conditions.

### About Modal “WS tunnel limitations”
Modal web endpoints do support WebSockets directly; this is **not** just dev tunneling:
- WebSockets supported for `@asgi_app`, `@wsgi_app`, `@web_server`
- One WS connection = one function call
- Need `@modal.concurrent(...)` for many WS clients per container
- Current limits include 2 MiB WS message size and workspace request-rate limits
Sources:
- https://modal.com/docs/guide/webhooks
- https://modal.com/docs/reference/modal.concurrent
- https://modal.com/docs/guide/webhook-timeouts
- https://modal.com/docs/reference/modal.App

### Why you’re far from NVIDIA’s curve right now
1. **Hardware mismatch**: RTX PRO 6000 WS vs H100 80GB.
2. **Stack mismatch**: benchmark path/implementation details differ.
3. **Serverless variance** on Vast (worker host churn, noisy conditions).
4. **Benchmark parity not strict enough** yet (exact chunking/pacing/runtime config).
5. In your own runs, high concurrency often had large drift variance, even when acceptance succeeded.

### Best way to actually reproduce similar results
1. **First reproduce on H100 with strict parity**, then port.
2. Use **exact model/runtime path** from NVIDIA setup (English `nvidia/nemotron-speech-streaming-en-0.6b`, NeMo 25.11, chunk modes 160/320/560).
3. Use **single fixed GPU worker** (no autoscale churn during benchmark).
4. Run **3-minute real-time WS clients** (127 at 560ms; sweep for 320ms).
5. Measure per-stream: connect, first token, total, **audio_timestamp drift**, error rate.
6. Repeat each point at least 3 times and report median + p95/p99 + variance.
7. Only after H100 parity, migrate same binary/config to Vast (prefer fixed dedicated instance for benchmark, not elastic serverless).

If you want, I can execute this exact parity protocol next and give you a clean “how close are we to NVIDIA” report with pass/fail thresholds.