Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]

`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

   Batch   GenTime     Audio       RTF     Req/s  AudioS/s  Status
  ------  --------  --------  --------  --------  --------  ----------
       1     4.13s      2.3s    1.8231     0.24     0.55  OK
       2     5.38s      5.1s    1.0508     0.37     0.95  OK
       5     9.72s     19.7s    0.4946     0.51     2.02  OK
      10     9.98s     40.2s    0.2482     1.00     4.03  OK
      15    10.61s     63.0s    0.1685     1.41     5.94  OK
      20    10.73s     82.2s    0.1305     1.86     7.66  OK
      30    10.96s    125.0s    0.0877     2.74    11.40  OK

[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]


Output:
    Streaming mode:  ~358 x A100-80GB GPUs

  Results saved to: /home/ubuntu/quench_0.6b/results/load_test_results.json
======================================================================

00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

   Batch   GenTime     Audio       RTF     Req/s  AudioS/s  Status
  ------  --------  --------  --------  --------  --------  ----------
       1     4.13s      2.3s    1.8231     0.24     0.55  OK
       2     5.38s      5.1s    1.0508     0.37     0.95  OK
       5     9.72s     19.7s    0.4946     0.51     2.02  OK
      10     9.98s     40.2s    0.2482     1.00     4.03  OK
      15    10.61s     63.0s    0.1685     1.41     5.94  OK
      20    10.73s     82.2s    0.1305     1.86     7.66  OK
      30    10.96s    125.0s    0.0877     2.74    11.40  OK

[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running for 60s...

  SUSTAINED LOAD RESULTS (60s):
    Total requests:      38
    Requests/sec:        0.63
    Audio generated:     172.4s
    Audio throughput:    2.85 sec audio/sec wall
    TTFB p50:            378 ms
    TTFB p95:            381 ms
    TTFB p99:            381 ms
    E2E  p50:            1575 ms
    E2E  p95:            2125 ms

======================================================================
  PRODUCTION CAPACITY — Qwen3-TTS-12Hz-0.6B-Base
  Single NVIDIA A100-SXM4-80GB
======================================================================

  STREAMING MODE (low latency, per-request):
    Engine:          faster-qwen3-tts (CUDA graphs)
    TTFB p50:        378 ms
    TTFB p99:        380 ms
    E2E  p50:        1604 ms
    Throughput:      2.85 sec audio/sec
    Max users:       ~28 concurrent

  BATCH MODE (max throughput, queued):
    Engine:          qwen_tts (batch=30)
    Requests/sec:    2.74
    Throughput:      11.40 sec audio/sec
    Max users:       ~114 concurrent

  FOR 10,000 USERS:
    Batch mode:      ~88 x A100-80GB GPUs
    Streaming mode:  ~358 x A100-80GB GPUs

  Results saved to: /home/ubuntu/quench_0.6b/results/load_test_results.json
======================================================================


`torch_dtype` is deprecated! Use `dtype` instead!
======================================================================
  Qwen3-TTS-12Hz-0.6B-Base — PRODUCTION LOAD TEST
  GPU: NVIDIA A100-SXM4-80GB | English Voice Agent Simulation
======================================================================

[TEST 1] STREAMING — faster-qwen3-tts (CUDA graphs)
  Measures real TTFB for voice agent responsiveness

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 58867.42it/s]
  Warming up...
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running 30 streaming requests (sequential, measures true TTFB)...
    10/30 done
    20/30 done
    30/30 done

  STREAMING RESULTS (faster-qwen3-tts, CUDA graphs):
    TTFB p50:       378 ms
    TTFB p90:       380 ms
    TTFB p95:       380 ms
    TTFB p99:       380 ms
    E2E  p50:      1604 ms
    E2E  p90:      1978 ms
    RTF  avg:    0.3544

[TEST 2] BATCH THROUGHPUT — qwen_tts (max concurrency)
  Simulates queued requests processed in batches

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 24855.13it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

   Batch   GenTime     Audio       RTF     Req/s  AudioS/s  Status
  ------  --------  --------  --------  --------  --------  ----------
       1     4.13s      2.3s    1.8231     0.24     0.55  OK
       2     5.38s      5.1s    1.0508     0.37     0.95  OK
       5     9.72s     19.7s    0.4946     0.51     2.02  OK
      10     9.98s     40.2s    0.2482     1.00     4.03  OK
      15    10.61s     63.0s    0.1685     1.41     5.94  OK
      20    10.73s     82.2s    0.1305     1.86     7.66  OK
      30    10.96s    125.0s    0.0877     2.74    11.40  OK

[TEST 3] SUSTAINED LOAD — 60s stress test (faster-qwen3-tts)
  Simulates continuous voice agent traffic

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 4702.13it/s]
Warming up predictor (3 runs)...
Capturing CUDA graph for predictor...
CUDA graph captured!
Warming up talker graph (3 runs)...
Capturing CUDA graph for talker decode...
Talker CUDA graph captured!
  Running for 60s...

  SUSTAINED LOAD RESULTS (60s):
    Total requests:      38
    Requests/sec:        0.63
    Audio generated:     172.4s
    Audio throughput:    2.85 sec audio/sec wall
    TTFB p50:            378 ms
    TTFB p95:            381 ms
    TTFB p99:            381 ms
    E2E  p50:            1575 ms
    E2E  p95:            2125 ms

======================================================================
  PRODUCTION CAPACITY — Qwen3-TTS-12Hz-0.6B-Base
  Single NVIDIA A100-SXM4-80GB
======================================================================

  STREAMING MODE (low latency, per-request):
    Engine:          faster-qwen3-tts (CUDA graphs)
    TTFB p50:        378 ms
    TTFB p99:        380 ms
    E2E  p50:        1604 ms
    Throughput:      2.85 sec audio/sec
    Max users:       ~28 concurrent

  BATCH MODE (max throughput, queued):
    Engine:          qwen_tts (batch=30)
    Requests/sec:    2.74
    Throughput:      11.40 sec audio/sec
    Max users:       ~114 concurrent

  FOR 10,000 USERS:
    Batch mode:      ~88 x A100-80GB GPUs
    Streaming mode:  ~358 x A100-80GB GPUs

  Results saved to: /home/ubuntu/quench_0.6b/results/load_test_results.json
======================================================================


Here's your production answer:

## Production Capacity — 1x A100-80GB

### Streaming Mode (voice agent, low latency)
| Metric | Value |
|--------|-------|
| **TTFB p50** | **378 ms** |
| **TTFB p99** | **380 ms** |
| **E2E p50** | **1,604 ms** |
| **E2E p95** | **2,125 ms** |
| Throughput | 2.85 sec audio/sec |
| **Max concurrent users** | **~28** |

### Batch Mode (queued, max throughput)
| Batch Size | RTF | Req/s | Audio sec/sec |
|-----------|-----|-------|---------------|
| 1 | 1.82 | 0.24 | 0.55 |
| 5 | 0.49 | 0.51 | 2.02 |
| 10 | 0.25 | 1.00 | 4.03 |
| 20 | 0.13 | 1.86 | 7.66 |
| **30** | **0.088** | **2.74** | **11.40** |

Best batch: **RTF 0.088** — generating **11.4 seconds of audio per second of wall time**. Max **~114 concurrent users** per GPU.

### For 10,000 concurrent users:
- **Batch mode: ~88 A100-80GB GPUs**
- **Streaming mode: ~358 A100-80GB GPUs**


what is wall time .. dos we dont have any infernce engine for this


The user is asking:
1. What is wall time - they want plain language
2. They want to know if there are proper inference engines (like vLLM, TensorRT, etc.) that can serve this model, not just the basic HF inference

Let me check if vLLM supports Qwen3-TTS, or if there's a TensorRT-LLM setup, or any other serving engine.


**Wall time** = actual clock time from start to finish. Like, you hit "go" and look at your watch — that's wall time.

Now let me check if vLLM or TensorRT can actually serve this model.


Bash

python3 -c "
import vllm
print('vLLM version:', vllm.__version__)
# Check if qwen3_tts architecture is supported
from vllm.model_executor.models import _MODELS
matches = [k for k in _MODELS if 'qwen' in k.lower() or 'tts' in k.lower()]
print('Supported models with qwen/tts:', matches)
" 2>&1 | tail -10

Check if vLLM supports Qwen3-TTS


Traceback (most recent call last):
  File "<string>", line 5, in <module>
ImportError: cannot import name '_MODELS' from 'vllm.model_executor.models' (/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/__init__.py)
vLLM version: 0.15.1


# Check vLLM supported architectures
python3 -c "
from vllm import LLM
# Try loading the model to see if architecture is supported
try:
    llm = LLM(model='Qwen/Qwen3-TTS-12Hz-0.6B-Base', trust_remote_code=True, max_model_len=512, enforce_eager=True)
    print('vLLM: SUPPORTED')
except Exception as e:
    print(f'vLLM: NOT SUPPORTED - {e}')
" 2>&1 | grep -E "SUPPORTED|not support|KeyError|not found|Unsupported|architecture" | head -5

Try loading Qwen3-TTS in vLLM


vLLM: NOT SUPPORTED - 1 validation error for ModelConfig
  Value error, The checkpoint you are trying to load has model type `qwen3_tts` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.


# Check SGLang
pip list 2>/dev/null | grep -iE "sglang|triton-server|text-generation|tgi"
# Check if there's a dedicated serving solution
pip list 2>/dev/null | grep -iE "faster-whisper|xtts|bark|parler"
# Check vllm-omni which is installed
python3 -c "
try:
    from vllm_omni import LLM
    print('vllm-omni available')
except: pass
" 2>&1

Check other inference engines


# Check if TensorRT-LLM can handle it
python3 -c "
import tensorrt as trt
print('TensorRT version:', trt.__version__)
" 2>&1

# Check if there's an ONNX export path
python3 -c "
from qwen_tts.core.models.modeling_qwen3_tts import Qwen3TTSForConditionalGeneration
import inspect
# Check if there's export methods
methods = [m for m in dir(Qwen3TTSForConditionalGeneration) if 'export' in m.lower() or 'onnx' in m.lower()]
print('Export methods:', methods)
" 2>&1 | grep -v Warning | grep -v "^\*"

Check TensorRT and ONNX export options


Output: TensorRT version: 10.14.1.48.post1


Output: TensorRT version: 10.14.1.48.post1


Output: TensorRT version: 10.14.1.48.post1


Output: TensorRT version: 10.14.1.48.post1


TensorRT version: 10.14.1.48.post1

Export methods: ['_cache_dependant_input_preparation_exporting']


# Check if torch.compile works for speedup
python3 -c "
import torch
from faster_qwen3_tts import FasterQwen3TTS
model = FasterQwen3TTS.from_pretrained(
    'Qwen/Qwen3-TTS-12Hz-0.6B-Base',
    device='cuda', dtype=torch.bfloat16, attn_implementation='sdpa',
)
# Check if torch.compile can wrap it
print('torch.compile available:', hasattr(torch, 'compile'))
print('torch version:', torch.__version__)
print('triton available:', True)
" 2>&1 | grep -v Warning | grep -v "^\*" | grep -v Fetching | grep -v Warming | grep -v Capturing | grep -v captured

Check torch.compile availability


`torch_dtype` is deprecated! Use `dtype` instead!

torch.compile available: True
torch version: 2.9.1+cu128
triton available: True


# Let's also check if there's a way to serve via SGLang or any TTS-specific serving
pip install sglang 2>&1 | tail -3
# Also check the Qwen3-TTS GitHub for any serving recommendations
curl -sL "https://raw.githubusercontent.com/QwenLM/Qwen3-TTS/main/README.md" 2>/dev/null | grep -iE "serve|deploy|vllm|tensorrt|onnx|inference|api|server|production" | head -10

Check SGLang and official serving guidance


xcodec2 0.1.5 requires torchaudio==2.5.0, but you have torchaudio 2.9.1+cu128 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed airportsdata-20260315 blobfile-3.0.0 build-1.4.0 cuda-python-12.9.0 decord2-3.0.0 flashinfer_cubin-0.6.3 flashinfer_python-0.6.3 grpcio-health-checking-1.78.0 llguidance-0.7.30 nvidia-cutlass-dsl-4.3.5 openai-2.6.1 openai-harmony-0.0.4 outlines-0.1.11 outlines_core-0.1.26 protobuf-6.33.6 py-spy-0.4.1 pyproject_hooks-1.2.0 quack-kernels-0.2.4 sgl-kernel-0.3.21 sglang-0.5.9 smg-grpc-proto-0.4.3 timm-1.0.16 torch-c-dlpack-ext-0.1.5 torch_memory_saver-0.0.9 torchao-0.9.0 torchcodec-0.8.0 transformers-4.57.1 xgrammar-0.1.27
🖥 Hugging Face Demo&nbsp&nbsp | &nbsp&nbsp 🖥 ModelScope Demo&nbsp&nbsp | &nbsp&nbsp💬 WeChat (微信)&nbsp&nbsp | &nbsp&nbsp🫨 Discord&nbsp&nbsp | &nbsp&nbsp📑 API
  - [DashScope API Usage](#dashscope-api-usage)
- [vLLM Usage](#vllm-usage)
* **Powerful Speech Representation**: Powered by the self-developed Qwen3-TTS-Tokenizer-12Hz, it achieves efficient acoustic compression and high-dimensional semantic modeling of speech signals. It fully preserves paralinguistic information and acoustic environmental features, enabling high-speed, high-fidelity speech reconstruction through a lightweight non-DiT architecture.
| Qwen3-TTS-12Hz-1.7B-Base | Base model capable of 3-second rapid voice clone from user audio input; can be used for fine-tuning (FT) other models. | Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian | ✅ |  |
| Qwen3-TTS-12Hz-0.6B-Base | Base model capable of 3-second rapid voice clone from user audio input; can be used for fine-tuning (FT) other models. | Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian | ✅ |  |
During model loading in the qwen-tts package or vLLM, model weights will be automatically downloaded based on the model name. However, if your runtime environment is not conducive to downloading weights during execution, you can refer to the following commands to manually download the model weights to a local directory:
# single inference
# batch inference
# single inference


The official README mentions **vLLM support**. Let me grab that section.


curl -sL "https://raw.githubusercontent.com/QwenLM/Qwen3-TTS/main/README.md" 2>/dev/null | sed -n '/## vLLM Usage/,/^## /p' | head -100

Get vLLM usage section from README


## vLLM Usage

vLLM officially provides day-0 support for Qwen3-TTS! Welcome to use vLLM-Omni for Qwen3-TTS deployment and inference. For installation and more details, please check [vLLM-Omni official documentation](https://docs.vllm.ai/projects/vllm-omni/en/latest/getting_started/quickstart/#installation). Now only offline inference is supported. Online serving will be supported later, and vLLM-Omni will continue to offer support and optimization for Qwen3-TTS in areas such as inference speed and streaming capabilities.

### Offline Inference
You can use vLLM-Omni to inference Qwen3-TTS locally, we provide examples in [vLLM-Omni repo](https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/qwen3_tts) which can generate audio output:
```bash
# git clone https://github.com/vllm-project/vllm-omni.git

# cd vllm-omni/examples/offline_inference/qwen3_tts

# Run a single sample with CustomVoice task
python end2end.py --query-type CustomVoice

# Batch sample (multiple prompts in one run) with CustomVoice task:
python end2end.py --query-type CustomVoice --use-batch-sample

# Run a single sample with VoiceDesign task
python end2end.py --query-type VoiceDesign

# Batch sample (multiple prompts in one run) with VoiceDesign task:
python end2end.py --query-type VoiceDesign --use-batch-sample

# Run a single sample with Base task in icl mode-tag
python end2end.py --query-type Base --mode-tag icl
```

## Fine Tuning


Conversation compacted