Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — CONCURRENT Max Accuracy Eval
============================================================

[1/2] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 63072.24it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — CONCURRENT Max Accuracy Eval
============================================================

[1/2] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 63072.24it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — CONCURRENT Max Accuracy Eval
============================================================

[1/2] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 63072.24it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
  Spanish          0.00%     0.00%     0.00%     0.00%  1.797    1

  Results + audio: /home/ubuntu/quench_0.6b/results/
=================================================================

Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
  TTS model + voice prompt cached in 4.5s

[2/2] Loading Whisper large-v2 for transcription...
  Whisper loaded in 15.5s

Running CONCURRENT evaluation on 21 test cases...
------------------------------------------------------------

  BATCH [English] — 10 samples
    Batch TTS done in 12.18s (10 samples)
    en_pangram            PERFECT     dur:4.2s
    en_tech               PERFECT     dur:5.9s
    en_proper_nouns       PERFECT     dur:5.7s
    en_complex            PERFECT     dur:5.3s
    en_weather            PERFECT     dur:5.0s
    en_arts               PERFECT     dur:5.8s
    en_energy             PERFECT     dur:6.1s
    en_philosophy         PERFECT     dur:5.7s
    en_global             PERFECT     dur:6.1s
    en_food               PERFECT     dur:6.2s
    Whisper done in 14.61s

  BATCH [Chinese] — 4 samples
    Batch TTS done in 9.18s (4 samples)
    zh_ai                 PERFECT     dur:4.8s
    zh_weather            WER:100.0%  dur:4.2s
    zh_science            WER:100.0%  dur:3.8s
    zh_culture            WER:100.0%  dur:3.9s
    Whisper done in 5.94s

  BATCH [Japanese] — 2 samples
    Batch TTS done in 9.81s (2 samples)
    ja_tokyo              PERFECT     dur:5.1s
    ja_cherry             PERFECT     dur:4.6s
    Whisper done in 3.40s

  BATCH [Korean] — 2 samples
    Batch TTS done in 7.63s (2 samples)
    ko_ai                 PERFECT     dur:4.0s
    ko_weather            PERFECT     dur:3.8s
    Whisper done in 2.79s

  BATCH [German] — 1 samples
    Batch TTS done in 6.81s (1 samples)
    de_science            PERFECT     dur:3.8s
    Whisper done in 1.33s

  BATCH [French] — 1 samples
    Batch TTS done in 7.93s (1 samples)
    fr_tech               PERFECT     dur:4.4s
    Whisper done in 1.21s

  BATCH [Spanish] — 1 samples
    Batch TTS done in 11.07s (1 samples)
    es_ai                 PERFECT     dur:6.2s
    Whisper done in 1.48s

=================================================================
  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base (CONCURRENT)
  Max Accuracy on NVIDIA A100-SXM4-80GB
=================================================================
  Precision:     bfloat16 (no quantization)
  Mode:          Batch TTS generation per language
  Transcriber:   Whisper large-v2 (beam=5, best_of=5)
  Test cases:    21/21 successful
  Perfect (0%):  18/21 (85.7%)

  Timing:
    Total wall time:     95.4s
    TTS generation:      64.6s
    Audio produced:      104.4s
    Overall TTS RTF:     0.619

  Accuracy (Normalized):
    WER:  14.29%
    CER:  2.89%

  Accuracy (Raw):
    WER:  20.41%
    CER:  6.29%

  Language       NormWER   NormCER    RawWER    RawCER     RTF    n
  ------------  --------  --------  --------  --------  ------  ---
  Chinese         75.00%    15.20%   100.00%    32.02%  0.556    4
  English          0.00%     0.00%     2.87%     0.39%  0.221   10
  French           0.00%     0.00%     0.00%     0.00%  1.803    1
  German           0.00%     0.00%     0.00%     0.00%  1.812    1
  Japanese         0.00%     0.00%     0.00%     0.00%  1.007    2
  Korean           0.00%     0.00%     0.00%     0.00%  0.973    2
  Spanish          0.00%     0.00%     0.00%     0.00%  1.797    1

  Results + audio: /home/ubuntu/quench_0.6b/results/
=================================================================


Output:
  Spanish          0.00%     0.00%     0.00%     0.00%  1.797    1

  Results + audio: /home/ubuntu/quench_0.6b/results/
=================================================================


********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — CONCURRENT Max Accuracy Eval
============================================================

[1/2] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 63072.24it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
  TTS model + voice prompt cached in 4.5s

[2/2] Loading Whisper large-v2 for transcription...
  Whisper loaded in 15.5s

Running CONCURRENT evaluation on 21 test cases...
------------------------------------------------------------

  BATCH [English] — 10 samples
    Batch TTS done in 12.18s (10 samples)
    en_pangram            PERFECT     dur:4.2s
    en_tech               PERFECT     dur:5.9s
    en_proper_nouns       PERFECT     dur:5.7s
    en_complex            PERFECT     dur:5.3s
    en_weather            PERFECT     dur:5.0s
    en_arts               PERFECT     dur:5.8s
    en_energy             PERFECT     dur:6.1s
    en_philosophy         PERFECT     dur:5.7s
    en_global             PERFECT     dur:6.1s
    en_food               PERFECT     dur:6.2s
    Whisper done in 14.61s

  BATCH [Chinese] — 4 samples
    Batch TTS done in 9.18s (4 samples)
    zh_ai                 PERFECT     dur:4.8s
    zh_weather            WER:100.0%  dur:4.2s
    zh_science            WER:100.0%  dur:3.8s
    zh_culture            WER:100.0%  dur:3.9s
    Whisper done in 5.94s

  BATCH [Japanese] — 2 samples
    Batch TTS done in 9.81s (2 samples)
    ja_tokyo              PERFECT     dur:5.1s
    ja_cherry             PERFECT     dur:4.6s
    Whisper done in 3.40s

  BATCH [Korean] — 2 samples
    Batch TTS done in 7.63s (2 samples)
    ko_ai                 PERFECT     dur:4.0s
    ko_weather            PERFECT     dur:3.8s
    Whisper done in 2.79s

  BATCH [German] — 1 samples
    Batch TTS done in 6.81s (1 samples)
    de_science            PERFECT     dur:3.8s
    Whisper done in 1.33s

  BATCH [French] — 1 samples
    Batch TTS done in 7.93s (1 samples)
    fr_tech               PERFECT     dur:4.4s
    Whisper done in 1.21s

  BATCH [Spanish] — 1 samples
    Batch TTS done in 11.07s (1 samples)
    es_ai                 PERFECT     dur:6.2s
    Whisper done in 1.48s

=================================================================
  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base (CONCURRENT)
  Max Accuracy on NVIDIA A100-SXM4-80GB
=================================================================
  Precision:     bfloat16 (no quantization)
  Mode:          Batch TTS generation per language
  Transcriber:   Whisper large-v2 (beam=5, best_of=5)
  Test cases:    21/21 successful
  Perfect (0%):  18/21 (85.7%)

  Timing:
    Total wall time:     95.4s
    TTS generation:      64.6s
    Audio produced:      104.4s
    Overall TTS RTF:     0.619

  Accuracy (Normalized):
    WER:  14.29%
    CER:  2.89%

  Accuracy (Raw):
    WER:  20.41%
    CER:  6.29%

  Language       NormWER   NormCER    RawWER    RawCER     RTF    n
  ------------  --------  --------  --------  --------  ------  ---
  Chinese         75.00%    15.20%   100.00%    32.02%  0.556    4
  English          0.00%     0.00%     2.87%     0.39%  0.221   10
  French           0.00%     0.00%     0.00%     0.00%  1.803    1
  German           0.00%     0.00%     0.00%     0.00%  1.812    1
  Japanese         0.00%     0.00%     0.00%     0.00%  1.007    2
  Korean           0.00%     0.00%     0.00%     0.00%  0.973    2
  Spanish          0.00%     0.00%     0.00%     0.00%  1.797    1

  Results + audio: /home/ubuntu/quench_0.6b/results/
=================================================================


Massive speedup with concurrency. Here's the comparison:

### Sequential vs Concurrent

| Metric | Sequential | Concurrent | Speedup |
|--------|-----------|------------|---------|
| **Total wall time** | ~190s | **95.4s** | **2x** |
| **TTS generation** | ~190s | **64.6s** | **2.9x** |
| **Overall TTS RTF** | 1.770 | **0.619** | **2.9x faster** |
| **English RTF** | 1.784 | **0.221** | **8x faster** |

Key wins:
- **Batch TTS generation** — 10 English samples generated in 12.18s total (vs ~100s sequential). The model processes the batch internally on the GPU.
- **Voice prompt cached once** — `create_voice_clone_prompt()` called once and reused for all batches, eliminating redundant ref audio processing.
- **English RTF dropped to 0.221** — generating speech 4.5x faster than real-time.
- **Accuracy unchanged** — 18/21 perfect, same 3 Chinese samples with traditional/simplified mismatch from Whisper.


just tst concunnecy and rtf just use english ... and can we use any infernce engines


The user wants me to:
1. Test only English for concurrency/RTF
2. Try inference engines (like vLLM, TensorRT, etc.) to push RTF even lower

Let me check what inference engines could work with Qwen3-TTS. vLLM might support it, or we could try TensorRT-LLM, or the `faster-qwen3-tts` package I saw earlier as a dependency.


Let me check what inference engines are available for Qwen3-TTS, then blast English-only RTF tests.


Bash

pip show faster-qwen3-tts 2>/dev/null; python3 -c "import faster_qwen3_tts; print(dir(faster_qwen3_tts))" 2>/dev/null; pip list 2>/dev/null | grep -iE "faster-qwen|vllm|tensorrt|onnx|triton"

Check available inference engines


Output:
License: MIT
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, qwen-tts, soundfile, torch, transformers
Required-by:

Name: faster-qwen3-tts
Version: 0.2.4
Summary: Real-time Qwen3-TTS inference using manual CUDA graph capture
Home-page: https://github.com/andimarafioti/faster-qwen3-tts
Author: Andres Marafioti
Author-email:
License: MIT
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, qwen-tts, soundfile, torch, transformers
Required-by:


Output:
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, qwen-tts, soundfile, torch, transformers
Required-by:
['FasterQwen3TTS', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'model', 'utils']

Name: faster-qwen3-tts
Version: 0.2.4
Summary: Real-time Qwen3-TTS inference using manual CUDA graph capture
Home-page: https://github.com/andimarafioti/faster-qwen3-tts
Author: Andres Marafioti
Author-email:
License: MIT
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, qwen-tts, soundfile, torch, transformers
Required-by:
['FasterQwen3TTS', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'model', 'utils']


Name: faster-qwen3-tts
Version: 0.2.4
Summary: Real-time Qwen3-TTS inference using manual CUDA graph capture
Home-page: https://github.com/andimarafioti/faster-qwen3-tts
Author: Andres Marafioti
Author-email:
License: MIT
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: huggingface-hub, numpy, qwen-tts, soundfile, torch, transformers
Required-by:
['FasterQwen3TTS', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'model', 'utils']
faster-qwen3-tts                         0.2.4
onnx                                     1.20.1
onnx-ir                                  0.1.16
onnxruntime                              1.23.2
onnxruntime-gpu                          1.23.2
onnxscript                               0.6.2
tensorrt                                 10.14.1.48.post1
tensorrt_cu12                            10.15.1.29
tensorrt_cu12_bindings                   10.15.1.29
tensorrt_cu12_libs                       10.15.1.29
tensorrt_cu13                            10.14.1.48.post1
tensorrt_cu13_bindings                   10.14.1.48.post1
tensorrt_cu13_libs                       10.14.1.48.post1
torch_tensorrt                           2.10.0
triton                                   3.5.1
vllm                                     0.15.1
vllm-omni                                0.16.0


`faster-qwen3-tts` with CUDA graphs is already installed. Plus we have vLLM and TensorRT. Let me test all engines head-to-head.


python3 -c "
from faster_qwen3_tts import FasterQwen3TTS
import inspect
sig = inspect.signature(FasterQwen3TTS.__init__)
print('FasterQwen3TTS init:', sig)
methods = [m for m in dir(FasterQwen3TTS) if not m.startswith('_')]
print('Methods:', methods)
" 2>&1 | grep -v Warning

Check FasterQwen3TTS API


FasterQwen3TTS init: (self, base_model, predictor_graph, talker_graph, device: str = 'cuda', dtype: torch.dtype = torch.bfloat16, max_seq_len: int = 2048)
Methods: ['from_pretrained', 'generate', 'generate_custom_voice', 'generate_custom_voice_streaming', 'generate_voice_clone', 'generate_voice_clone_streaming', 'generate_voice_design', 'generate_voice_design_streaming']


python3 -c "
from faster_qwen3_tts import FasterQwen3TTS
import inspect
sig = inspect.signature(FasterQwen3TTS.from_pretrained)
print('from_pretrained:', sig)
sig2 = inspect.signature(FasterQwen3TTS.generate_voice_clone)
print('generate_voice_clone:', sig2)
" 2>&1 | grep -v Warning

Check FasterQwen3TTS signatures


from_pretrained: (model_name: str, device: str = 'cuda', dtype: Union[str, torch.dtype] = torch.bfloat16, attn_implementation: str = 'sdpa', max_seq_len: int = 2048)
generate_voice_clone: (self, text: str, language: str, ref_audio: Union[str, pathlib.Path], ref_text: str, max_new_tokens: int = 2048, min_new_tokens: int = 2, temperature: float = 0.9, top_k: int = 50, top_p: float = 1.0, do_sample: bool = True, repetition_penalty: float = 1.05, xvec_only: bool = True, non_streaming_mode: bool = True, append_silence: bool = True) -> Tuple[list, int]


Write

"""
Qwen3-TTS-12Hz-0.6B-Base — RTF Benchmark (English only)
Tests: baseline (qwen_tts) vs faster-qwen3-tts (CUDA graphs)
"""

import torch
import soundfile as sf
import numpy as np
import os
import time
import json

OUTPUT_DIR = "/home/ubuntu/quench_0.6b/results"
os.makedirs(OUTPUT_DIR, exist_ok=True)

REF_AUDIO = "/home/ubuntu/quench_0.6b/ref_audio.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

# English test sentences — variety of lengths
SENTENCES = [
    "The quick brown fox jumps over the lazy dog near the riverbank.",
    "Artificial intelligence is transforming how we approach complex problems in science and engineering.",
    "The Massachusetts Institute of Technology announced groundbreaking research in superconductivity.",
    "She couldn't believe it and then whispered this changes everything we know about quantum mechanics.",
    "Beautiful weather today with clear skies and a gentle breeze coming from the northwest.",
    "The orchestra performed a magnificent symphony that left the audience speechless and deeply moved.",
    "Renewable energy sources such as solar and wind power are becoming increasingly important worldwide.",
    "The philosopher argued that consciousness emerges from the complex interactions of neural networks.",
    "International cooperation is essential for addressing global challenges like climate change and poverty.",
    "The restaurant serves traditional Mediterranean cuisine with fresh ingredients sourced from local farmers.",
    "Machine learning algorithms can identify patterns in large datasets that would be impossible for humans to detect manually.",
    "The astronaut described the view of Earth from space as the most humbling experience of her entire career.",
    "Quantum computing promises to revolutionize cryptography and drug discovery within the next decade.",
    "The ancient library contained thousands of manuscripts dating back to the early medieval period.",
    "Global temperatures have risen significantly over the past century due to increased greenhouse gas emissions.",
]

NUM_WARMUP = 2
NUM_RUNS = 3  # run each sentence this many times for stable RTF

def benchmark_engine(name, generate_fn, sentences, warmup=NUM_WARMUP, runs=NUM_RUNS):
    """Benchmark a TTS engine. Returns list of per-sentence results."""
    print(f"\n  Warmup ({warmup} sentences)...")
    for i in range(warmup):
        torch.cuda.synchronize()
        _ = generate_fn(sentences[i % len(sentences)])
        torch.cuda.synchronize()

    print(f"  Running {len(sentences)} sentences x {runs} runs...")
    results = []
    for si, text in enumerate(sentences):
        times = []
        durations = []
        for r in range(runs):
            torch.cuda.synchronize()
            t0 = time.time()
            wavs, sr = generate_fn(text)
            torch.cuda.synchronize()
            elapsed = time.time() - t0
            audio_dur = len(wavs[0]) / sr
            times.append(elapsed)
            durations.append(audio_dur)

        avg_time = np.mean(times)
        avg_dur = np.mean(durations)
        rtf = avg_time / avg_dur
        results.append({
            "text": text,
            "avg_gen_s": round(avg_time, 3),
            "avg_audio_s": round(avg_dur, 2),
            "rtf": round(rtf, 4),
            "tokens_per_s": round(len(text) / avg_time, 1),
        })
        print(f"    [{si+1:2d}/{len(sentences)}] RTF={rtf:.4f}  gen={avg_time:.2f}s  audio={avg_dur:.1f}s  | {text[:50]}...")

    rtfs = [r["rtf"] for r in results]
    total_gen = sum(r["avg_gen_s"] for r in results)
    total_audio = sum(r["avg_audio_s"] for r in results)
    return {
        "engine": name,
        "num_sentences": len(sentences),
        "num_runs": runs,
        "avg_rtf": round(np.mean(rtfs), 4),
        "median_rtf": round(np.median(rtfs), 4),
        "min_rtf": round(np.min(rtfs), 4),
        "max_rtf": round(np.max(rtfs), 4),
        "std_rtf": round(np.std(rtfs), 4),
        "total_gen_s": round(total_gen, 2),
        "total_audio_s": round(total_audio, 2),
        "overall_rtf": round(total_gen / total_audio, 4),
        "per_sentence": results,
    }

# ═══════════════════════════════════════════════
print("=" * 65)
print("  Qwen3-TTS RTF Benchmark — English Only")
print("  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16")
print("=" * 65)

all_results = {}

# ─────────────────────────────────────────────
# Engine 1: Baseline qwen_tts (sequential)
# ─────────────────────────────────────────────
print("\n[ENGINE 1] qwen_tts (baseline, sequential)")
from qwen_tts import Qwen3TTSModel

tts_base = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
voice_prompt = tts_base.create_voice_clone_prompt(ref_audio=REF_AUDIO, ref_text=REF_TEXT)

def gen_baseline(text):
    return tts_base.generate_voice_clone(
        text=text, language="English",
        voice_clone_prompt=voice_prompt,
        non_streaming_mode=True,
    )

r1 = benchmark_engine("qwen_tts_baseline", gen_baseline, SENTENCES)
all_results["qwen_tts_baseline"] = r1

# Free baseline model
del tts_base, voice_prompt
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Engine 1b: Baseline qwen_tts (BATCH)
# ─────────────────────────────────────────────
print("\n[ENGINE 1b] qwen_tts (batch generation)")
tts_batch = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
voice_prompt_b = tts_batch.create_voice_clone_prompt(ref_audio=REF_AUDIO, ref_text=REF_TEXT)

print(f"  Running full batch of {len(SENTENCES)} sentences x {NUM_RUNS} runs...")
batch_times = []
batch_audio_durs = []
for r in range(NUM_WARMUP + NUM_RUNS):
    torch.cuda.synchronize()
    t0 = time.time()
    wavs, sr = tts_batch.generate_voice_clone(
        text=SENTENCES,
        language=["English"] * len(SENTENCES),
        voice_clone_prompt=voice_prompt_b,
        non_streaming_mode=True,
    )
    torch.cuda.synchronize()
    elapsed = time.time() - t0
    total_audio = sum(len(w) / sr for w in wavs)

    if r >= NUM_WARMUP:
        batch_times.append(elapsed)
        batch_audio_durs.append(total_audio)
        print(f"    Run {r-NUM_WARMUP+1}/{NUM_RUNS}: gen={elapsed:.2f}s  audio={total_audio:.1f}s  RTF={elapsed/total_audio:.4f}")

avg_batch_time = np.mean(batch_times)
avg_batch_audio = np.mean(batch_audio_durs)
r1b = {
    "engine": "qwen_tts_batch",
    "num_sentences": len(SENTENCES),
    "num_runs": NUM_RUNS,
    "batch_size": len(SENTENCES),
    "avg_total_gen_s": round(avg_batch_time, 2),
    "avg_total_audio_s": round(avg_batch_audio, 2),
    "overall_rtf": round(avg_batch_time / avg_batch_audio, 4),
    "per_sample_gen_s": round(avg_batch_time / len(SENTENCES), 3),
}
all_results["qwen_tts_batch"] = r1b

del tts_batch, voice_prompt_b
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Engine 2: faster-qwen3-tts (CUDA graphs)
# ─────────────────────────────────────────────
print("\n[ENGINE 2] faster-qwen3-tts (CUDA graphs)")
from faster_qwen3_tts import FasterQwen3TTS

tts_fast = FasterQwen3TTS.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device="cuda",
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

def gen_fast(text):
    return tts_fast.generate_voice_clone(
        text=text, language="English",
        ref_audio=REF_AUDIO, ref_text=REF_TEXT,
        non_streaming_mode=True,
    )

r2 = benchmark_engine("faster_qwen3_tts", gen_fast, SENTENCES)
all_results["faster_qwen3_tts"] = r2

del tts_fast
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Final Comparison
# ─────────────────────────────────────────────
results_path = os.path.join(OUTPUT_DIR, "rtf_benchmark.json")
with open(results_path, "w") as f:
    json.dump(all_results, f, indent=2)

print("\n" + "=" * 65)
print("  RTF BENCHMARK RESULTS — Qwen3-TTS-12Hz-0.6B-Base")
print("  GPU: NVIDIA A100-SXM4-80GB | English Only")
print("=" * 65)
print()
print(f"  {'Engine':30s}  {'RTF':>8s}  {'Gen/sample':>10s}  {'Note':s}")
print(f"  {'-'*30}  {'-'*8}  {'-'*10}  {'-'*20}")
print(f"  {'qwen_tts (sequential)':30s}  {r1['overall_rtf']:8.4f}  {r1['total_gen_s']/len(SENTENCES):9.2f}s  baseline")
print(f"  {'qwen_tts (batch x' + str(len(SENTENCES)) + ')':30s}  {r1b['overall_rtf']:8.4f}  {r1b['per_sample_gen_s']:9.3f}s  batched")
print(f"  {'faster-qwen3-tts (CUDA graphs)':30s}  {r2['overall_rtf']:8.4f}  {r2['total_gen_s']/len(SENTENCES):9.2f}s  CUDA graphs")
print()

best_engine = min(
    [("baseline_seq", r1["overall_rtf"]),
     ("baseline_batch", r1b["overall_rtf"]),
     ("faster_cuda_graphs", r2["overall_rtf"])],
    key=lambda x: x[1]
)
print(f"  BEST ENGINE: {best_engine[0]} with RTF = {best_engine[1]:.4f}")
speedup_over_baseline = r1["overall_rtf"] / best_engine[1]
print(f"  Speedup over sequential baseline: {speedup_over_baseline:.2f}x")
print()
print(f"  RTF < 1.0 = faster than real-time")
print(f"  Results saved to: {results_path}")
print("=" * 65)


File created successfully at: /home/ubuntu/quench_0.6b/bench_rtf.py

"""
Qwen3-TTS-12Hz-0.6B-Base — RTF Benchmark (English only)
Tests: baseline (qwen_tts) vs faster-qwen3-tts (CUDA graphs)
"""

import torch
import soundfile as sf
import numpy as np
import os
import time
import json

OUTPUT_DIR = "/home/ubuntu/quench_0.6b/results"
os.makedirs(OUTPUT_DIR, exist_ok=True)

REF_AUDIO = "/home/ubuntu/quench_0.6b/ref_audio.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

# English test sentences — variety of lengths
SENTENCES = [
    "The quick brown fox jumps over the lazy dog near the riverbank.",
    "Artificial intelligence is transforming how we approach complex problems in science and engineering.",
    "The Massachusetts Institute of Technology announced groundbreaking research in superconductivity.",
    "She couldn't believe it and then whispered this changes everything we know about quantum mechanics.",
    "Beautiful weather today with clear skies and a gentle breeze coming from the northwest.",
    "The orchestra performed a magnificent symphony that left the audience speechless and deeply moved.",
    "Renewable energy sources such as solar and wind power are becoming increasingly important worldwide.",
    "The philosopher argued that consciousness emerges from the complex interactions of neural networks.",
    "International cooperation is essential for addressing global challenges like climate change and poverty.",
    "The restaurant serves traditional Mediterranean cuisine with fresh ingredients sourced from local farmers.",
    "Machine learning algorithms can identify patterns in large datasets that would be impossible for humans to detect manually.",
    "The astronaut described the view of Earth from space as the most humbling experience of her entire career.",
    "Quantum computing promises to revolutionize cryptography and drug discovery within the next decade.",
    "The ancient library contained thousands of manuscripts dating back to the early medieval period.",
    "Global temperatures have risen significantly over the past century due to increased greenhouse gas emissions.",
]

NUM_WARMUP = 2
NUM_RUNS = 3  # run each sentence this many times for stable RTF

def benchmark_engine(name, generate_fn, sentences, warmup=NUM_WARMUP, runs=NUM_RUNS):
    """Benchmark a TTS engine. Returns list of per-sentence results."""
    print(f"\n  Warmup ({warmup} sentences)...")
    for i in range(warmup):
        torch.cuda.synchronize()
        _ = generate_fn(sentences[i % len(sentences)])
        torch.cuda.synchronize()

    print(f"  Running {len(sentences)} sentences x {runs} runs...")
    results = []
    for si, text in enumerate(sentences):
        times = []
        durations = []
        for r in range(runs):
            torch.cuda.synchronize()
            t0 = time.time()
            wavs, sr = generate_fn(text)
            torch.cuda.synchronize()
            elapsed = time.time() - t0
            audio_dur = len(wavs[0]) / sr
            times.append(elapsed)
            durations.append(audio_dur)

        avg_time = np.mean(times)
        avg_dur = np.mean(durations)
        rtf = avg_time / avg_dur
        results.append({
            "text": text,
            "avg_gen_s": round(avg_time, 3),
            "avg_audio_s": round(avg_dur, 2),
            "rtf": round(rtf, 4),
            "tokens_per_s": round(len(text) / avg_time, 1),
        })
        print(f"    [{si+1:2d}/{len(sentences)}] RTF={rtf:.4f}  gen={avg_time:.2f}s  audio={avg_dur:.1f}s  | {text[:50]}...")

    rtfs = [r["rtf"] for r in results]
    total_gen = sum(r["avg_gen_s"] for r in results)
    total_audio = sum(r["avg_audio_s"] for r in results)
    return {
        "engine": name,
        "num_sentences": len(sentences),
        "num_runs": runs,
        "avg_rtf": round(np.mean(rtfs), 4),
        "median_rtf": round(np.median(rtfs), 4),
        "min_rtf": round(np.min(rtfs), 4),
        "max_rtf": round(np.max(rtfs), 4),
        "std_rtf": round(np.std(rtfs), 4),
        "total_gen_s": round(total_gen, 2),
        "total_audio_s": round(total_audio, 2),
        "overall_rtf": round(total_gen / total_audio, 4),
        "per_sentence": results,
    }

# ═══════════════════════════════════════════════
print("=" * 65)
print("  Qwen3-TTS RTF Benchmark — English Only")
print("  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16")
print("=" * 65)

all_results = {}

# ─────────────────────────────────────────────
# Engine 1: Baseline qwen_tts (sequential)
# ─────────────────────────────────────────────
print("\n[ENGINE 1] qwen_tts (baseline, sequential)")
from qwen_tts import Qwen3TTSModel

tts_base = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
voice_prompt = tts_base.create_voice_clone_prompt(ref_audio=REF_AUDIO, ref_text=REF_TEXT)

def gen_baseline(text):
    return tts_base.generate_voice_clone(
        text=text, language="English",
        voice_clone_prompt=voice_prompt,
        non_streaming_mode=True,
    )

r1 = benchmark_engine("qwen_tts_baseline", gen_baseline, SENTENCES)
all_results["qwen_tts_baseline"] = r1

# Free baseline model
del tts_base, voice_prompt
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Engine 1b: Baseline qwen_tts (BATCH)
# ─────────────────────────────────────────────
print("\n[ENGINE 1b] qwen_tts (batch generation)")
tts_batch = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
voice_prompt_b = tts_batch.create_voice_clone_prompt(ref_audio=REF_AUDIO, ref_text=REF_TEXT)

print(f"  Running full batch of {len(SENTENCES)} sentences x {NUM_RUNS} runs...")
batch_times = []
batch_audio_durs = []
for r in range(NUM_WARMUP + NUM_RUNS):
    torch.cuda.synchronize()
    t0 = time.time()
    wavs, sr = tts_batch.generate_voice_clone(
        text=SENTENCES,
        language=["English"] * len(SENTENCES),
        voice_clone_prompt=voice_prompt_b,
        non_streaming_mode=True,
    )
    torch.cuda.synchronize()
    elapsed = time.time() - t0
    total_audio = sum(len(w) / sr for w in wavs)

    if r >= NUM_WARMUP:
        batch_times.append(elapsed)
        batch_audio_durs.append(total_audio)
        print(f"    Run {r-NUM_WARMUP+1}/{NUM_RUNS}: gen={elapsed:.2f}s  audio={total_audio:.1f}s  RTF={elapsed/total_audio:.4f}")

avg_batch_time = np.mean(batch_times)
avg_batch_audio = np.mean(batch_audio_durs)
r1b = {
    "engine": "qwen_tts_batch",
    "num_sentences": len(SENTENCES),
    "num_runs": NUM_RUNS,
    "batch_size": len(SENTENCES),
    "avg_total_gen_s": round(avg_batch_time, 2),
    "avg_total_audio_s": round(avg_batch_audio, 2),
    "overall_rtf": round(avg_batch_time / avg_batch_audio, 4),
    "per_sample_gen_s": round(avg_batch_time / len(SENTENCES), 3),
}
all_results["qwen_tts_batch"] = r1b

del tts_batch, voice_prompt_b
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Engine 2: faster-qwen3-tts (CUDA graphs)
# ─────────────────────────────────────────────
print("\n[ENGINE 2] faster-qwen3-tts (CUDA graphs)")
from faster_qwen3_tts import FasterQwen3TTS

tts_fast = FasterQwen3TTS.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device="cuda",
    dtype=torch.bfloat16,
    attn_implementation="sdpa",
)

def gen_fast(text):
    return tts_fast.generate_voice_clone(
        text=text, language="English",
        ref_audio=REF_AUDIO, ref_text=REF_TEXT,
        non_streaming_mode=True,
    )

r2 = benchmark_engine("faster_qwen3_tts", gen_fast, SENTENCES)
all_results["faster_qwen3_tts"] = r2

del tts_fast
torch.cuda.empty_cache()

# ─────────────────────────────────────────────
# Final Comparison
# ─────────────────────────────────────────────
results_path = os.path.join(OUTPUT_DIR, "rtf_benchmark.json")
with open(results_path, "w") as f:
    json.dump(all_results, f, indent=2)

print("\n" + "=" * 65)
print("  RTF BENCHMARK RESULTS — Qwen3-TTS-12Hz-0.6B-Base")
print("  GPU: NVIDIA A100-SXM4-80GB | English Only")
print("=" * 65)
print()
print(f"  {'Engine':30s}  {'RTF':>8s}  {'Gen/sample':>10s}  {'Note':s}")
print(f"  {'-'*30}  {'-'*8}  {'-'*10}  {'-'*20}")
print(f"  {'qwen_tts (sequential)':30s}  {r1['overall_rtf']:8.4f}  {r1['total_gen_s']/len(SENTENCES):9.2f}s  baseline")
print(f"  {'qwen_tts (batch x' + str(len(SENTENCES)) + ')':30s}  {r1b['overall_rtf']:8.4f}  {r1b['per_sample_gen_s']:9.3f}s  batched")
print(f"  {'faster-qwen3-tts (CUDA graphs)':30s}  {r2['overall_rtf']:8.4f}  {r2['total_gen_s']/len(SENTENCES):9.2f}s  CUDA graphs")
print()

best_engine = min(
    [("baseline_seq", r1["overall_rtf"]),
     ("baseline_batch", r1b["overall_rtf"]),
     ("faster_cuda_graphs", r2["overall_rtf"])],
    key=lambda x: x[1]
)
print(f"  BEST ENGINE: {best_engine[0]} with RTF = {best_engine[1]:.4f}")
speedup_over_baseline = r1["overall_rtf"] / best_engine[1]
print(f"  Speedup over sequential baseline: {speedup_over_baseline:.2f}x")
print()
print(f"  RTF < 1.0 = faster than real-time")
print(f"  Results saved to: {results_path}")
print("=" * 65)


Bash

Run RTF benchmark: baseline vs batch vs CUDA graphs


Output:
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]


Output:
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]


Output:
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]


Output:
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

=================================================================
  Qwen3-TTS RTF Benchmark — English Only
  GPU: NVIDIA A100-SXM4-80GB | dtype: bfloat16
=================================================================

[ENGINE 1] qwen_tts (baseline, sequential)

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 55370.35it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.