Output:
��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]


Output:
��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]

��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]


Output:
��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]


Output:
��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]


Output:
��████████████████    | 2.58G/2.87G [00:28<00:02, 123MiB/s]
 90%|██████████████████████████████████▏   | 2.59G/2.87G [00:29<00:02, 114MiB/s]
 90%|██████████████████████████████████▎   | 2.60G/2.87G [00:29<00:02, 108MiB/s]
 91%|██████████████████████████████████▌   | 2.61G/2.87G [00:29<00:02, 111MiB/s]
 91%|██████████████████████████████████▋   | 2.62G/2.87G [00:29<00:02, 114MiB/s]
 92%|██████████████████████████████████▊   | 2.63G/2.87G [00:29<00:02, 121MiB/s]
 92%|██████████████████████████████████▉   | 2.65G/2.87G [00:29<00:01, 125MiB/s]
 92%|███████████████████████████████████▏  | 2.66G/2.87G [00:29<00:01, 125MiB/s]
 93%|███████████████████████████████████▎  | 2.67G/2.87G [00:29<00:01, 121MiB/s]
 93%|███████████████████████████████████▍  | 2.68G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▌  | 2.69G/2.87G [00:29<00:01, 114MiB/s]
 94%|███████████████████████████████████▋  | 2.70G/2.87G [00:30<00:01, 111MiB/s]
 94%|███████████████████████████████████▉  | 2.72G/2.87G [00:30<00:01, 116MiB/s]
 95%|████████████████████████████████████  | 2.73G/2.87G [00:30<00:01, 122MiB/s]
 95%|████████████████████████████████████▏ | 2.74G/2.87G [00:30<00:01, 119MiB/s]
 96%|████████████████████████████████████▎ | 2.75G/2.87G [00:30<00:01, 122MiB/s]
 96%|████████████████████████████████████▌ | 2.76G/2.87G [00:30<00:00, 125MiB/s]
 97%|████████████████████████████████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]


Output:
  Per-Language Breakdown:

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================

��██████████▋ | 2.78G/2.87G [00:30<00:00, 124MiB/s]
 97%|████████████████████████████████████▊ | 2.79G/2.87G [00:30<00:00, 122MiB/s]
 97%|█████████████████████████████████████ | 2.80G/2.87G [00:30<00:00, 125MiB/s]
 98%|█████████████████████████████████████▏| 2.81G/2.87G [00:31<00:00, 121MiB/s]
 98%|█████████████████████████████████████▎| 2.82G/2.87G [00:31<00:00, 111MiB/s]
 99%|█████████████████████████████████████▍| 2.84G/2.87G [00:31<00:00, 117MiB/s]
 99%|█████████████████████████████████████▌| 2.85G/2.87G [00:31<00:00, 118MiB/s]
100%|█████████████████████████████████████▊| 2.86G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████▉| 2.87G/2.87G [00:31<00:00, 127MiB/s]
100%|█████████████████████████████████████| 2.87G/2.87G [00:31<00:00, 97.7MiB/s]
  Whisper loaded in 46.4s

[3/3] Running evaluation on 12 test cases...
------------------------------------------------------------

  [1/12] en_simple (English)
    Input: The quick brown fox jumps over the lazy dog near the riverbank.
    ERROR: HTTP Error 403: Forbidden

  [2/12] en_medium (English)
    Input: Artificial intelligence is transforming how we approach complex problems in scie...
    ERROR: HTTP Error 403: Forbidden

  [3/12] en_numbers (English)
    Input: In two thousand twenty-four, approximately three point seven billion people used...
    ERROR: HTTP Error 403: Forbidden

  [4/12] en_punctuation (English)
    Input: She said, 'I can't believe it!' and then whispered, 'This changes everything we ...
    ERROR: HTTP Error 403: Forbidden

  [5/12] en_medical (English)
    Input: Dr. Smith prescribed two hundred and fifty milligrams of amoxicillin three times...
    ERROR: HTTP Error 403: Forbidden

  [6/12] en_proper_nouns (English)
    Input: The Massachusetts Institute of Technology announced groundbreaking research in s...
    ERROR: HTTP Error 403: Forbidden

  [7/12] en_address (English)
    Input: Please navigate to one twenty-three Main Street, apartment four B, New York City...
    ERROR: HTTP Error 403: Forbidden

  [8/12] en_financial (English)
    Input: The stock price fluctuated between forty-seven dollars and sixty-three cents and...
    ERROR: HTTP Error 403: Forbidden

  [9/12] zh_simple (Chinese)
    Input: 人工智能正在深刻改变着我们的日常生活和工作方式。
    ERROR: HTTP Error 403: Forbidden

  [10/12] zh_historical (Chinese)
    Input: 中华人民共和国于一九四九年十月一日正式成立。
    ERROR: HTTP Error 403: Forbidden

  [11/12] ja_simple (Japanese)
    Input: 東京は日本の首都であり、世界で最も人口の多い都市の一つです。
    ERROR: HTTP Error 403: Forbidden

  [12/12] ko_simple (Korean)
    Input: 인공지능 기술은 우리의 일상생활을 크게 변화시키고 있습니다.
    ERROR: HTTP Error 403: Forbidden

============================================================
  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base
============================================================
  GPU:           NVIDIA A100-SXM4-80GB
  Precision:     bfloat16 (max accuracy)
  Transcriber:   Whisper large-v2
  Test cases:    0/12 successful
  Avg WER:       -1.00%
  Avg CER:       -1.00%
  Avg RTF:       -1.000

  Per-Language Breakdown:

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================


Output:
  Per-Language Breakdown:

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================


<persisted-output>
Output too large (36.5KB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/byh6msiqo.txt

Preview (first 2KB):
********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 67650.06it/s]
  TTS model loaded in 3.1s

[2/3] Loading Whisper large-v3 for transcription (max accuracy)...

  0%|                                              | 0.00/2.87G [00:00<?, ?iB/s]
  0%|                                     | 8.59M/2.87G [00:00<00:34, 89.6MiB/s]
  1%|▏                                    | 17.1M/2.87G [00:00<00:37, 82.9MiB/s]
  1%|▎                                    | 25.7M/2.87G [00:00<00:35, 85.6MiB/s]
  1%|▍                                    | 33.9M/2.87G [00:00<00:36, 82.6MiB/s]
  1%|▌                                    | 41.9M/2.87G [00:00<00:36, 83.0MiB/s]
  2%|▋                                    | 49.8M/2.87G [00:00<00:36, 82.1MiB/s]
  2%|▋                                    | 57.6M/2.87G [00:00<00:38, 77.8MiB/s]
  2%|▊                                    | 69.5M/2.87G [00:00<00:32, 91.9MiB/s]
  3%|▉                                    | 78.8M/2.87G [00:00<00:32, 93.5MiB/s]
  3%|█                                    | 88.7M/2.87G [00:01<00:31, 95.6MiB/s]
  3%|█▏                                   | 97.9M/2.87G [00:01<00:31, 95.6MiB/s]
  4%|█▍                                     | 109M/2.87G [00:01<00:29, 102MiB/s]
  4%|█▌                                     | 119M/2.87G [00:01<00:29, 102MiB/s]
  4%|█▋                                    | 129M/2.87G [00:01<00:30, 98.3MiB/s]
  5%|█▊                                    | 138M/2.87G [00:01<00:30, 96.5MiB/s]
  5%|█▉                                    | 148M/2.87G [00:01<00:30, 97.3MiB/s]
  5%|██
...
</persisted-output>


Bash

grep -A 50 "FINAL RESULTS" /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/byh6msiqo.txt

Extract final results


  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base
============================================================
  GPU:           NVIDIA A100-SXM4-80GB
  Precision:     bfloat16 (max accuracy)
  Transcriber:   Whisper large-v2
  Test cases:    0/12 successful
  Avg WER:       -1.00%
  Avg CER:       -1.00%
  Avg RTF:       -1.000

  Per-Language Breakdown:

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================


All 12 failed. Let me check what errors happened.


grep -E "ERROR|error|Error" /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/byh6msiqo.txt | head -20

Check errors from eval


    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden
    ERROR: HTTP Error 403: Forbidden


The reference audio URL is 403 forbidden. Let me download it differently or create a local reference audio.


# Try to get a reference audio - generate one with the model itself or find an alternative
# First let's check if the model can work without voice cloning (using a built-in speaker)
python3 -c "
from qwen_tts import Qwen3TTSModel
m = Qwen3TTSModel.from_pretrained('Qwen/Qwen3-TTS-12Hz-0.6B-Base', device_map='cuda:0', dtype='bfloat16')
print('Supported languages:', m.get_supported_languages())
print('Supported speakers:', m.get_supported_speakers())
" 2>&1 | grep -v Warning | grep -v "^\*"

Check model capabilities without ref audio


Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 44501.90it/s]
Supported languages: ['auto', 'chinese', 'english', 'french', 'german', 'italian', 'japanese', 'korean', 'portuguese', 'russian', 'spanish']
Supported speakers: []


No built-in speakers (it's a base model, voice-clone only). Let me create a reference audio locally using a simple approach, or try downloading from a different source.


# Try downloading the ref audio with wget/curl with user-agent
wget -q "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone.wav" -O /home/ubuntu/quench_0.6b/ref_audio.wav --user-agent="Mozilla/5.0" 2>&1; ls -la /home/ubuntu/quench_0.6b/ref_audio.wav 2>/dev/null

Download reference audio with user-agent


-rw-rw-r-- 1 ubuntu ubuntu 0 Mar 21 11:37 /home/ubuntu/quench_0.6b/ref_audio.wav


Empty file. Let me try from the GitHub repo or generate a reference audio using another TTS.


# Check if there's sample audio in the qwen3-tts github repo
curl -sL "https://api.github.com/repos/QwenLM/Qwen3-TTS/contents/" 2>/dev/null | python3 -c "import sys,json; [print(f['name']) for f in json.load(sys.stdin)]" 2>/dev/null

Check Qwen3-TTS repo contents


.github
.gitignore
LICENSE
MANIFEST.in
README.md
assets
examples
finetuning
pyproject.toml
qwen_tts


curl -sL "https://api.github.com/repos/QwenLM/Qwen3-TTS/contents/examples" 2>/dev/null | python3 -c "import sys,json; [print(f['name'], f.get('download_url','')) for f in json.load(sys.stdin)]" 2>/dev/null

Check examples directory for sample audio


Check official example code for the base model


# coding=utf-8
# Copyright 2026 The Alibaba Qwen team.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import time
import torch
import soundfile as sf

from qwen_tts import Qwen3TTSModel

def ensure_dir(d: str):
    os.makedirs(d, exist_ok=True)

def run_case(tts: Qwen3TTSModel, out_dir: str, case_name: str, call_fn):
    torch.cuda.synchronize()
    t0 = time.time()

    wavs, sr = call_fn()

    torch.cuda.synchronize()
    t1 = time.time()
    print(f"[{case_name}] time: {t1 - t0:.3f}s, n_wavs={len(wavs)}, sr={sr}")

    for i, w in enumerate(wavs):
        sf.write(os.path.join(out_dir, f"{case_name}_{i}.wav"), w, sr)

def main():
    device = "cuda:0"
    MODEL_PATH = "Qwen/Qwen3-TTS-12Hz-1.7B-Base/"
    OUT_DIR = "qwen3_tts_test_voice_clone_output_wav"
    ensure_dir(OUT_DIR)

    tts = Qwen3TTSModel.from_pretrained(
        MODEL_PATH,
        device_map=device,
        dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
    )

    # Reference audio(s)
    ref_audio_path_1 = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_2.wav"
    ref_audio_path_2 = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone_1.wav"

    ref_audio_single = ref_audio_path_1
    ref_audio_batch = [ref_audio_path_1, ref_audio_path_2]

    ref_text_single = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."
    ref_text_batch = [
        "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you.",
        "甚至出现交易几乎停滞的情况。",
    ]

    # Synthesis targets
    syn_text_single = "Good one. Okay, fine, I'm just gonna leave this sock monkey here. Goodbye."
    syn_lang_single = "Auto"

    syn_text_batch = [
        "Good one. Okay, fine, I'm just gonna leave this sock monkey here. Goodbye.",
        "其实我真的有发现，我是一个特别善于观察别人情绪的人。",
    ]
    syn_lang_batch = ["Chinese", "English"]

    common_gen_kwargs = dict(
        max_new_tokens=2048,
        do_sample=True,
        top_k=50,
        top_p=1.0,
        temperature=0.9,
        repetition_penalty=1.05,
        subtalker_dosample=True,
        subtalker_top_k=50,
        subtalker_top_p=1.0,
        subtalker_temperature=0.9,
    )

    for xvec_only in [False, True]:
        mode_tag = "xvec_only" if xvec_only else "icl"

        # Case 1: prompt single + synth single, direct
        run_case(
            tts, OUT_DIR, f"case1_promptSingle_synSingle_direct_{mode_tag}",
            lambda: tts.generate_voice_clone(
                text=syn_text_single,
                language=syn_lang_single,
                ref_audio=ref_audio_single,
                ref_text=ref_text_single,
                x_vector_only_mode=xvec_only,
                **common_gen_kwargs,
            ),
        )

        # Case 1b: prompt single + synth single, via create_voice_clone_prompt
        def _case1b():
            prompt_items = tts.create_voice_clone_prompt(
                ref_audio=ref_audio_single,
                ref_text=ref_text_single,
                x_vector_only_mode=xvec_only,
            )
            return tts.generate_voice_clone(
                text=syn_text_single,
                language=syn_lang_single,
                voice_clone_prompt=prompt_items,
                **common_gen_kwargs,
            )

        run_case(
            tts, OUT_DIR, f"case1_promptSingle_synSingle_promptThenGen_{mode_tag}",
            _case1b,
        )

        # Case 2: prompt single + synth batch, direct
        run_case(
            tts, OUT_DIR, f"case2_promptSingle_synBatch_direct_{mode_tag}",
            lambda: tts.generate_voice_clone(
                text=syn_text_batch,
                language=syn_lang_batch,
                ref_audio=ref_audio_single,
                ref_text=ref_text_single,
                x_vector_only_mode=xvec_only,
                **common_gen_kwargs,
            ),
        )

        # Case 2b: prompt single + synth batch, via create_voice_clone_prompt
        def _case2b():
            prompt_items = tts.create_voice_clone_prompt(
                ref_audio=ref_audio_single,
                ref_text=ref_text_single,
                x_vector_only_mode=xvec_only,
            )
            return tts.generate_voice_clone(
                text=syn_text_batch,
                language=syn_lang_batch,
                voice_clone_prompt=prompt_items,
                **common_gen_kwargs,
            )

        run_case(
            tts, OUT_DIR, f"case2_promptSingle_synBatch_promptThenGen_{mode_tag}",
            _case2b,
        )

        # Case 3: prompt batch + synth batch, direct
        run_case(
            tts, OUT_DIR, f"case3_promptBatch_synBatch_direct_{mode_tag}",
            lambda: tts.generate_voice_clone(
                text=syn_text_batch,
                language=syn_lang_batch,
                ref_audio=ref_audio_batch,
                ref_text=ref_text_batch,
                x_vector_only_mode=[xvec_only, xvec_only],
                **common_gen_kwargs,
            ),
        )

        # Case 3b: prompt batch + synth batch, via create_voice_clone_prompt
        def _case3b():
            prompt_items = tts.create_voice_clone_prompt(
                ref_audio=ref_audio_batch,
                ref_text=ref_text_batch,
                x_vector_only_mode=[xvec_only, xvec_only],
            )
            return tts.generate_voice_clone(
                text=syn_text_batch,
                language=syn_lang_batch,
                voice_clone_prompt=prompt_items,
                **common_gen_kwargs,
            )

        run_case(
            tts, OUT_DIR, f"case3_promptBatch_synBatch_promptThenGen_{mode_tag}",
            _case3b,
        )

if __name__ == "__main__":
    main()


The official example also uses the same OSS URLs. The 403 is likely a network/region issue. Let me create a reference audio locally using a simple tone or download a sample from another source.


# Generate a simple reference audio with speech-like characteristics using torchaudio
# Or better: try the x_vector_only_mode which might need less reference fidelity
# First let's try creating a simple reference wav with espeak or gtts
pip install gtts 2>&1 | tail -3

Install gTTS for reference audio generation


wandb 0.25.1 requires protobuf!=5.28.0,!=5.29.0,<7,>4.21.0, but you have protobuf 7.34.0 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed click-8.1.8 gtts-2.5.4


python3 -c "
from gtts import gTTS
import subprocess
# Generate reference speech
tts = gTTS('Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you.', lang='en')
tts.save('/home/ubuntu/quench_0.6b/ref_audio.mp3')
# Convert to wav
subprocess.run(['ffmpeg', '-y', '-i', '/home/ubuntu/quench_0.6b/ref_audio.mp3', '-ar', '24000', '-ac', '1', '/home/ubuntu/quench_0.6b/ref_audio.wav'], capture_output=True)
import soundfile as sf
data, sr = sf.read('/home/ubuntu/quench_0.6b/ref_audio.wav')
print(f'Reference audio: {len(data)/sr:.2f}s, sr={sr}, shape={data.shape}')
" 2>&1

Generate reference audio using gTTS


Reference audio: 8.78s, sr=24000, shape=(210816,)


Reference audio created. Now let me update the eval script to use the local file.


The file /home/ubuntu/quench_0.6b/eval_tts.py has been updated successfully.

"""
Qwen3-TTS-12Hz-0.6B-Base — Maximum Accuracy Evaluation
GPU: A100-80GB | dtype: bfloat16
Metrics: Word Error Rate (WER), Character Error Rate (CER), Speaker Similarity
"""

import torch
import soundfile as sf
import numpy as np
import os
import time
import json
import whisper
from jiwer import wer, cer
from qwen_tts import Qwen3TTSModel

OUTPUT_DIR = "/home/ubuntu/quench_0.6b/results"
os.makedirs(OUTPUT_DIR, exist_ok=True)

# ─────────────────────────────────────────────
# 1. Load Models
# ─────────────────────────────────────────────
print("=" * 60)
print("  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval")
print("=" * 60)

print("\n[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...")
t0 = time.time()
tts_model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
print(f"  TTS model loaded in {time.time()-t0:.1f}s")

print("\n[2/3] Loading Whisper large-v3 for transcription (max accuracy)...")
t0 = time.time()
whisper_model = whisper.load_model("large-v2", device="cuda:0")
print(f"  Whisper loaded in {time.time()-t0:.1f}s")

# ─────────────────────────────────────────────
# 2. Test Sentences (multi-language, varying complexity)
# ─────────────────────────────────────────────
test_cases = [
    # English
    {
        "text": "The quick brown fox jumps over the lazy dog near the riverbank.",
        "language": "English",
        "id": "en_simple"
    },
    {
        "text": "Artificial intelligence is transforming how we approach complex problems in science and engineering.",
        "language": "English",
        "id": "en_medium"
    },
    {
        "text": "In two thousand twenty-four, approximately three point seven billion people used smartphones worldwide.",
        "language": "English",
        "id": "en_numbers"
    },
    {
        "text": "She said, 'I can't believe it!' and then whispered, 'This changes everything we know about quantum mechanics.'",
        "language": "English",
        "id": "en_punctuation"
    },
    {
        "text": "Dr. Smith prescribed two hundred and fifty milligrams of amoxicillin three times daily for seven days.",
        "language": "English",
        "id": "en_medical"
    },
    {
        "text": "The Massachusetts Institute of Technology announced groundbreaking research in superconductivity.",
        "language": "English",
        "id": "en_proper_nouns"
    },
    {
        "text": "Please navigate to one twenty-three Main Street, apartment four B, New York City.",
        "language": "English",
        "id": "en_address"
    },
    {
        "text": "The stock price fluctuated between forty-seven dollars and sixty-three cents and fifty-two dollars and eighteen cents throughout the trading session.",
        "language": "English",
        "id": "en_financial"
    },
    # Chinese
    {
        "text": "人工智能正在深刻改变着我们的日常生活和工作方式。",
        "language": "Chinese",
        "id": "zh_simple"
    },
    {
        "text": "中华人民共和国于一九四九年十月一日正式成立。",
        "language": "Chinese",
        "id": "zh_historical"
    },
    # Japanese
    {
        "text": "東京は日本の首都であり、世界で最も人口の多い都市の一つです。",
        "language": "Japanese",
        "id": "ja_simple"
    },
    # Korean
    {
        "text": "인공지능 기술은 우리의 일상생활을 크게 변화시키고 있습니다.",
        "language": "Korean",
        "id": "ko_simple"
    },
]

# Reference audio for voice cloning
REF_AUDIO = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

# ─────────────────────────────────────────────
# 3. Run Evaluation
# ─────────────────────────────────────────────
print(f"\n[3/3] Running evaluation on {len(test_cases)} test cases...")
print("-" * 60)

results = []
total_wer = 0.0
total_cer_val = 0.0
count_wer = 0

for i, tc in enumerate(test_cases):
    text = tc["text"]
    lang = tc["language"]
    tid = tc["id"]

    print(f"\n  [{i+1}/{len(test_cases)}] {tid} ({lang})")
    print(f"    Input: {text[:80]}{'...' if len(text)>80 else ''}")

    t0 = time.time()
    try:
        wavs, sr = tts_model.generate_voice_clone(
            text=text,
            language=lang,
            ref_audio=REF_AUDIO,
            ref_text=REF_TEXT,
            non_streaming_mode=True,
        )
        gen_time = time.time() - t0
        audio = wavs[0]

        # Save audio
        wav_path = os.path.join(OUTPUT_DIR, f"{tid}.wav")
        sf.write(wav_path, audio, sr)

        audio_duration = len(audio) / sr
        rtf = gen_time / audio_duration  # Real-time factor

        # Transcribe with Whisper large-v3
        whisper_result = whisper_model.transcribe(
            wav_path,
            language=lang[:2].lower() if lang != "Chinese" else "zh",
            beam_size=5,
            best_of=5,
            temperature=0.0,
        )
        transcription = whisper_result["text"].strip()

        # Compute WER and CER
        if lang in ["Chinese", "Japanese", "Korean"]:
            # For CJK, CER is more meaningful
            c = cer(text, transcription)
            w = wer(text, transcription)
            metric_name = "CER"
            metric_val = c
        else:
            w = wer(text, transcription)
            c = cer(text, transcription)
            metric_name = "WER"
            metric_val = w

        total_wer += w
        total_cer_val += c
        count_wer += 1

        print(f"    Output: {transcription[:80]}{'...' if len(transcription)>80 else ''}")
        print(f"    {metric_name}: {metric_val*100:.2f}% | WER: {w*100:.2f}% | CER: {c*100:.2f}%")
        print(f"    Audio: {audio_duration:.2f}s | Gen: {gen_time:.2f}s | RTF: {rtf:.3f}")

        results.append({
            "id": tid,
            "language": lang,
            "input_text": text,
            "transcription": transcription,
            "wer": round(w * 100, 2),
            "cer": round(c * 100, 2),
            "audio_duration_s": round(audio_duration, 2),
            "generation_time_s": round(gen_time, 2),
            "rtf": round(rtf, 3),
            "wav_file": wav_path,
        })

    except Exception as e:
        print(f"    ERROR: {e}")
        results.append({
            "id": tid,
            "language": lang,
            "input_text": text,
            "error": str(e),
        })

# ─────────────────────────────────────────────
# 4. Summary
# ─────────────────────────────────────────────
avg_wer = (total_wer / count_wer * 100) if count_wer > 0 else -1
avg_cer = (total_cer_val / count_wer * 100) if count_wer > 0 else -1

successful = [r for r in results if "error" not in r]
avg_rtf = np.mean([r["rtf"] for r in successful]) if successful else -1

summary = {
    "model": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    "gpu": "NVIDIA A100-SXM4-80GB",
    "dtype": "bfloat16",
    "whisper_model": "large-v2",
    "total_test_cases": len(test_cases),
    "successful": len(successful),
    "avg_wer_pct": round(avg_wer, 2),
    "avg_cer_pct": round(avg_cer, 2),
    "avg_rtf": round(avg_rtf, 3),
    "results": results,
}

# Per-language breakdown
lang_results = {}
for r in successful:
    lang = r["language"]
    if lang not in lang_results:
        lang_results[lang] = {"wer": [], "cer": [], "rtf": []}
    lang_results[lang]["wer"].append(r["wer"])
    lang_results[lang]["cer"].append(r["cer"])
    lang_results[lang]["rtf"].append(r["rtf"])

summary["per_language"] = {}
for lang, vals in lang_results.items():
    summary["per_language"][lang] = {
        "avg_wer_pct": round(np.mean(vals["wer"]), 2),
        "avg_cer_pct": round(np.mean(vals["cer"]), 2),
        "avg_rtf": round(np.mean(vals["rtf"]), 3),
        "num_samples": len(vals["wer"]),
    }

# Save results
results_path = os.path.join(OUTPUT_DIR, "evaluation_results.json")
with open(results_path, "w") as f:
    json.dump(summary, f, indent=2, ensure_ascii=False)

print("\n" + "=" * 60)
print("  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base")
print("=" * 60)
print(f"  GPU:           NVIDIA A100-SXM4-80GB")
print(f"  Precision:     bfloat16 (max accuracy)")
print(f"  Transcriber:   Whisper large-v2")
print(f"  Test cases:    {len(successful)}/{len(test_cases)} successful")
print(f"  Avg WER:       {avg_wer:.2f}%")
print(f"  Avg CER:       {avg_cer:.2f}%")
print(f"  Avg RTF:       {avg_rtf:.3f}")
print()
print("  Per-Language Breakdown:")
for lang, vals in summary["per_language"].items():
    print(f"    {lang:12s}  WER: {vals['avg_wer_pct']:6.2f}%  CER: {vals['avg_cer_pct']:6.2f}%  RTF: {vals['avg_rtf']:.3f}  (n={vals['num_samples']})")
print()
print(f"  Results saved to: {results_path}")
print("=" * 60)


Run TTS evaluation with local reference audio


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

============================================================
  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval
============================================================

[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 100%|██████████| 4/4 [00:00<00:00, 15462.87it/s]
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:2150 for open-end generation.


Output:
    Korean        WER:   0.00%  CER:   0.00%  RTF: 1.765  (n=1)

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================

lish)
    Input: Artificial intelligence is transforming how we approach complex problems in scie...
    Output: Artificial intelligence is transforming how we approach complex problems in scie...
    WER: 0.00% | WER: 0.00% | CER: 0.00%
    Audio: 5.84s | Gen: 10.36s | RTF: 1.775

  [3/12] en_numbers (English)
    Input: In two thousand twenty-four, approximately three point seven billion people used...
    Output: In 2024, approximately 3.7 billion people used smartphones worldwide.
    WER: 46.15% | WER: 46.15% | CER: 39.81%
    Audio: 5.60s | Gen: 9.97s | RTF: 1.781

  [4/12] en_punctuation (English)
    Input: She said, 'I can't believe it!' and then whispered, 'This changes everything we ...
    Output: She said, I can't believe it, and then whispered, this changes everything we kno...
    WER: 23.53% | WER: 23.53% | CER: 5.45%
    Audio: 6.32s | Gen: 11.15s | RTF: 1.764

  [5/12] en_medical (English)
    Input: Dr. Smith prescribed two hundred and fifty milligrams of amoxicillin three times...
    Output: Dr. Smith prescribed 250 mg of amoxicillin three times daily for seven days.
    WER: 31.25% | WER: 31.25% | CER: 28.43%
    Audio: 6.24s | Gen: 10.98s | RTF: 1.760

  [6/12] en_proper_nouns (English)
    Input: The Massachusetts Institute of Technology announced groundbreaking research in s...
    Output: The Massachusetts Institute of Technology announced groundbreaking research in s...
    WER: 0.00% | WER: 0.00% | CER: 0.00%
    Audio: 5.68s | Gen: 10.00s | RTF: 1.760

  [7/12] en_address (English)
    Input: Please navigate to one twenty-three Main Street, apartment four B, New York City...
    Output: Please navigate to 123 Main Street, apartment 4B, New York City.
    WER: 30.77% | WER: 30.77% | CER: 25.93%
    Audio: 5.12s | Gen: 9.02s | RTF: 1.761

  [8/12] en_financial (English)
    Input: The stock price fluctuated between forty-seven dollars and sixty-three cents and...
    Output: The stock price fluctuated between $47.63 and $52.18 throughout the trading sess...
    WER: 50.00% | WER: 50.00% | CER: 51.68%
    Audio: 8.16s | Gen: 14.31s | RTF: 1.754

  [9/12] zh_simple (Chinese)
    Input: 人工智能正在深刻改变着我们的日常生活和工作方式。
    Output: 人工智能正在深刻改變著我們的日常生活和工作方式
    CER: 16.67% | WER: 100.00% | CER: 16.67%
    Audio: 4.48s | Gen: 7.92s | RTF: 1.768

  [10/12] zh_historical (Chinese)
    Input: 中华人民共和国于一九四九年十月一日正式成立。
    Output: 中華人民共和國於1949年10月1日正式成立
    CER: 50.00% | WER: 100.00% | CER: 50.00%
    Audio: 4.16s | Gen: 7.36s | RTF: 1.770

  [11/12] ja_simple (Japanese)
    Input: 東京は日本の首都であり、世界で最も人口の多い都市の一つです。
    Output: 東京は日本の首都であり、世界で最も人口の多い都市の一つです。
    CER: 0.00% | WER: 0.00% | CER: 0.00%
    Audio: 5.36s | Gen: 9.44s | RTF: 1.761

  [12/12] ko_simple (Korean)
    Input: 인공지능 기술은 우리의 일상생활을 크게 변화시키고 있습니다.
    Output: 인공지능 기술은 우리의 일상생활을 크게 변화시키고 있습니다.
    CER: 0.00% | WER: 0.00% | CER: 0.00%
    Audio: 4.64s | Gen: 8.19s | RTF: 1.765

============================================================
  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base
============================================================
  GPU:           NVIDIA A100-SXM4-80GB
  Precision:     bfloat16 (max accuracy)
  Transcriber:   Whisper large-v2
  Test cases:    12/12 successful
  Avg WER:       31.81%
  Avg CER:       18.16%
  Avg RTF:       1.796

  Per-Language Breakdown:
    English       WER:  22.71%  CER:  18.91%  RTF: 1.811  (n=8)
    Chinese       WER: 100.00%  CER:  33.34%  RTF: 1.769  (n=2)
    Japanese      WER:   0.00%  CER:   0.00%  RTF: 1.761  (n=1)
    Korean        WER:   0.00%  CER:   0.00%  RTF: 1.765  (n=1)

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================


Output:
    Korean        WER:   0.00%  CER:   0.00%  RTF: 1.765  (n=1)

  Results saved to: /home/ubuntu/quench_0.6b/results/evaluation_results.json
============================================================