<persisted-output>
Output too large (32.9KB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/bugvkd0y9.txt

Preview (first 2KB):
2026-03-21:10:37:11 INFO     [tasks:691]     Task: mmlu_professional_accounting (mmlu/default/mmlu_professional_accounting.yaml)
2026-03-21:10:37:11 WARNING  [evaluator:333] Overwriting default num_fewshot of mmlu_professional_accounting from None to 0
2026-03-21:10:37:20 INFO     [api.task:311] Building contexts for mmlu_professional_accounting on rank 0...
2026-03-21:10:37:44 INFO     [evaluator:584] Running generate_until requests

Running generate_until requests:   0%|          | 0/1319 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Running generate_until requests:   0%|          | 1/1319 [03:50<84:17:27, 230.23s/it]
Running generate_until requests:   0%|          | 2/1319 [03:58<36:21:58, 99.41s/it]
Running generate_until requests:   0%|          | 3/1319 [04:05<21:02:57, 57.58s/it]
Running generate_until requests:   0%|          | 4/1319 [04:13<13:51:34, 37.94s/it]
Running generate_until requests:   0%|          | 5/1319 [04:21<9:53:22, 27.09s/it]
Running generate_until requests:   0%|          | 6/1319 [04:29<7:29:10, 20.53s/it]
Running generate_until requests:   1%|          | 7/1319 [04:29<5:03:26, 13.88s/it]
Running generate_until requests:   1%|          | 8/1319 [04:37<4:20:42, 11.93s/it]
Running generate_until requests:   1%|          | 9/1319 [04:45<3:52:03, 10.63s/it]
Running generate_until requests:   1%|          | 10/1319 [04:52<3:32:41,  9.75s/it]
Running generate_until requests:   1%|          | 11/1319 [05:00<3:19:20,  9.14s/it]
Running generate_until requests:   1%|          | 12/1319 [05:08<3:10:08,  8.73s/it]
Running generate_until requests:   1%|          | 13/1319 [05:16<3:03:42,  8.44s/it]
Running generate_until requests:   1%|          | 14/1319 [05:23<2:59:09,  8.24s/it]
Running generate_until requests:   1%|          | 15/1319 [05:31<2:55:58,  8.10s/it]
Running generate_until requests:   1%|          | 16/1319 [05:
...
</persisted-output>

2026-03-21:10:37:11 INFO     [tasks:691]     Task: mmlu_professional_accounting (mmlu/default/mmlu_professional_accounting.yaml)
2026-03-21:10:37:11 WARNING  [evaluator:333] Overwriting default num_fewshot of mmlu_professional_accounting from None to 0
2026-03-21:10:37:20 INFO     [api.task:311] Building contexts for mmlu_professional_accounting on rank 0...
2026-03-21:10:37:44 INFO     [evaluator:584] Running generate_until requests

Running generate_until requests:   0%|          | 0/1319 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Running generate_until requests:   0%|          | 1/1319 [03:50<84:17:27, 230.23s/it]
Running generate_until requests:   0%|          | 2/1319 [03:58<36:21:58, 99.41s/it]
Running generate_until requests:   0%|          | 3/1319 [04:05<21:02:57, 57.58s/it]
Running generate_until requests:   0%|          | 4/1319 [04:13<13:51:34, 37.94s/it]
Running generate_until requests:   0%|          | 5/1319 [04:21<9:53:22, 27.09s/it]
Running generate_until requests:   0%|          | 6/1319 [04:29<7:29:10, 20.53s/it]
Running generate_until requests:   1%|          | 7/1319 [04:29<5:03:26, 13.88s/it]
Running generate_until requests:   1%|          | 8/1319 [04:37<4:20:42, 11.93s/it]
Running generate_until requests:   1%|          | 9/1319 [04:45<3:52:03, 10.63s/it]
Running generate_until requests:   1%|          | 10/1319 [04:52<3:32:41,  9.75s/it]
Running generate_until requests:   1%|          | 11/1319 [05:00<3:19:20,  9.14s/it]
Running generate_until requests:   1%|          | 12/1319 [05:08<3:10:08,  8.73s/it]
Running generate_until requests:   1%|          | 13/1319 [05:16<3:03:42,  8.44s/it]
Running generate_until requests:   1%|          | 14/1319 [05:23<2:59:09,  8.24s/it]
Running generate_until requests:   1%|          | 15/1319 [05:31<2:55:58,  8.10s/it]
Running generate_until requests:   1%|          | 16/1319 [05:39<2:53:40,  8.00s/it]
Running generate_until requests:   1%|▏         | 17/1319 [05:47<2:52:10,  7.93s/it]
Running generate_until requests:   1%|▏         | 18/1319 [05:55<2:51:07,  7.89s/it]
Running generate_until requests:   1%|▏         | 19/1319 [05:55<2:00:41,  5.57s/it]
Running generate_until requests:   2%|▏         | 20/1319 [06:03<2:15:04,  6.24s/it]
Running generate_until requests:   2%|▏         | 21/1319 [06:10<2:25:10,  6.71s/it]
Running generate_until requests:   2%|▏         | 22/1319 [06:18<2:32:08,  7.04s/it]
Running generate_until requests:   2%|▏         | 23/1319 [06:26<2:37:00,  7.27s/it]
Running generate_until requests:   2%|▏         | 24/1319 [06:34<2:40:19,  7.43s/it]
Running generate_until requests:   2%|▏         | 25/1319 [06:34<1:53:20,  5.26s/it]
Running generate_until requests:   2%|▏         | 26/1319 [06:42<2:09:42,  6.02s/it]
Running generate_until requests:   2%|▏         | 27/1319 [06:50<2:21:06,  6.55s/it]
Running generate_until requests:   2%|▏         | 28/1319 [06:57<2:29:03,  6.93s/it]
Running generate_until requests:   2%|▏         | 29/1319 [07:05<2:34:33,  7.19s/it]
Running generate_until requests:   2%|▏         | 30/1319 [07:13<2:38:18,  7.37s/it]
Running generate_until requests:   2%|▏         | 31/1319 [07:21<2:40:54,  7.50s/it]
Running generate_until requests:   2%|▏         | 32/1319 [07:29<2:42:52,  7.59s/it]
Running generate_until requests:   3%|▎         | 33/1319 [07:36<2:44:07,  7.66s/it]
Running generate_until requests:   3%|▎         | 34/1319 [07:44<2:45:04,  7.71s/it]
Running generate_until requests:   3%|▎         | 35/1319 [07:52<2:45:36,  7.74s/it]
Running generate_until requests:   3%|▎         | 36/1319 [08:00<2:45:55,  7.76s/it]
Running generate_until requests:   3%|▎         | 37/1319 [08:00<1:57:37,  5.51s/it]
Running generate_until requests:   3%|▎         | 38/1319 [08:00<1:23:16,  3.90s/it]
Running generate_until requests:   3%|▎         | 39/1319 [08:08<1:48:17,  5.08s/it]
Running generate_until requests:   3%|▎         | 40/1319 [08:08<1:17:07,  3.62s/it]
Running generate_until requests:   3%|▎         | 41/1319 [08:16<1:43:49,  4.87s/it]
Running generate_until requests:   3%|▎         | 42/1319 [08:24<2:02:24,  5.75s/it]
Running generate_until requests:   3%|▎         | 43/1319 [08:32<2:15:30,  6.37s/it]
Running generate_until requests:   3%|▎         | 44/1319 [08:32<1:36:08,  4.52s/it]
Running generate_until requests:   3%|▎         | 45/1319 [08:40<1:56:50,  5.50s/it]
Running generate_until requests:   3%|▎         | 46/1319 [08:47<2:11:08,  6.18s/it]
Running generate_until requests:   4%|▎         | 47/1319 [08:55<2:21:13,  6.66s/it]
Running generate_until requests:   4%|▎         | 48/1319 [09:03<2:28:15,  7.00s/it]
Running generate_until requests:   4%|▎         | 49/1319 [09:03<1:45:04,  4.96s/it]
Running generate_until requests:   4%|▍         | 50/1319 [09:11<2:02:51,  5.81s/it]
Running generate_until requests:   4%|▍         | 51/1319 [09:19<2:15:17,  6.40s/it]
Running generate_until requests:   4%|▍         | 52/1319 [09:27<2:23:48,  6.81s/it]
Running generate_until requests:   4%|▍         | 53/1319 [09:34<2:29:48,  7.10s/it]
Running generate_until requests:   4%|▍         | 54/1319 [09:42<2:33:59,  7.30s/it]
Running generate_until requests:   4%|▍         | 55/1319 [09:50<2:37:03,  7.46s/it]
Running generate_until requests:   4%|▍         | 56/1319 [09:50<1:51:01,  5.27s/it]
Running generate_until requests:   4%|▍         | 57/1319 [09:58<2:06:59,  6.04s/it]
Running generate_until requests:   4%|▍         | 58/1319 [10:06<2:18:15,  6.58s/it]
Running generate_until requests:   4%|▍         | 59/1319 [10:14<2:25:57,  6.95s/it]
Running generate_until requests:   5%|▍         | 60/1319 [10:21<2:31:11,  7.21s/it]
Running generate_until requests:   5%|▍         | 61/1319 [10:29<2:34:45,  7.38s/it]
Running generate_until requests:   5%|▍         | 62/1319 [10:37<2:37:14,  7.51s/it]
Running generate_until requests:   5%|▍         | 63/1319 [10:45<2:38:55,  7.59s/it]
Running generate_until requests:   5%|▍         | 64/1319 [10:52<2:39:59,  7.65s/it]
Running generate_until requests:   5%|▍         | 65/1319 [10:53<1:52:52,  5.40s/it]
Running generate_until requests:   5%|▌         | 66/1319 [11:00<2:07:42,  6.11s/it]
Running generate_until requests:   5%|▌         | 67/1319 [11:01<1:30:39,  4.34s/it]
Running generate_until requests:   5%|▌         | 68/1319 [11:08<1:52:06,  5.38s/it]
Running generate_until requests:   5%|▌         | 69/1319 [11:16<2:07:01,  6.10s/it]
Running generate_until requests:   5%|▌         | 70/1319 [11:16<1:30:11,  4.33s/it]
Running generate_until requests:   5%|▌         | 71/1319 [11:24<1:51:43,  5.37s/it]
Running generate_until requests:   5%|▌         | 72/1319 [11:32<2:06:44,  6.10s/it]
Running generate_until requests:   6%|▌         | 73/1319 [11:40<2:17:20,  6.61s/it]
Running generate_until requests:   6%|▌         | 74/1319 [11:48<2:24:42,  6.97s/it]
Running generate_until requests:   6%|▌         | 75/1319 [11:55<2:29:47,  7.22s/it]
Running generate_until requests:   6%|▌         | 76/1319 [11:56<1:46:18,  5.13s/it]
Running generate_until requests:   6%|▌         | 77/1319 [12:04<2:02:48,  5.93s/it]
Running generate_until requests:   6%|▌         | 78/1319 [12:11<2:14:11,  6.49s/it]
Running generate_until requests:   6%|▌         | 79/1319 [12:19<2:22:14,  6.88s/it]
Running generate_until requests:   6%|▌         | 80/1319 [12:27<2:27:47,  7.16s/it]
Running generate_until requests:   6%|▌         | 81/1319 [12:35<2:31:43,  7.35s/it]
Running generate_until requests:   6%|▌         | 82/1319 [12:43<2:34:31,  7.50s/it]
Running generate_until requests:   6%|▋         | 83/1319 [12:43<1:49:25,  5.31s/it]
Running generate_until requests:   6%|▋         | 84/1319 [12:43<1:17:51,  3.78s/it]
Running generate_until requests:   6%|▋         | 85/1319 [12:51<1:42:38,  4.99s/it]
Running generate_until requests:   7%|▋         | 86/1319 [12:59<1:59:52,  5.83s/it]
Running generate_until requests:   7%|▋         | 87/1319 [13:06<2:11:50,  6.42s/it]
Running generate_until requests:   7%|▋         | 88/1319 [13:14<2:20:07,  6.83s/it]
Running generate_until requests:   7%|▋         | 89/1319 [13:22<2:25:51,  7.12s/it]
Running generate_until requests:   7%|▋         | 90/1319 [13:30<2:29:47,  7.31s/it]
Running generate_until requests:   7%|▋         | 91/1319 [13:37<2:32:34,  7.46s/it]
Running generate_until requests:   7%|▋         | 92/1319 [13:45<2:34:24,  7.55s/it]
Running generate_until requests:   7%|▋         | 93/1319 [13:53<2:35:34,  7.61s/it]
Running generate_until requests:   7%|▋         | 94/1319 [13:53<1:50:18,  5.40s/it]
Running generate_until requests:   7%|▋         | 95/1319 [14:01<2:04:51,  6.12s/it]
Running generate_until requests:   7%|▋         | 96/1319 [14:09<2:15:02,  6.62s/it]
Running generate_until requests:   7%|▋         | 97/1319 [14:09<1:35:34,  4.69s/it]
Running generate_until requests:   7%|▋         | 98/1319 [14:09<1:07:58,  3.34s/it]
Running generate_until requests:   8%|▊         | 99/1319 [14:09<48:50,  2.40s/it]
Running generate_until requests:   8%|▊         | 100/1319 [14:17<1:21:39,  4.02s/it]
Running generate_until requests:   8%|▊         | 101/1319 [14:25<1:44:32,  5.15s/it]
Running generate_until requests:   8%|▊         | 102/1319 [14:33<2:00:15,  5.93s/it]
Running generate_until requests:   8%|▊         | 103/1319 [14:41<2:11:23,  6.48s/it]
Running generate_until requests:   8%|▊         | 104/1319 [14:48<2:19:03,  6.87s/it]
Running generate_until requests:   8%|▊         | 105/1319 [14:56<2:24:18,  7.13s/it]
Running generate_until requests:   8%|▊         | 106/1319 [14:56<1:42:02,  5.05s/it]
Running generate_until requests:   8%|▊         | 107/1319 [15:04<1:58:28,  5.86s/it]
Running generate_until requests:   8%|▊         | 108/1319 [15:04<1:24:31,  4.19s/it]
Running generate_until requests:   8%|▊         | 109/1319 [15:05<1:00:35,  3.00s/it]
Running generate_until requests:   8%|▊         | 110/1319 [15:12<1:29:20,  4.43s/it]
Running generate_until requests:   8%|▊         | 111/1319 [15:13<1:03:46,  3.17s/it]
Running generate_until requests:   8%|▊         | 112/1319 [15:20<1:31:27,  4.55s/it]
Running generate_until requests:   9%|▊         | 113/1319 [15:28<1:50:52,  5.52s/it]
Running generate_until requests:   9%|▊         | 114/1319 [15:36<2:04:29,  6.20s/it]
Running generate_until requests:   9%|▊         | 115/1319 [15:36<1:28:22,  4.40s/it]
Running generate_until requests:   9%|▉         | 116/1319 [15:44<1:48:32,  5.41s/it]
Running generate_until requests:   9%|▉         | 117/1319 [15:52<2:02:47,  6.13s/it]
Running generate_until requests:   9%|▉         | 118/1319 [15:59<2:12:30,  6.62s/it]
Running generate_until requests:   9%|▉         | 119/1319 [16:00<1:34:09,  4.71s/it]
Running generate_until requests:   9%|▉         | 120/1319 [16:07<1:52:27,  5.63s/it]
Running generate_until requests:   9%|▉         | 121/1319 [16:15<2:05:30,  6.29s/it]
Running generate_until requests:   9%|▉         | 122/1319 [16:16<1:29:25,  4.48s/it]
Running generate_until requests:   9%|▉         | 123/1319 [16:23<1:49:06,  5.47s/it]
Running generate_until requests:   9%|▉         | 124/1319 [16:31<2:02:48,  6.17s/it]
Running generate_until requests:   9%|▉         | 125/1319 [16:39<2:12:17,  6.65s/it]
Running generate_until requests:  10%|▉         | 126/1319 [16:42<1:50:37,  5.56s/it]
Running generate_until requests:  10%|▉         | 127/1319 [16:50<2:03:44,  6.23s/it]
Running generate_until requests:  10%|▉         | 128/1319 [16:50<1:27:39,  4.42s/it]
Running generate_until requests:  10%|▉         | 129/1319 [16:50<1:02:35,  3.16s/it]
Running generate_until requests:  10%|▉         | 130/1319 [16:58<1:29:53,  4.54s/it]
Running generate_until requests:  10%|▉         | 131/1319 [17:06<1:49:03,  5.51s/it]
Running generate_until requests:  10%|█         | 132/1319 [17:13<2:02:26,  6.19s/it]
Running generate_until requests:  10%|█         | 133/1319 [17:14<1:27:05,  4.41s/it]
Running generate_until requests:  10%|█         | 134/1319 [17:14<1:01:49,  3.13s/it]
Running generate_until requests:  10%|█         | 135/1319 [17:14<44:41,  2.26s/it]
Running generate_until requests:  10%|█         | 136/1319 [17:14<32:31,  1.65s/it]
Running generate_until requests:  10%|█         | 137/1319 [17:22<1:08:45,  3.49s/it]
Running generate_until requests:  10%|█         | 138/1319 [17:30<1:34:01,  4.78s/it]
Running generate_until requests:  11%|█         | 139/1319 [17:38<1:51:43,  5.68s/it]
Running generate_until requests:  11%|█         | 140/1319 [17:45<2:04:04,  6.31s/it]
Running generate_until requests:  11%|█         | 141/1319 [17:53<2:12:41,  6.76s/it]
Running generate_until requests:  11%|█         | 142/1319 [18:01<2:18:41,  7.07s/it]
Running generate_until requests:  11%|█         | 143/1319 [18:09<2:22:56,  7.29s/it]
Running generate_until requests:  11%|█         | 144/1319 [18:17<2:25:45,  7.44s/it]
Running generate_until requests:  11%|█         | 145/1319 [18:24<2:27:46,  7.55s/it]
Running generate_until requests:  11%|█         | 146/1319 [18:32<2:29:03,  7.62s/it]
Running generate_until requests:  11%|█         | 147/1319 [18:32<1:45:20,  5.39s/it]
Running generate_until requests:  11%|█         | 148/1319 [18:40<1:59:23,  6.12s/it]
Running generate_until requests:  11%|█▏        | 149/1319 [18:48<2:09:07,  6.62s/it]
Running generate_until requests:  11%|█▏        | 150/1319 [18:48<1:31:23,  4.69s/it]
Running generate_until requests:  11%|█▏        | 151/1319 [18:56<1:49:29,  5.62s/it]
Running generate_until requests:  12%|█▏        | 152/1319 [18:56<1:18:01,  4.01s/it]
Running generate_until requests:  12%|█▏        | 153/1319 [19:04<1:40:06,  5.15s/it]
Running generate_until requests:  12%|█▏        | 154/1319 [19:12<1:55:32,  5.95s/it]
Running generate_until requests:  12%|█▏        | 155/1319 [19:20<2:06:09,  6.50s/it]
Running generate_until requests:  12%|█▏        | 156/1319 [19:22<1:41:06,  5.22s/it]
Running generate_until requests:  12%|█▏        | 157/1319 [19:22<1:11:47,  3.71s/it]
Running generate_until requests:  12%|█▏        | 158/1319 [19:22<51:28,  2.66s/it]
Running generate_until requests:  12%|█▏        | 159/1319 [19:30<1:21:17,  4.20s/it]
Running generate_until requests:  12%|█▏        | 160/1319 [19:30<58:06,  3.01s/it]
Running generate_until requests:  12%|█▏        | 161/1319 [19:30<41:42,  2.16s/it]
Running generate_until requests:  12%|█▏        | 162/1319 [19:38<1:14:16,  3.85s/it]
Running generate_until requests:  12%|█▏        | 163/1319 [19:46<1:37:04,  5.04s/it]
Running generate_until requests:  12%|█▏        | 164/1319 [19:54<1:52:55,  5.87s/it]
Running generate_until requests:  13%|█▎        | 165/1319 [20:02<2:04:03,  6.45s/it]
Running generate_until requests:  13%|█▎        | 166/1319 [20:02<1:27:49,  4.57s/it]
Running generate_until requests:  13%|█▎        | 167/1319 [20:10<1:46:24,  5.54s/it]
Running generate_until requests:  13%|█▎        | 168/1319 [20:10<1:15:29,  3.94s/it]
Running generate_until requests:  13%|█▎        | 169/1319 [20:10<53:51,  2.81s/it]
Running generate_until requests:  13%|█▎        | 170/1319 [20:18<1:22:17,  4.30s/it]
Running generate_until requests:  13%|█▎        | 171/1319 [20:18<58:47,  3.07s/it]
Running generate_until requests:  13%|█▎        | 172/1319 [20:26<1:25:37,  4.48s/it]
Running generate_until requests:  13%|█▎        | 173/1319 [20:34<1:44:23,  5.47s/it]
Running generate_until requests:  13%|█▎        | 174/1319 [20:34<1:14:14,  3.89s/it]
Running generate_until requests:  13%|█▎        | 175/1319 [20:42<1:36:21,  5.05s/it]
Running generate_until requests:  13%|█▎        | 176/1319 [20:49<1:51:46,  5.87s/it]
Running generate_until requests:  13%|█▎        | 177/1319 [20:50<1:19:23,  4.17s/it]
Running generate_until requests:  13%|█▎        | 178/1319 [20:57<1:39:59,  5.26s/it]
Running generate_until requests:  14%|█▎        | 179/1319 [21:05<1:54:24,  6.02s/it]
Running generate_until requests:  14%|█▎        | 180/1319 [21:13<2:04:19,  6.55s/it]
Running generate_until requests:  14%|█▎        | 181/1319 [21:21<2:11:11,  6.92s/it]
Running generate_until requests:  14%|█▍        | 182/1319 [21:28<2:15:53,  7.17s/it]
Running generate_until requests:  14%|█▍        | 183/1319 [21:31<1:50:21,  5.83s/it]
Running generate_until requests:  14%|█▍        | 184/1319 [21:31<1:18:24,  4.14s/it]
Running generate_until requests:  14%|█▍        | 185/1319 [21:32<56:24,  2.98s/it]
Running generate_until requests:  14%|█▍        | 186/1319 [21:39<1:23:29,  4.42s/it]
Running generate_until requests:  14%|█▍        | 187/1319 [21:47<1:42:18,  5.42s/it]
Running generate_until requests:  14%|█▍        | 188/1319 [21:55<1:55:25,  6.12s/it]
Running generate_until requests:  14%|█▍        | 189/1319 [22:03<2:04:31,  6.61s/it]
Running generate_until requests:  14%|█▍        | 190/1319 [22:10<2:10:55,  6.96s/it]
Running generate_until requests:  14%|█▍        | 191/1319 [22:18<2:15:20,  7.20s/it]
Running generate_until requests:  15%|█▍        | 192/1319 [22:26<2:18:39,  7.38s/it]
Running generate_until requests:  15%|█▍        | 193/1319 [22:26<1:38:12,  5.23s/it]
Running generate_until requests:  15%|█▍        | 194/1319 [22:26<1:09:53,  3.73s/it]
Running generate_until requests:  15%|█▍        | 195/1319 [22:28<1:00:21,  3.22s/it]
Running generate_until requests:  15%|█▍        | 196/1319 [22:36<1:25:56,  4.59s/it]
Running generate_until requests:  15%|█▍        | 197/1319 [22:44<1:43:46,  5.55s/it]
Running generate_until requests:  15%|█▌        | 198/1319 [22:44<1:13:57,  3.96s/it]
Running generate_until requests:  15%|█▌        | 199/1319 [22:45<53:05,  2.84s/it]
Running generate_until requests:  15%|█▌        | 200/1319 [22:52<1:20:48,  4.33s/it]
Running generate_until requests:  15%|█▌        | 201/1319 [22:53<57:53,  3.11s/it]
Running generate_until requests:  15%|█▌        | 202/1319 [23:00<1:23:52,  4.51s/it]
Running generate_until requests:  15%|█▌        | 203/1319 [23:02<1:08:08,  3.66s/it]
Running generate_until requests:  15%|█▌        | 204/1319 [23:02<48:30,  2.61s/it]
Running generate_until requests:  16%|█▌        | 205/1319 [23:10<1:17:17,  4.16s/it]
Running generate_until requests:  16%|█▌        | 206/1319 [23:10<55:04,  2.97s/it]
Running generate_until requests:  16%|█▌        | 207/1319 [23:18<1:21:45,  4.41s/it]
Running generate_until requests:  16%|█▌        | 208/1319 [23:18<58:22,  3.15s/it]
Running generate_until requests:  16%|█▌        | 209/1319 [23:18<41:51,  2.26s/it]
Running generate_until requests:  16%|█▌        | 210/1319 [23:26<1:12:25,  3.92s/it]
Running generate_until requests:  16%|█▌        | 211/1319 [23:26<51:40,  2.80s/it]
Running generate_until requests:  16%|█▌        | 212/1319 [23:34<1:19:09,  4.29s/it]
Running generate_until requests:  16%|█▌        | 213/1319 [23:42<1:38:20,  5.33s/it]
Running generate_until requests:  16%|█▌        | 214/1319 [23:50<1:51:39,  6.06s/it]
Running generate_until requests:  16%|█▋        | 215/1319 [23:57<2:00:57,  6.57s/it]
Running generate_until requests:  16%|█▋        | 216/1319 [24:05<2:07:22,  6.93s/it]
Running generate_until requests:  16%|█▋        | 217/1319 [24:05<1:30:05,  4.91s/it]
Running generate_until requests:  17%|█▋        | 218/1319 [24:13<1:45:44,  5.76s/it]
Running generate_until requests:  17%|█▋        | 219/1319 [24:21<1:56:45,  6.37s/it]
Running generate_until requests:  17%|█▋        | 220/1319 [24:29<2:04:29,  6.80s/it]
Running generate_until requests:  17%|█▋        | 221/1319 [24:31<1:39:14,  5.42s/it]
Running generate_until requests:  17%|█▋        | 222/1319 [24:39<1:52:04,  6.13s/it]
Running generate_until requests:  17%|█▋        | 223/1319 [24:46<2:00:55,  6.62s/it]
Running generate_until requests:  17%|█▋        | 224/1319 [24:54<2:07:05,  6.96s/it]
Running generate_until requests:  17%|█▋        | 225/1319 [25:02<2:11:26,  7.21s/it]
Running generate_until requests:  17%|█▋        | 226/1319 [25:10<2:14:30,  7.38s/it]
Running generate_until requests:  17%|█▋        | 227/1319 [25:10<1:35:05,  5.22s/it]
Running generate_until requests:  17%|█▋        | 228/1319 [25:18<1:49:03,  6.00s/it]
Running generate_until requests:  17%|█▋        | 229/1319 [25:26<1:58:44,  6.54s/it]
Running generate_until requests:  17%|█▋        | 230/1319 [25:33<2:05:20,  6.91s/it]
Running generate_until requests:  18%|█▊        | 231/1319 [25:36<1:39:51,  5.51s/it]
Running generate_until requests:  18%|█▊        | 232/1319 [25:43<1:52:14,  6.20s/it]
Running generate_until requests:  18%|█▊        | 233/1319 [25:51<2:00:47,  6.67s/it]
Running generate_until requests:  18%|█▊        | 234/1319 [25:51<1:25:29,  4.73s/it]
Running generate_until requests:  18%|█▊        | 235/1319 [25:59<1:42:16,  5.66s/it]
Running generate_until requests:  18%|█▊        | 236/1319 [25:59<1:12:51,  4.04s/it]
Running generate_until requests:  18%|█▊        | 237/1319 [26:00<51:58,  2.88s/it]
Running generate_until requests:  18%|█▊        | 238/1319 [26:07<1:18:17,  4.35s/it]
Running generate_until requests:  18%|█▊        | 239/1319 [26:08<55:54,  3.11s/it]
Running generate_until requests:  18%|█▊        | 240/1319 [26:15<1:20:55,  4.50s/it]
Running generate_until requests:  18%|█▊        | 241/1319 [26:16<57:45,  3.21s/it]
Running generate_until requests:  18%|█▊        | 242/1319 [26:23<1:22:03,  4.57s/it]
Running generate_until requests:  18%|█▊        | 243/1319 [26:24<58:42,  3.27s/it]
Running generate_until requests:  18%|█▊        | 244/1319 [26:24<42:02,  2.35s/it]
Running generate_until requests:  19%|█▊        | 245/1319 [26:32<1:11:14,  3.98s/it]
Running generate_until requests:  19%|█▊        | 246/1319 [26:39<1:31:27,  5.11s/it]
Running generate_until requests:  19%|█▊        | 247/1319 [26:47<1:45:33,  5.91s/it]
Running generate_until requests:  19%|█▉        | 248/1319 [26:49<1:24:50,  4.75s/it]
Running generate_until requests:  19%|█▉        | 249/1319 [26:57<1:40:52,  5.66s/it]
Running generate_until requests:  19%|█▉        | 250/1319 [27:05<1:52:01,  6.29s/it]
Running generate_until requests:  19%|█▉        | 251/1319 [27:12<1:59:53,  6.74s/it]
Running generate_until requests:  19%|█▉        | 252/1319 [27:13<1:25:18,  4.80s/it]
Running generate_until requests:  19%|█▉        | 253/1319 [27:20<1:41:07,  5.69s/it]
Running generate_until requests:  19%|█▉        | 254/1319 [27:28<1:52:02,  6.31s/it]
Running generate_until requests:  19%|█▉        | 255/1319 [27:36<1:59:41,  6.75s/it]
Running generate_until requests:  19%|█▉        | 256/1319 [27:44<2:04:51,  7.05s/it]
Running generate_until requests:  19%|█▉        | 257/1319 [27:45<1:33:34,  5.29s/it]
Running generate_until requests:  20%|█▉        | 258/1319 [27:45<1:06:34,  3.76s/it]
Running generate_until requests:  20%|█▉        | 259/1319 [27:53<1:27:40,  4.96s/it]
Running generate_until requests:  20%|█▉        | 260/1319 [28:01<1:42:18,  5.80s/it]
Running generate_until requests:  20%|█▉        | 261/1319 [28:01<1:12:32,  4.11s/it]
Running generate_until requests:  20%|█▉        | 262/1319 [28:09<1:31:49,  5.21s/it]
Running generate_until requests:  20%|█▉        | 263/1319 [28:09<1:05:21,  3.71s/it]
Running generate_until requests:  20%|██        | 264/1319 [28:17<1:26:46,  4.94s/it]
Running generate_until requests:  20%|██        | 265/1319 [28:19<1:13:40,  4.19s/it]
Running generate_until requests:  20%|██        | 266/1319 [28:27<1:32:28,  5.27s/it]
Running generate_until requests:  20%|██        | 267/1319 [28:27<1:06:17,  3.78s/it]
Running generate_until requests:  20%|██        | 268/1319 [28:35<1:27:06,  4.97s/it]
Running generate_until requests:  20%|██        | 269/1319 [28:35<1:02:02,  3.55s/it]
Running generate_until requests:  20%|██        | 270/1319 [28:43<1:23:59,  4.80s/it]
Running generate_until requests:  21%|██        | 271/1319 [28:51<1:39:23,  5.69s/it]
Running generate_until requests:  21%|██        | 272/1319 [28:58<1:50:05,  6.31s/it]
Running generate_until requests:  21%|██        | 273/1319 [29:06<1:57:30,  6.74s/it]
Running generate_until requests:  21%|██        | 274/1319 [29:14<2:02:46,  7.05s/it]
Running generate_until requests:  21%|██        | 275/1319 [29:22<2:06:25,  7.27s/it]
Running generate_until requests:  21%|██        | 276/1319 [29:29<2:09:00,  7.42s/it]
Running generate_until requests:  21%|██        | 277/1319 [29:37<2:10:44,  7.53s/it]
Running generate_until requests:  21%|██        | 278/1319 [29:45<2:12:07,  7.62s/it]
Running generate_until requests:  21%|██        | 279/1319 [29:53<2:13:01,  7.67s/it]
Running generate_until requests:  21%|██        | 280/1319 [30:01<2:13:23,  7.70s/it]
Running generate_until requests:  21%|██▏       | 281/1319 [30:08<2:13:35,  7.72s/it]
Running generate_until requests:  21%|██▏       | 282/1319 [30:16<2:13:36,  7.73s/it]
Running generate_until requests:  21%|██▏       | 283/1319 [30:24<2:13:40,  7.74s/it]
Running generate_until requests:  22%|██▏       | 284/1319 [30:32<2:13:37,  7.75s/it]
Running generate_until requests:  22%|██▏       | 285/1319 [30:39<2:13:35,  7.75s/it]
Running generate_until requests:  22%|██▏       | 286/1319 [30:40<1:34:23,  5.48s/it]
Running generate_until requests:  22%|██▏       | 287/1319 [30:47<1:46:03,  6.17s/it]
Running generate_until requests:  22%|██▏       | 288/1319 [30:51<1:30:34,  5.27s/it]
Running generate_until requests:  22%|██▏       | 289/1319 [30:58<1:43:23,  6.02s/it]
Running generate_until requests:  22%|██▏       | 290/1319 [30:59<1:13:25,  4.28s/it]
Running generate_until requests:  22%|██▏       | 291/1319 [31:00<56:29,  3.30s/it]
Running generate_until requests:  22%|██▏       | 292/1319 [31:00<40:36,  2.37s/it]
Running generate_until requests:  22%|██▏       | 293/1319 [31:08<1:08:22,  4.00s/it]
Running generate_until requests:  22%|██▏       | 294/1319 [31:15<1:27:43,  5.13s/it]
Running generate_until requests:  22%|██▏       | 295/1319 [31:23<1:41:13,  5.93s/it]
Running generate_until requests:  22%|██▏       | 296/1319 [31:31<1:50:40,  6.49s/it]
Running generate_until requests:  23%|██▎       | 297/1319 [31:39<1:57:12,  6.88s/it]
Running generate_until requests:  23%|██▎       | 298/1319 [31:39<1:23:03,  4.88s/it]
Running generate_until requests:  23%|██▎       | 299/1319 [31:47<1:37:50,  5.76s/it]
Running generate_until requests:  23%|██▎       | 300/1319 [31:55<1:48:04,  6.36s/it]
Running generate_until requests:  23%|██▎       | 301/1319 [32:02<1:55:21,  6.80s/it]
Running generate_until requests:  23%|██▎       | 302/1319 [32:10<2:00:33,  7.11s/it]
Running generate_until requests:  23%|██▎       | 303/1319 [32:10<1:25:14,  5.03s/it]
Running generate_until requests:  23%|██▎       | 304/1319 [32:18<1:39:23,  5.88s/it]
Running generate_until requests:  23%|██▎       | 305/1319 [32:26<1:49:05,  6.46s/it]
Running generate_until requests:  23%|██▎       | 306/1319 [32:29<1:29:55,  5.33s/it]
Running generate_until requests:  23%|██▎       | 307/1319 [32:36<1:42:09,  6.06s/it]
Running generate_until requests:  23%|██▎       | 308/1319 [32:37<1:12:31,  4.30s/it]
Running generate_until requests:  23%|██▎       | 309/1319 [32:44<1:29:55,  5.34s/it]
Running generate_until requests:  24%|██▎       | 310/1319 [32:52<1:42:00,  6.07s/it]
Running generate_until requests:  24%|██▎       | 311/1319 [33:00<1:50:29,  6.58s/it]
Running generate_until requests:  24%|██▎       | 312/1319 [33:08<1:56:22,  6.93s/it]
Running generate_until requests:  24%|██▎       | 313/1319 [33:08<1:22:19,  4.91s/it]
Running generate_until requests:  24%|██▍       | 314/1319 [33:16<1:36:40,  5.77s/it]
Running generate_until requests:  24%|██▍       | 315/1319 [33:16<1:08:50,  4.11s/it]
Running generate_until requests:  24%|██▍       | 316/1319 [33:24<1:27:09,  5.21s/it]
Running generate_until requests:  24%|██▍       | 317/1319 [33:24<1:01:52,  3.71s/it]
Running generate_until requests:  24%|██▍       | 318/1319 [33:32<1:22:07,  4.92s/it]
Running generate_until requests:  24%|██▍       | 319/1319 [33:39<1:36:13,  5.77s/it]
Running generate_until requests:  24%|██▍       | 320/1319 [33:47<1:45:59,  6.37s/it]
Running generate_until requests:  24%|██▍       | 321/1319 [33:55<1:52:47,  6.78s/it]
Running generate_until requests:  24%|██▍       | 322/1319 [34:03<1:57:26,  7.07s/it]
Running generate_until requests:  24%|██▍       | 323/1319 [34:05<1:35:30,  5.75s/it]
Running generate_until requests:  25%|██▍       | 324/1319 [34:13<1:45:20,  6.35s/it]
Running generate_until requests:  25%|██▍       | 325/1319 [34:21<1:52:09,  6.77s/it]
Running generate_until requests:  25%|██▍       | 326/1319 [34:29<1:56:55,  7.06s/it]
Running generate_until requests:  25%|██▍       | 327/1319 [34:29<1:22:49,  5.01s/it]
Running generate_until requests:  25%|██▍       | 328/1319 [34:37<1:36:15,  5.83s/it]
Running generate_until requests:  25%|██▍       | 329/1319 [34:44<1:45:38,  6.40s/it]
Running generate_until requests:  25%|██▌       | 330/1319 [


Bash

grep "generate_until" /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null | tail -3

Check latest generate_until progress


<persisted-output>
Output too large (32.6KB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/bd9wti2t3.txt

Preview (first 2KB):
2026-03-21:10:37:44 INFO     [evaluator:584] Running generate_until requests

Running generate_until requests:   0%|          | 0/1319 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Running generate_until requests:   0%|          | 1/1319 [03:50<84:17:27, 230.23s/it]
Running generate_until requests:   0%|          | 2/1319 [03:58<36:21:58, 99.41s/it]
Running generate_until requests:   0%|          | 3/1319 [04:05<21:02:57, 57.58s/it]
Running generate_until requests:   0%|          | 4/1319 [04:13<13:51:34, 37.94s/it]
Running generate_until requests:   0%|          | 5/1319 [04:21<9:53:22, 27.09s/it]
Running generate_until requests:   0%|          | 6/1319 [04:29<7:29:10, 20.53s/it]
Running generate_until requests:   1%|          | 7/1319 [04:29<5:03:26, 13.88s/it]
Running generate_until requests:   1%|          | 8/1319 [04:37<4:20:42, 11.93s/it]
Running generate_until requests:   1%|          | 9/1319 [04:45<3:52:03, 10.63s/it]
Running generate_until requests:   1%|          | 10/1319 [04:52<3:32:41,  9.75s/it]
Running generate_until requests:   1%|          | 11/1319 [05:00<3:19:20,  9.14s/it]
Running generate_until requests:   1%|          | 12/1319 [05:08<3:10:08,  8.73s/it]
Running generate_until requests:   1%|          | 13/1319 [05:16<3:03:42,  8.44s/it]
Running generate_until requests:   1%|          | 14/1319 [05:23<2:59:09,  8.24s/it]
Running generate_until requests:   1%|          | 15/1319 [05:31<2:55:58,  8.10s/it]
Running generate_until requests:   1%|          | 16/1319 [05:39<2:53:40,  8.00s/it]
Running generate_until requests:   1%|▏         | 17/1319 [05:47<2:52:10,  7.93s/it]
Running generate_until requests:   1%|▏         | 18/1319 [05:55<2:51:07,  7.89s/it]
Running generate_until requests:   1%|▏         | 19/1319 [05:55<2:00:41,  5.57s/it]
Running generate_until requests:   2%|▏         | 20/1319 [06:03<2:15:04,  6.24s/it
...
</persisted-output>

2026-03-21:10:37:44 INFO     [evaluator:584] Running generate_until requests

Running generate_until requests:   0%|          | 0/1319 [00:00<?, ?it/s]The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Running generate_until requests:   0%|          | 1/1319 [03:50<84:17:27, 230.23s/it]
Running generate_until requests:   0%|          | 2/1319 [03:58<36:21:58, 99.41s/it]
Running generate_until requests:   0%|          | 3/1319 [04:05<21:02:57, 57.58s/it]
Running generate_until requests:   0%|          | 4/1319 [04:13<13:51:34, 37.94s/it]
Running generate_until requests:   0%|          | 5/1319 [04:21<9:53:22, 27.09s/it]
Running generate_until requests:   0%|          | 6/1319 [04:29<7:29:10, 20.53s/it]
Running generate_until requests:   1%|          | 7/1319 [04:29<5:03:26, 13.88s/it]
Running generate_until requests:   1%|          | 8/1319 [04:37<4:20:42, 11.93s/it]
Running generate_until requests:   1%|          | 9/1319 [04:45<3:52:03, 10.63s/it]
Running generate_until requests:   1%|          | 10/1319 [04:52<3:32:41,  9.75s/it]
Running generate_until requests:   1%|          | 11/1319 [05:00<3:19:20,  9.14s/it]
Running generate_until requests:   1%|          | 12/1319 [05:08<3:10:08,  8.73s/it]
Running generate_until requests:   1%|          | 13/1319 [05:16<3:03:42,  8.44s/it]
Running generate_until requests:   1%|          | 14/1319 [05:23<2:59:09,  8.24s/it]
Running generate_until requests:   1%|          | 15/1319 [05:31<2:55:58,  8.10s/it]
Running generate_until requests:   1%|          | 16/1319 [05:39<2:53:40,  8.00s/it]
Running generate_until requests:   1%|▏         | 17/1319 [05:47<2:52:10,  7.93s/it]
Running generate_until requests:   1%|▏         | 18/1319 [05:55<2:51:07,  7.89s/it]
Running generate_until requests:   1%|▏         | 19/1319 [05:55<2:00:41,  5.57s/it]
Running generate_until requests:   2%|▏         | 20/1319 [06:03<2:15:04,  6.24s/it]
Running generate_until requests:   2%|▏         | 21/1319 [06:10<2:25:10,  6.71s/it]
Running generate_until requests:   2%|▏         | 22/1319 [06:18<2:32:08,  7.04s/it]
Running generate_until requests:   2%|▏         | 23/1319 [06:26<2:37:00,  7.27s/it]
Running generate_until requests:   2%|▏         | 24/1319 [06:34<2:40:19,  7.43s/it]
Running generate_until requests:   2%|▏         | 25/1319 [06:34<1:53:20,  5.26s/it]
Running generate_until requests:   2%|▏         | 26/1319 [06:42<2:09:42,  6.02s/it]
Running generate_until requests:   2%|▏         | 27/1319 [06:50<2:21:06,  6.55s/it]
Running generate_until requests:   2%|▏         | 28/1319 [06:57<2:29:03,  6.93s/it]
Running generate_until requests:   2%|▏         | 29/1319 [07:05<2:34:33,  7.19s/it]
Running generate_until requests:   2%|▏         | 30/1319 [07:13<2:38:18,  7.37s/it]
Running generate_until requests:   2%|▏         | 31/1319 [07:21<2:40:54,  7.50s/it]
Running generate_until requests:   2%|▏         | 32/1319 [07:29<2:42:52,  7.59s/it]
Running generate_until requests:   3%|▎         | 33/1319 [07:36<2:44:07,  7.66s/it]
Running generate_until requests:   3%|▎         | 34/1319 [07:44<2:45:04,  7.71s/it]
Running generate_until requests:   3%|▎         | 35/1319 [07:52<2:45:36,  7.74s/it]
Running generate_until requests:   3%|▎         | 36/1319 [08:00<2:45:55,  7.76s/it]
Running generate_until requests:   3%|▎         | 37/1319 [08:00<1:57:37,  5.51s/it]
Running generate_until requests:   3%|▎         | 38/1319 [08:00<1:23:16,  3.90s/it]
Running generate_until requests:   3%|▎         | 39/1319 [08:08<1:48:17,  5.08s/it]
Running generate_until requests:   3%|▎         | 40/1319 [08:08<1:17:07,  3.62s/it]
Running generate_until requests:   3%|▎         | 41/1319 [08:16<1:43:49,  4.87s/it]
Running generate_until requests:   3%|▎         | 42/1319 [08:24<2:02:24,  5.75s/it]
Running generate_until requests:   3%|▎         | 43/1319 [08:32<2:15:30,  6.37s/it]
Running generate_until requests:   3%|▎         | 44/1319 [08:32<1:36:08,  4.52s/it]
Running generate_until requests:   3%|▎         | 45/1319 [08:40<1:56:50,  5.50s/it]
Running generate_until requests:   3%|▎         | 46/1319 [08:47<2:11:08,  6.18s/it]
Running generate_until requests:   4%|▎         | 47/1319 [08:55<2:21:13,  6.66s/it]
Running generate_until requests:   4%|▎         | 48/1319 [09:03<2:28:15,  7.00s/it]
Running generate_until requests:   4%|▎         | 49/1319 [09:03<1:45:04,  4.96s/it]
Running generate_until requests:   4%|▍         | 50/1319 [09:11<2:02:51,  5.81s/it]
Running generate_until requests:   4%|▍         | 51/1319 [09:19<2:15:17,  6.40s/it]
Running generate_until requests:   4%|▍         | 52/1319 [09:27<2:23:48,  6.81s/it]
Running generate_until requests:   4%|▍         | 53/1319 [09:34<2:29:48,  7.10s/it]
Running generate_until requests:   4%|▍         | 54/1319 [09:42<2:33:59,  7.30s/it]
Running generate_until requests:   4%|▍         | 55/1319 [09:50<2:37:03,  7.46s/it]
Running generate_until requests:   4%|▍         | 56/1319 [09:50<1:51:01,  5.27s/it]
Running generate_until requests:   4%|▍         | 57/1319 [09:58<2:06:59,  6.04s/it]
Running generate_until requests:   4%|▍         | 58/1319 [10:06<2:18:15,  6.58s/it]
Running generate_until requests:   4%|▍         | 59/1319 [10:14<2:25:57,  6.95s/it]
Running generate_until requests:   5%|▍         | 60/1319 [10:21<2:31:11,  7.21s/it]
Running generate_until requests:   5%|▍         | 61/1319 [10:29<2:34:45,  7.38s/it]
Running generate_until requests:   5%|▍         | 62/1319 [10:37<2:37:14,  7.51s/it]
Running generate_until requests:   5%|▍         | 63/1319 [10:45<2:38:55,  7.59s/it]
Running generate_until requests:   5%|▍         | 64/1319 [10:52<2:39:59,  7.65s/it]
Running generate_until requests:   5%|▍         | 65/1319 [10:53<1:52:52,  5.40s/it]
Running generate_until requests:   5%|▌         | 66/1319 [11:00<2:07:42,  6.11s/it]
Running generate_until requests:   5%|▌         | 67/1319 [11:01<1:30:39,  4.34s/it]
Running generate_until requests:   5%|▌         | 68/1319 [11:08<1:52:06,  5.38s/it]
Running generate_until requests:   5%|▌         | 69/1319 [11:16<2:07:01,  6.10s/it]
Running generate_until requests:   5%|▌         | 70/1319 [11:16<1:30:11,  4.33s/it]
Running generate_until requests:   5%|▌         | 71/1319 [11:24<1:51:43,  5.37s/it]
Running generate_until requests:   5%|▌         | 72/1319 [11:32<2:06:44,  6.10s/it]
Running generate_until requests:   6%|▌         | 73/1319 [11:40<2:17:20,  6.61s/it]
Running generate_until requests:   6%|▌         | 74/1319 [11:48<2:24:42,  6.97s/it]
Running generate_until requests:   6%|▌         | 75/1319 [11:55<2:29:47,  7.22s/it]
Running generate_until requests:   6%|▌         | 76/1319 [11:56<1:46:18,  5.13s/it]
Running generate_until requests:   6%|▌         | 77/1319 [12:04<2:02:48,  5.93s/it]
Running generate_until requests:   6%|▌         | 78/1319 [12:11<2:14:11,  6.49s/it]
Running generate_until requests:   6%|▌         | 79/1319 [12:19<2:22:14,  6.88s/it]
Running generate_until requests:   6%|▌         | 80/1319 [12:27<2:27:47,  7.16s/it]
Running generate_until requests:   6%|▌         | 81/1319 [12:35<2:31:43,  7.35s/it]
Running generate_until requests:   6%|▌         | 82/1319 [12:43<2:34:31,  7.50s/it]
Running generate_until requests:   6%|▋         | 83/1319 [12:43<1:49:25,  5.31s/it]
Running generate_until requests:   6%|▋         | 84/1319 [12:43<1:17:51,  3.78s/it]
Running generate_until requests:   6%|▋         | 85/1319 [12:51<1:42:38,  4.99s/it]
Running generate_until requests:   7%|▋         | 86/1319 [12:59<1:59:52,  5.83s/it]
Running generate_until requests:   7%|▋         | 87/1319 [13:06<2:11:50,  6.42s/it]
Running generate_until requests:   7%|▋         | 88/1319 [13:14<2:20:07,  6.83s/it]
Running generate_until requests:   7%|▋         | 89/1319 [13:22<2:25:51,  7.12s/it]
Running generate_until requests:   7%|▋         | 90/1319 [13:30<2:29:47,  7.31s/it]
Running generate_until requests:   7%|▋         | 91/1319 [13:37<2:32:34,  7.46s/it]
Running generate_until requests:   7%|▋         | 92/1319 [13:45<2:34:24,  7.55s/it]
Running generate_until requests:   7%|▋         | 93/1319 [13:53<2:35:34,  7.61s/it]
Running generate_until requests:   7%|▋         | 94/1319 [13:53<1:50:18,  5.40s/it]
Running generate_until requests:   7%|▋         | 95/1319 [14:01<2:04:51,  6.12s/it]
Running generate_until requests:   7%|▋         | 96/1319 [14:09<2:15:02,  6.62s/it]
Running generate_until requests:   7%|▋         | 97/1319 [14:09<1:35:34,  4.69s/it]
Running generate_until requests:   7%|▋         | 98/1319 [14:09<1:07:58,  3.34s/it]
Running generate_until requests:   8%|▊         | 99/1319 [14:09<48:50,  2.40s/it]
Running generate_until requests:   8%|▊         | 100/1319 [14:17<1:21:39,  4.02s/it]
Running generate_until requests:   8%|▊         | 101/1319 [14:25<1:44:32,  5.15s/it]
Running generate_until requests:   8%|▊         | 102/1319 [14:33<2:00:15,  5.93s/it]
Running generate_until requests:   8%|▊         | 103/1319 [14:41<2:11:23,  6.48s/it]
Running generate_until requests:   8%|▊         | 104/1319 [14:48<2:19:03,  6.87s/it]
Running generate_until requests:   8%|▊         | 105/1319 [14:56<2:24:18,  7.13s/it]
Running generate_until requests:   8%|▊         | 106/1319 [14:56<1:42:02,  5.05s/it]
Running generate_until requests:   8%|▊         | 107/1319 [15:04<1:58:28,  5.86s/it]
Running generate_until requests:   8%|▊         | 108/1319 [15:04<1:24:31,  4.19s/it]
Running generate_until requests:   8%|▊         | 109/1319 [15:05<1:00:35,  3.00s/it]
Running generate_until requests:   8%|▊         | 110/1319 [15:12<1:29:20,  4.43s/it]
Running generate_until requests:   8%|▊         | 111/1319 [15:13<1:03:46,  3.17s/it]
Running generate_until requests:   8%|▊         | 112/1319 [15:20<1:31:27,  4.55s/it]
Running generate_until requests:   9%|▊         | 113/1319 [15:28<1:50:52,  5.52s/it]
Running generate_until requests:   9%|▊         | 114/1319 [15:36<2:04:29,  6.20s/it]
Running generate_until requests:   9%|▊         | 115/1319 [15:36<1:28:22,  4.40s/it]
Running generate_until requests:   9%|▉         | 116/1319 [15:44<1:48:32,  5.41s/it]
Running generate_until requests:   9%|▉         | 117/1319 [15:52<2:02:47,  6.13s/it]
Running generate_until requests:   9%|▉         | 118/1319 [15:59<2:12:30,  6.62s/it]
Running generate_until requests:   9%|▉         | 119/1319 [16:00<1:34:09,  4.71s/it]
Running generate_until requests:   9%|▉         | 120/1319 [16:07<1:52:27,  5.63s/it]
Running generate_until requests:   9%|▉         | 121/1319 [16:15<2:05:30,  6.29s/it]
Running generate_until requests:   9%|▉         | 122/1319 [16:16<1:29:25,  4.48s/it]
Running generate_until requests:   9%|▉         | 123/1319 [16:23<1:49:06,  5.47s/it]
Running generate_until requests:   9%|▉         | 124/1319 [16:31<2:02:48,  6.17s/it]
Running generate_until requests:   9%|▉         | 125/1319 [16:39<2:12:17,  6.65s/it]
Running generate_until requests:  10%|▉         | 126/1319 [16:42<1:50:37,  5.56s/it]
Running generate_until requests:  10%|▉         | 127/1319 [16:50<2:03:44,  6.23s/it]
Running generate_until requests:  10%|▉         | 128/1319 [16:50<1:27:39,  4.42s/it]
Running generate_until requests:  10%|▉         | 129/1319 [16:50<1:02:35,  3.16s/it]
Running generate_until requests:  10%|▉         | 130/1319 [16:58<1:29:53,  4.54s/it]
Running generate_until requests:  10%|▉         | 131/1319 [17:06<1:49:03,  5.51s/it]
Running generate_until requests:  10%|█         | 132/1319 [17:13<2:02:26,  6.19s/it]
Running generate_until requests:  10%|█         | 133/1319 [17:14<1:27:05,  4.41s/it]
Running generate_until requests:  10%|█         | 134/1319 [17:14<1:01:49,  3.13s/it]
Running generate_until requests:  10%|█         | 135/1319 [17:14<44:41,  2.26s/it]
Running generate_until requests:  10%|█         | 136/1319 [17:14<32:31,  1.65s/it]
Running generate_until requests:  10%|█         | 137/1319 [17:22<1:08:45,  3.49s/it]
Running generate_until requests:  10%|█         | 138/1319 [17:30<1:34:01,  4.78s/it]
Running generate_until requests:  11%|█         | 139/1319 [17:38<1:51:43,  5.68s/it]
Running generate_until requests:  11%|█         | 140/1319 [17:45<2:04:04,  6.31s/it]
Running generate_until requests:  11%|█         | 141/1319 [17:53<2:12:41,  6.76s/it]
Running generate_until requests:  11%|█         | 142/1319 [18:01<2:18:41,  7.07s/it]
Running generate_until requests:  11%|█         | 143/1319 [18:09<2:22:56,  7.29s/it]
Running generate_until requests:  11%|█         | 144/1319 [18:17<2:25:45,  7.44s/it]
Running generate_until requests:  11%|█         | 145/1319 [18:24<2:27:46,  7.55s/it]
Running generate_until requests:  11%|█         | 146/1319 [18:32<2:29:03,  7.62s/it]
Running generate_until requests:  11%|█         | 147/1319 [18:32<1:45:20,  5.39s/it]
Running generate_until requests:  11%|█         | 148/1319 [18:40<1:59:23,  6.12s/it]
Running generate_until requests:  11%|█▏        | 149/1319 [18:48<2:09:07,  6.62s/it]
Running generate_until requests:  11%|█▏        | 150/1319 [18:48<1:31:23,  4.69s/it]
Running generate_until requests:  11%|█▏        | 151/1319 [18:56<1:49:29,  5.62s/it]
Running generate_until requests:  12%|█▏        | 152/1319 [18:56<1:18:01,  4.01s/it]
Running generate_until requests:  12%|█▏        | 153/1319 [19:04<1:40:06,  5.15s/it]
Running generate_until requests:  12%|█▏        | 154/1319 [19:12<1:55:32,  5.95s/it]
Running generate_until requests:  12%|█▏        | 155/1319 [19:20<2:06:09,  6.50s/it]
Running generate_until requests:  12%|█▏        | 156/1319 [19:22<1:41:06,  5.22s/it]
Running generate_until requests:  12%|█▏        | 157/1319 [19:22<1:11:47,  3.71s/it]
Running generate_until requests:  12%|█▏        | 158/1319 [19:22<51:28,  2.66s/it]
Running generate_until requests:  12%|█▏        | 159/1319 [19:30<1:21:17,  4.20s/it]
Running generate_until requests:  12%|█▏        | 160/1319 [19:30<58:06,  3.01s/it]
Running generate_until requests:  12%|█▏        | 161/1319 [19:30<41:42,  2.16s/it]
Running generate_until requests:  12%|█▏        | 162/1319 [19:38<1:14:16,  3.85s/it]
Running generate_until requests:  12%|█▏        | 163/1319 [19:46<1:37:04,  5.04s/it]
Running generate_until requests:  12%|█▏        | 164/1319 [19:54<1:52:55,  5.87s/it]
Running generate_until requests:  13%|█▎        | 165/1319 [20:02<2:04:03,  6.45s/it]
Running generate_until requests:  13%|█▎        | 166/1319 [20:02<1:27:49,  4.57s/it]
Running generate_until requests:  13%|█▎        | 167/1319 [20:10<1:46:24,  5.54s/it]
Running generate_until requests:  13%|█▎        | 168/1319 [20:10<1:15:29,  3.94s/it]
Running generate_until requests:  13%|█▎        | 169/1319 [20:10<53:51,  2.81s/it]
Running generate_until requests:  13%|█▎        | 170/1319 [20:18<1:22:17,  4.30s/it]
Running generate_until requests:  13%|█▎        | 171/1319 [20:18<58:47,  3.07s/it]
Running generate_until requests:  13%|█▎        | 172/1319 [20:26<1:25:37,  4.48s/it]
Running generate_until requests:  13%|█▎        | 173/1319 [20:34<1:44:23,  5.47s/it]
Running generate_until requests:  13%|█▎        | 174/1319 [20:34<1:14:14,  3.89s/it]
Running generate_until requests:  13%|█▎        | 175/1319 [20:42<1:36:21,  5.05s/it]
Running generate_until requests:  13%|█▎        | 176/1319 [20:49<1:51:46,  5.87s/it]
Running generate_until requests:  13%|█▎        | 177/1319 [20:50<1:19:23,  4.17s/it]
Running generate_until requests:  13%|█▎        | 178/1319 [20:57<1:39:59,  5.26s/it]
Running generate_until requests:  14%|█▎        | 179/1319 [21:05<1:54:24,  6.02s/it]
Running generate_until requests:  14%|█▎        | 180/1319 [21:13<2:04:19,  6.55s/it]
Running generate_until requests:  14%|█▎        | 181/1319 [21:21<2:11:11,  6.92s/it]
Running generate_until requests:  14%|█▍        | 182/1319 [21:28<2:15:53,  7.17s/it]
Running generate_until requests:  14%|█▍        | 183/1319 [21:31<1:50:21,  5.83s/it]
Running generate_until requests:  14%|█▍        | 184/1319 [21:31<1:18:24,  4.14s/it]
Running generate_until requests:  14%|█▍        | 185/1319 [21:32<56:24,  2.98s/it]
Running generate_until requests:  14%|█▍        | 186/1319 [21:39<1:23:29,  4.42s/it]
Running generate_until requests:  14%|█▍        | 187/1319 [21:47<1:42:18,  5.42s/it]
Running generate_until requests:  14%|█▍        | 188/1319 [21:55<1:55:25,  6.12s/it]
Running generate_until requests:  14%|█▍        | 189/1319 [22:03<2:04:31,  6.61s/it]
Running generate_until requests:  14%|█▍        | 190/1319 [22:10<2:10:55,  6.96s/it]
Running generate_until requests:  14%|█▍        | 191/1319 [22:18<2:15:20,  7.20s/it]
Running generate_until requests:  15%|█▍        | 192/1319 [22:26<2:18:39,  7.38s/it]
Running generate_until requests:  15%|█▍        | 193/1319 [22:26<1:38:12,  5.23s/it]
Running generate_until requests:  15%|█▍        | 194/1319 [22:26<1:09:53,  3.73s/it]
Running generate_until requests:  15%|█▍        | 195/1319 [22:28<1:00:21,  3.22s/it]
Running generate_until requests:  15%|█▍        | 196/1319 [22:36<1:25:56,  4.59s/it]
Running generate_until requests:  15%|█▍        | 197/1319 [22:44<1:43:46,  5.55s/it]
Running generate_until requests:  15%|█▌        | 198/1319 [22:44<1:13:57,  3.96s/it]
Running generate_until requests:  15%|█▌        | 199/1319 [22:45<53:05,  2.84s/it]
Running generate_until requests:  15%|█▌        | 200/1319 [22:52<1:20:48,  4.33s/it]
Running generate_until requests:  15%|█▌        | 201/1319 [22:53<57:53,  3.11s/it]
Running generate_until requests:  15%|█▌        | 202/1319 [23:00<1:23:52,  4.51s/it]
Running generate_until requests:  15%|█▌        | 203/1319 [23:02<1:08:08,  3.66s/it]
Running generate_until requests:  15%|█▌        | 204/1319 [23:02<48:30,  2.61s/it]
Running generate_until requests:  16%|█▌        | 205/1319 [23:10<1:17:17,  4.16s/it]
Running generate_until requests:  16%|█▌        | 206/1319 [23:10<55:04,  2.97s/it]
Running generate_until requests:  16%|█▌        | 207/1319 [23:18<1:21:45,  4.41s/it]
Running generate_until requests:  16%|█▌        | 208/1319 [23:18<58:22,  3.15s/it]
Running generate_until requests:  16%|█▌        | 209/1319 [23:18<41:51,  2.26s/it]
Running generate_until requests:  16%|█▌        | 210/1319 [23:26<1:12:25,  3.92s/it]
Running generate_until requests:  16%|█▌        | 211/1319 [23:26<51:40,  2.80s/it]
Running generate_until requests:  16%|█▌        | 212/1319 [23:34<1:19:09,  4.29s/it]
Running generate_until requests:  16%|█▌        | 213/1319 [23:42<1:38:20,  5.33s/it]
Running generate_until requests:  16%|█▌        | 214/1319 [23:50<1:51:39,  6.06s/it]
Running generate_until requests:  16%|█▋        | 215/1319 [23:57<2:00:57,  6.57s/it]
Running generate_until requests:  16%|█▋        | 216/1319 [24:05<2:07:22,  6.93s/it]
Running generate_until requests:  16%|█▋        | 217/1319 [24:05<1:30:05,  4.91s/it]
Running generate_until requests:  17%|█▋        | 218/1319 [24:13<1:45:44,  5.76s/it]
Running generate_until requests:  17%|█▋        | 219/1319 [24:21<1:56:45,  6.37s/it]
Running generate_until requests:  17%|█▋        | 220/1319 [24:29<2:04:29,  6.80s/it]
Running generate_until requests:  17%|█▋        | 221/1319 [24:31<1:39:14,  5.42s/it]
Running generate_until requests:  17%|█▋        | 222/1319 [24:39<1:52:04,  6.13s/it]
Running generate_until requests:  17%|█▋        | 223/1319 [24:46<2:00:55,  6.62s/it]
Running generate_until requests:  17%|█▋        | 224/1319 [24:54<2:07:05,  6.96s/it]
Running generate_until requests:  17%|█▋        | 225/1319 [25:02<2:11:26,  7.21s/it]
Running generate_until requests:  17%|█▋        | 226/1319 [25:10<2:14:30,  7.38s/it]
Running generate_until requests:  17%|█▋        | 227/1319 [25:10<1:35:05,  5.22s/it]
Running generate_until requests:  17%|█▋        | 228/1319 [25:18<1:49:03,  6.00s/it]
Running generate_until requests:  17%|█▋        | 229/1319 [25:26<1:58:44,  6.54s/it]
Running generate_until requests:  17%|█▋        | 230/1319 [25:33<2:05:20,  6.91s/it]
Running generate_until requests:  18%|█▊        | 231/1319 [25:36<1:39:51,  5.51s/it]
Running generate_until requests:  18%|█▊        | 232/1319 [25:43<1:52:14,  6.20s/it]
Running generate_until requests:  18%|█▊        | 233/1319 [25:51<2:00:47,  6.67s/it]
Running generate_until requests:  18%|█▊        | 234/1319 [25:51<1:25:29,  4.73s/it]
Running generate_until requests:  18%|█▊        | 235/1319 [25:59<1:42:16,  5.66s/it]
Running generate_until requests:  18%|█▊        | 236/1319 [25:59<1:12:51,  4.04s/it]
Running generate_until requests:  18%|█▊        | 237/1319 [26:00<51:58,  2.88s/it]
Running generate_until requests:  18%|█▊        | 238/1319 [26:07<1:18:17,  4.35s/it]
Running generate_until requests:  18%|█▊        | 239/1319 [26:08<55:54,  3.11s/it]
Running generate_until requests:  18%|█▊        | 240/1319 [26:15<1:20:55,  4.50s/it]
Running generate_until requests:  18%|█▊        | 241/1319 [26:16<57:45,  3.21s/it]
Running generate_until requests:  18%|█▊        | 242/1319 [26:23<1:22:03,  4.57s/it]
Running generate_until requests:  18%|█▊        | 243/1319 [26:24<58:42,  3.27s/it]
Running generate_until requests:  18%|█▊        | 244/1319 [26:24<42:02,  2.35s/it]
Running generate_until requests:  19%|█▊        | 245/1319 [26:32<1:11:14,  3.98s/it]
Running generate_until requests:  19%|█▊        | 246/1319 [26:39<1:31:27,  5.11s/it]
Running generate_until requests:  19%|█▊        | 247/1319 [26:47<1:45:33,  5.91s/it]
Running generate_until requests:  19%|█▉        | 248/1319 [26:49<1:24:50,  4.75s/it]
Running generate_until requests:  19%|█▉        | 249/1319 [26:57<1:40:52,  5.66s/it]
Running generate_until requests:  19%|█▉        | 250/1319 [27:05<1:52:01,  6.29s/it]
Running generate_until requests:  19%|█▉        | 251/1319 [27:12<1:59:53,  6.74s/it]
Running generate_until requests:  19%|█▉        | 252/1319 [27:13<1:25:18,  4.80s/it]
Running generate_until requests:  19%|█▉        | 253/1319 [27:20<1:41:07,  5.69s/it]
Running generate_until requests:  19%|█▉        | 254/1319 [27:28<1:52:02,  6.31s/it]
Running generate_until requests:  19%|█▉        | 255/1319 [27:36<1:59:41,  6.75s/it]
Running generate_until requests:  19%|█▉        | 256/1319 [27:44<2:04:51,  7.05s/it]
Running generate_until requests:  19%|█▉        | 257/1319 [27:45<1:33:34,  5.29s/it]
Running generate_until requests:  20%|█▉        | 258/1319 [27:45<1:06:34,  3.76s/it]
Running generate_until requests:  20%|█▉        | 259/1319 [27:53<1:27:40,  4.96s/it]
Running generate_until requests:  20%|█▉        | 260/1319 [28:01<1:42:18,  5.80s/it]
Running generate_until requests:  20%|█▉        | 261/1319 [28:01<1:12:32,  4.11s/it]
Running generate_until requests:  20%|█▉        | 262/1319 [28:09<1:31:49,  5.21s/it]
Running generate_until requests:  20%|█▉        | 263/1319 [28:09<1:05:21,  3.71s/it]
Running generate_until requests:  20%|██        | 264/1319 [28:17<1:26:46,  4.94s/it]
Running generate_until requests:  20%|██        | 265/1319 [28:19<1:13:40,  4.19s/it]
Running generate_until requests:  20%|██        | 266/1319 [28:27<1:32:28,  5.27s/it]
Running generate_until requests:  20%|██        | 267/1319 [28:27<1:06:17,  3.78s/it]
Running generate_until requests:  20%|██        | 268/1319 [28:35<1:27:06,  4.97s/it]
Running generate_until requests:  20%|██        | 269/1319 [28:35<1:02:02,  3.55s/it]
Running generate_until requests:  20%|██        | 270/1319 [28:43<1:23:59,  4.80s/it]
Running generate_until requests:  21%|██        | 271/1319 [28:51<1:39:23,  5.69s/it]
Running generate_until requests:  21%|██        | 272/1319 [28:58<1:50:05,  6.31s/it]
Running generate_until requests:  21%|██        | 273/1319 [29:06<1:57:30,  6.74s/it]
Running generate_until requests:  21%|██        | 274/1319 [29:14<2:02:46,  7.05s/it]
Running generate_until requests:  21%|██        | 275/1319 [29:22<2:06:25,  7.27s/it]
Running generate_until requests:  21%|██        | 276/1319 [29:29<2:09:00,  7.42s/it]
Running generate_until requests:  21%|██        | 277/1319 [29:37<2:10:44,  7.53s/it]
Running generate_until requests:  21%|██        | 278/1319 [29:45<2:12:07,  7.62s/it]
Running generate_until requests:  21%|██        | 279/1319 [29:53<2:13:01,  7.67s/it]
Running generate_until requests:  21%|██        | 280/1319 [30:01<2:13:23,  7.70s/it]
Running generate_until requests:  21%|██▏       | 281/1319 [30:08<2:13:35,  7.72s/it]
Running generate_until requests:  21%|██▏       | 282/1319 [30:16<2:13:36,  7.73s/it]
Running generate_until requests:  21%|██▏       | 283/1319 [30:24<2:13:40,  7.74s/it]
Running generate_until requests:  22%|██▏       | 284/1319 [30:32<2:13:37,  7.75s/it]
Running generate_until requests:  22%|██▏       | 285/1319 [30:39<2:13:35,  7.75s/it]
Running generate_until requests:  22%|██▏       | 286/1319 [30:40<1:34:23,  5.48s/it]
Running generate_until requests:  22%|██▏       | 287/1319 [30:47<1:46:03,  6.17s/it]
Running generate_until requests:  22%|██▏       | 288/1319 [30:51<1:30:34,  5.27s/it]
Running generate_until requests:  22%|██▏       | 289/1319 [30:58<1:43:23,  6.02s/it]
Running generate_until requests:  22%|██▏       | 290/1319 [30:59<1:13:25,  4.28s/it]
Running generate_until requests:  22%|██▏       | 291/1319 [31:00<56:29,  3.30s/it]
Running generate_until requests:  22%|██▏       | 292/1319 [31:00<40:36,  2.37s/it]
Running generate_until requests:  22%|██▏       | 293/1319 [31:08<1:08:22,  4.00s/it]
Running generate_until requests:  22%|██▏       | 294/1319 [31:15<1:27:43,  5.13s/it]
Running generate_until requests:  22%|██▏       | 295/1319 [31:23<1:41:13,  5.93s/it]
Running generate_until requests:  22%|██▏       | 296/1319 [31:31<1:50:40,  6.49s/it]
Running generate_until requests:  23%|██▎       | 297/1319 [31:39<1:57:12,  6.88s/it]
Running generate_until requests:  23%|██▎       | 298/1319 [31:39<1:23:03,  4.88s/it]
Running generate_until requests:  23%|██▎       | 299/1319 [31:47<1:37:50,  5.76s/it]
Running generate_until requests:  23%|██▎       | 300/1319 [31:55<1:48:04,  6.36s/it]
Running generate_until requests:  23%|██▎       | 301/1319 [32:02<1:55:21,  6.80s/it]
Running generate_until requests:  23%|██▎       | 302/1319 [32:10<2:00:33,  7.11s/it]
Running generate_until requests:  23%|██▎       | 303/1319 [32:10<1:25:14,  5.03s/it]
Running generate_until requests:  23%|██▎       | 304/1319 [32:18<1:39:23,  5.88s/it]
Running generate_until requests:  23%|██▎       | 305/1319 [32:26<1:49:05,  6.46s/it]
Running generate_until requests:  23%|██▎       | 306/1319 [32:29<1:29:55,  5.33s/it]
Running generate_until requests:  23%|██▎       | 307/1319 [32:36<1:42:09,  6.06s/it]
Running generate_until requests:  23%|██▎       | 308/1319 [32:37<1:12:31,  4.30s/it]
Running generate_until requests:  23%|██▎       | 309/1319 [32:44<1:29:55,  5.34s/it]
Running generate_until requests:  24%|██▎       | 310/1319 [32:52<1:42:00,  6.07s/it]
Running generate_until requests:  24%|██▎       | 311/1319 [33:00<1:50:29,  6.58s/it]
Running generate_until requests:  24%|██▎       | 312/1319 [33:08<1:56:22,  6.93s/it]
Running generate_until requests:  24%|██▎       | 313/1319 [33:08<1:22:19,  4.91s/it]
Running generate_until requests:  24%|██▍       | 314/1319 [33:16<1:36:40,  5.77s/it]
Running generate_until requests:  24%|██▍       | 315/1319 [33:16<1:08:50,  4.11s/it]
Running generate_until requests:  24%|██▍       | 316/1319 [33:24<1:27:09,  5.21s/it]
Running generate_until requests:  24%|██▍       | 317/1319 [33:24<1:01:52,  3.71s/it]
Running generate_until requests:  24%|██▍       | 318/1319 [33:32<1:22:07,  4.92s/it]
Running generate_until requests:  24%|██▍       | 319/1319 [33:39<1:36:13,  5.77s/it]
Running generate_until requests:  24%|██▍       | 320/1319 [33:47<1:45:59,  6.37s/it]
Running generate_until requests:  24%|██▍       | 321/1319 [33:55<1:52:47,  6.78s/it]
Running generate_until requests:  24%|██▍       | 322/1319 [34:03<1:57:26,  7.07s/it]
Running generate_until requests:  24%|██▍       | 323/1319 [34:05<1:35:30,  5.75s/it]
Running generate_until requests:  25%|██▍       | 324/1319 [34:13<1:45:20,  6.35s/it]
Running generate_until requests:  25%|██▍       | 325/1319 [34:21<1:52:09,  6.77s/it]
Running generate_until requests:  25%|██▍       | 326/1319 [34:29<1:56:55,  7.06s/it]
Running generate_until requests:  25%|██▍       | 327/1319 [34:29<1:22:49,  5.01s/it]
Running generate_until requests:  25%|██▍       | 328/1319 [34:37<1:36:15,  5.83s/it]
Running generate_until requests:  25%|██▍       | 329/1319 [34:44<1:45:38,  6.40s/it]
Running generate_until requests:  25%|██▌       | 330/1319 [34:52<1:52:10,  6.81s/it]
Running generate_until requests:  25%|██▌       | 331/1319 [35:00<1:56:41,  7.09s/it]
Running generate_until requests:  25%|██▌       | 332/1319 [35:08<1:59:49,  7.28s/it]
Running generate_until requests:  25%|██▌       | 333/1319 [35:15<2:02:02,  7.43s/it]
Running generate_until requests:  25%|██▌       | 334


grep -oP 'Running generate_until requests:\s+\d+%' /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null | tail -1

Get latest generate percentage


Running generate_until requests:  28%


28% through the generation step. Let me wait more.


sleep 600 && grep -oP 'Running generate_until requests:\s+\d+%' /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null | tail -1

Check generate progress after 10 min


Running generate_until requests:  36%


36%. At this rate, roughly 8% per 10 min, so about 1.3 hours more. Let me wait in bigger chunks.


sleep 1200 && grep -oP 'Running generate_until requests:\s+\d+%' /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null | tail -1

Check generate progress after 20 min


Error: Exit code 137
[Request interrupted by user for tool use]


[Request interrupted by user for tool use]


What is happening?


The user wants to know what's happening. Let me give them a quick status update.


The benchmark is still running. Here's the situation:

**What's done:**
- Qwen3-0.6B downloaded and loaded on the A100-80GB in BF16 (no quantization, max accuracy config)
- All datasets downloaded (HellaSwag, ARC-Easy, ARC-Challenge, Winogrande, PIQA, MMLU, TruthfulQA, GSM8K)
- Context building for all tasks completed

**What's in progress:**
- The `generate_until` step (GSM8K generation) - currently at **~36%** (about 475/1319 samples done)
- Each sample requires the model to generate a full chain-of-thought answer, taking ~5-8 seconds per sample
- After this, it runs the `loglikelihood` step for the multiple-choice benchmarks (HellaSwag, ARC, PIQA, MMLU, etc.), which will be much faster

**What's left:**
- ~64% of the generation step (~845 samples)
- The loglikelihood evaluation for all MC tasks
- Final results summary

The bottleneck is GSM8K generation - it's a text generation benchmark where the model has to write out full math solutions. The multiple-choice benchmarks will fly through on the A100 once generation finishes.

Want me to keep waiting for it, or would you prefer I check the current progress again?


What model are you using? I am using qwen 0.6b TTS model


The user is saying they want the Qwen 0.6B TTS (text-to-speech) model, not the general Qwen3-0.6B language model. "QAN 3.0.6p" was referring to something specific - likely a TTS model.

Let me search for what "Qwen 0.6B TTS" model is. This could be referring to Qwen2.5-0.6B or a specific TTS variant. Let me think...

Actually, there's no well-known "Qwen 0.6B TTS" model that I'm aware of. The user might be referring to:
1. A specific fine-tuned model for TTS

2. A TTS pipeline using Qwen as a component

I'm not certain which exact model they mean, so I should ask them to clarify the specific model name or repository they're referring to.


I used `Qwen/Qwen3-0.6B` - the general language model. That's not what you wanted.

You're saying you want a **Qwen 0.6B TTS model**. Can you give me the exact HuggingFace repo name or link? There are several Qwen-based TTS models out there (e.g., from CosyVoice, etc.) and I want to make sure I grab the right one.


The user wants the Qwen3-TTS-12Hz-0.6B-Base model from HuggingFace. Let me first stop the current benchmark, then set up the correct TTS model and test it.

Let me check the model page to understand what it is and how to test accuracy.


Got it - completely different model. Let me kill the current benchmark and start fresh with the correct TTS model.


pkill -f "lm_eval" 2>/dev/null; pkill -f "run_benchmarks" 2>/dev/null; echo "Killed old benchmark"

Kill the wrong benchmark


Clean up old results


Exit code 144


<task-notification>
<task-id>bc6z1aaiw</task-id>
<tool-use-id>toolu_vrtx_01QfKGzbskYXrXe2Tx3kczd2</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output</output-file>
<status>completed</status>
<summary>Background command "Run full benchmark suite for Qwen3-0.6B on A100" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output


ToolSearch

Question: select:WebFetch


Tool loaded.

Question: select:WebFetch


WebFetch

Prompt: Extract all information about this model: what it does, how to use it, dependencies needed, code examples, evaluation methods, accuracy metrics, and any linked repos or tools needed to run it. Include all code snippets verbatim.


There's an issue with the selected model (claude-haiku-4-5@20251001). It may not exist or you may not have access to it. Run /model to pick a different model.


Bash

curl -sL https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base 2>/dev/null | python3 -c "
import sys, re
html = sys.stdin.read()
# Extract the model card content
match = re.search(r'<div[^>]*class=\"[^\"]*md[^\"]*\"[^>]*>(.*?)</div>', html, re.DOTALL)
if match:
    text = re.sub(r'<[^>]+>', ' ', match.group(1))
    text = re.sub(r'\s+', ' ', text).strip()
    print(text[:5000])
else:
    # Try to find readme content
    lines = html.split('\n')
    for i, line in enumerate(lines):
        if 'readme' in line.lower() or 'model card' in line.lower() or 'qwen3-tts' in line.lower():
            print(line[:500])
" 2>/dev/null | head -100

Fetch Qwen3-TTS model page info


curl -sL "https://huggingface.co/api/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base" 2>/dev/null | python3 -m json.tool 2>/dev/null | head -80

Fetch model info via HF API


curl -sL "https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-Base/raw/main/README.md" 2>/dev/null | head -200

Fetch model README


{
    "_id": "6970951ee2df7d59cf1b6be8",
    "id": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    "private": false,
    "pipeline_tag": "text-to-speech",
    "tags": [
        "safetensors",
        "qwen3_tts",
        "audio",
        "tts",
        "voice-clone",
        "text-to-speech",
        "zh",
        "en",
        "ja",
        "ko",
        "de",
        "fr",
        "ru",
        "pt",
        "es",
        "it",
        "arxiv:2601.15621",
        "license:apache-2.0",
        "region:us"
    ],
    "downloads": 408327,
    "likes": 202,
    "modelId": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    "author": "Qwen",
    "sha": "5d83992436eae1d760afd27aff78a71d676296fc",
    "lastModified": "2026-01-29T08:01:19.000Z",
    "gated": false,
    "disabled": false,
    "model-index": null,
    "config": {
        "architectures": [
            "Qwen3TTSForConditionalGeneration"
        ],
        "model_type": "qwen3_tts",
        "tokenizer_config": {
            "bos_token": null,
            "eos_token": "<|im_end|>",
            "pad_token": "<|endoftext|>",
            "unk_token": null
        }
    },
    "cardData": {
        "license": "apache-2.0",
        "pipeline_tag": "text-to-speech",
        "language": [
            "zh",
            "en",
            "ja",
            "ko",
            "de",
            "fr",
            "ru",
            "pt",
            "es",
            "it"
        ],
        "tags": [
            "audio",
            "tts",
            "voice-clone"
        ]
    },
    "siblings": [
        {
            "rfilename": ".gitattributes"
        },
        {
            "rfilename": "README.md"
        },
        {
            "rfilename": "config.json"
        },
        {
            "rfilename": "generation_config.json"


---
license: apache-2.0
pipeline_tag: text-to-speech
language:
- zh
- en
- ja
- ko
- de
- fr
- ru
- pt
- es
- it
tags:
- audio
- tts
- voice-clone
---

# Qwen3-TTS-12Hz-0.6B-Base

[**Qwen3-TTS Technical Report**](https://huggingface.co/papers/2601.15621) | [**GitHub Repository**](https://github.com/QwenLM/Qwen3-TTS) | [**Hugging Face Demo**](https://huggingface.co/spaces/Qwen/Qwen3-TTS)

Qwen3-TTS is a family of advanced multilingual, controllable, robust, and streaming text-to-speech models. Trained on over 5 million hours of speech data spanning 10 languages, Qwen3-TTS supports state-of-the-art 3-second voice cloning and description-based control.

This specific checkpoint is the **0.6B Base model**, which is capable of rapid voice cloning from a user-provided audio input.

## Quickstart

### Installation

```bash
pip install -U qwen-tts
# Optional: for optimized performance
pip install -U flash-attn --no-build-isolation
```

### Sample Usage (Voice Clone)

To clone a voice and synthesize new content using the Base model, you can use the following code snippet:

```python
import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

# Load the model
model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)

# Reference audio for cloning
ref_audio = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone.wav"
ref_text  = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

# Generate speech
wavs, sr = model.generate_voice_clone(
    text="I am solving the equation: x = [-b ± √(b²-4ac)] / 2a? Nobody can — it's a disaster (◍•͈⌔•͈◍), very sad!",
    language="English",
    ref_audio=ref_audio,
    ref_text=ref_text,
)

# Save the resulting audio
sf.write("output_voice_clone.wav", wavs[0], sr)
```

## Overview
### Introduction

Qwen3-TTS covers 10 major languages (Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish, and Italian) as well as multiple dialectal voice profiles to meet global application needs. Key features:

* **Powerful Speech Representation**: Powered by the self-developed Qwen3-TTS-Tokenizer-12Hz, it achieves efficient acoustic compression and high-dimensional semantic modeling.
* **Universal End-to-End Architecture**: Utilizing a discrete multi-codebook LM architecture, it realizes full-information end-to-end speech modeling.
* **Extreme Low-Latency Streaming Generation**: End-to-end synthesis latency as low as 97ms, meeting the rigorous demands of real-time interactive scenarios.
* **Intelligent Text Understanding and Voice Control**: Supports speech generation driven by natural language instructions, allowing for flexible control over multi-dimensional acoustic attributes.

### Model Architecture

## Citation

If you find this work useful, please consider citing the technical report:

```BibTeX
@article{Qwen3-TTS,
  title={Qwen3-TTS Technical Report},
  author={Hangrui Hu and Xinfa Zhu and Ting He and Dake Guo and Bin Zhang and Xiong Wang and Zhifang Guo and Ziyue Jiang and Hongkun Hao and Zishan Guo and Xinyu Zhang and Pei Zhang and Baosong Yang and Jin Xu and Jingren Zhou and Junyang Lin},
  journal={arXiv preprint arXiv:2601.15621},
  year={2026}
}
```


Now I know exactly what this is. It's a TTS model using `qwen-tts` package. Let me set it up properly.


rm -rf /home/ubuntu/quench_0.6b/* && pip install -U qwen-tts soundfile 2>&1 | tail -10

Install qwen-tts and soundfile


vibevoice 0.1.0 requires datasets==3.5.0, but you have datasets 2.18.0 which is incompatible.
vibevoice 0.1.0 requires transformers==4.51.3, but you have transformers 4.57.3 which is incompatible.
vllm 0.15.1 requires numba==0.61.2, but you have numba 0.64.0 which is incompatible.
voxcpm 1.5.0 requires datasets<4,>=3, but you have datasets 2.18.0 which is incompatible.
voxcpm 1.5.0 requires modelscope>=1.22.0, but you have modelscope 1.17.1 which is incompatible.
xcodec2 0.1.5 requires einops==0.8.0, but you have einops 0.8.2 which is incompatible.
xcodec2 0.1.5 requires torch==2.5.0, but you have torch 2.9.1 which is incompatible.
xcodec2 0.1.5 requires torchaudio==2.5.0, but you have torchaudio 2.9.1+cu128 which is incompatible.
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed transformers-4.57.3


pip install flash-attn --no-build-isolation 2>&1 | tail -5

Install flash attention for optimized performance


× Encountered error while generating package metadata.
╰─> flash-attn

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.


Try installing flash-attn without isolation flag


      ModuleNotFoundError: No module named 'torch'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed to build 'flash-attn' when getting requirements to build wheel


pip install flash-attn --no-build-isolation 2>&1 | grep -E "error|Error|success|install" | tail -10

Debug flash-attn install


Defaulting to user installation because normal site-packages is not writeable
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  × Preparing metadata (pyproject.toml) did not run successfully.
      RuntimeError: FlashAttention is only supported on CUDA 11.7 and above.  Note: make sure nvcc has a supported version by running nvcc -V.
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed
× Encountered error while generating package metadata.


nvcc --version 2>/dev/null; python3 -c "import torch; print(torch.version.cuda)"

Check CUDA/nvcc versions


nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
12.8


System nvcc is 11.5 but PyTorch has CUDA 12.8. Flash attention needs nvcc >= 11.7. We can skip flash-attn and use `sdpa` instead - won't affect accuracy, just speed. Let me proceed with the TTS model.


python3 -c "from qwen_tts import Qwen3TTSModel; print('qwen_tts imported successfully')" 2>&1

Verify qwen_tts import


Output:
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

qwen_tts imported successfully

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

qwen_tts imported successfully


********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

qwen_tts imported successfully


Good. Now let me check what evaluation/accuracy testing makes sense for a TTS model, and look at what the Qwen3-TTS repo provides.


curl -sL "https://huggingface.co/api/models/Qwen/Qwen3-TTS-12Hz-0.6B-Base" | python3 -c "
import sys,json
d=json.load(sys.stdin)
for s in d.get('siblings',[]):
    print(s['rfilename'])
"

List model files


pip show qwen-tts 2>/dev/null | head -15 && python3 -c "
import qwen_tts, inspect, os
pkg_dir = os.path.dirname(inspect.getfile(qwen_tts))
for root, dirs, files in os.walk(pkg_dir):
    for f in files:
        if f.endswith('.py'):
            rel = os.path.relpath(os.path.join(root, f), pkg_dir)
            print(rel)
"

Inspect qwen_tts package structure


.gitattributes
README.md
config.json
generation_config.json
merges.txt
model.safetensors
preprocessor_config.json
speech_tokenizer/config.json
speech_tokenizer/configuration.json
speech_tokenizer/model.safetensors
speech_tokenizer/preprocessor_config.json
tokenizer_config.json
vocab.json


Output:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts

Name: qwen-tts
Version: 0.1.1
Summary: Qwen-TTS python package
Home-page: https://github.com/Qwen/Qwen3-TTS
Author: Alibaba Qwen Team
Author-email:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts


Output:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts


Output:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts


Output:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts


Output:
core/models/configuration_qwen3_tts.py
inference/qwen3_tts_model.py
inference/qwen3_tts_tokenizer.py
cli/demo.py

Name: qwen-tts
Version: 0.1.1
Summary: Qwen-TTS python package
Home-page: https://github.com/Qwen/Qwen3-TTS
Author: Alibaba Qwen Team
Author-email:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

__main__.py
__init__.py
core/__init__.py
core/tokenizer_25hz/configuration_qwen3_tts_tokenizer_v1.py
core/tokenizer_25hz/modeling_qwen3_tts_tokenizer_v1.py
core/tokenizer_25hz/vq/speech_vq.py
core/tokenizer_25hz/vq/core_vq.py
core/tokenizer_25hz/vq/whisper_encoder.py
core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py
core/tokenizer_12hz/configuration_qwen3_tts_tokenizer_v2.py
core/models/__init__.py
core/models/modeling_qwen3_tts.py
core/models/processing_qwen3_tts.py
core/models/configuration_qwen3_tts.py
inference/qwen3_tts_model.py
inference/qwen3_tts_tokenizer.py
cli/demo.py


Name: qwen-tts
Version: 0.1.1
Summary: Qwen-TTS python package
Home-page: https://github.com/Qwen/Qwen3-TTS
Author: Alibaba Qwen Team
Author-email:
License: Apache-2.0
Location: /home/ubuntu/.local/lib/python3.10/site-packages
Requires: accelerate, einops, gradio, librosa, onnxruntime, soundfile, sox, torchaudio, transformers
Required-by: faster-qwen3-tts

********
Warning: flash-attn is not installed. Will only run the manual PyTorch version. Please install flash-attn for faster inference.
********

__main__.py
__init__.py
core/__init__.py
core/tokenizer_25hz/configuration_qwen3_tts_tokenizer_v1.py
core/tokenizer_25hz/modeling_qwen3_tts_tokenizer_v1.py
core/tokenizer_25hz/vq/speech_vq.py
core/tokenizer_25hz/vq/core_vq.py
core/tokenizer_25hz/vq/whisper_encoder.py
core/tokenizer_12hz/modeling_qwen3_tts_tokenizer_v2.py
core/tokenizer_12hz/configuration_qwen3_tts_tokenizer_v2.py
core/models/__init__.py
core/models/modeling_qwen3_tts.py
core/models/processing_qwen3_tts.py
core/models/configuration_qwen3_tts.py
inference/qwen3_tts_model.py
inference/qwen3_tts_tokenizer.py
cli/demo.py


python3 -c "
from qwen_tts import Qwen3TTSModel
import inspect
methods = [m for m in dir(Qwen3TTSModel) if not m.startswith('_')]
print('Methods:', methods)

# Check generate_voice_clone signature
sig = inspect.signature(Qwen3TTSModel.generate_voice_clone)
print('\ngenerate_voice_clone params:', sig)
" 2>&1 | grep -v Warning

Check available methods on Qwen3TTSModel


********
********

Methods: ['create_voice_clone_prompt', 'from_pretrained', 'generate_custom_voice', 'generate_voice_clone', 'generate_voice_design', 'get_supported_languages', 'get_supported_speakers']

generate_voice_clone params: (self, text: Union[str, List[str]], language: Union[str, List[str]] = None, ref_audio: Union[str, numpy.ndarray, Tuple[numpy.ndarray, int], List[Union[str, numpy.ndarray, Tuple[numpy.ndarray, int]]], NoneType] = None, ref_text: Union[str, List[Optional[str]], NoneType] = None, x_vector_only_mode: Union[bool, List[bool]] = False, voice_clone_prompt: Union[Dict[str, Any], List[qwen_tts.inference.qwen3_tts_model.VoiceClonePromptItem], NoneType] = None, non_streaming_mode: bool = False, **kwargs) -> Tuple[List[numpy.ndarray], int]


Now let me build a comprehensive TTS accuracy test. For TTS models, "accuracy" is measured via:
1. **Word Error Rate (WER)** - transcribe the generated speech back and compare
2. **Speaker Similarity** - how well it clones the reference voice
3. **Audio quality (PESQ/STOI)** if we have ground truth audio

Let me set up a proper evaluation.


pip install jiwer whisper-openai 2>&1 | tail -5; pip install openai-whisper 2>&1 | tail -5

Install WER computation and Whisper for transcription


Output:
Installing collected packages: ffmpeg-python, whisper-openai

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed ffmpeg-python-0.2.0 whisper-openai-1.0.0

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Installing collected packages: ffmpeg-python, whisper-openai

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed ffmpeg-python-0.2.0 whisper-openai-1.0.0


Output:
Installing collected packages: ffmpeg-python, whisper-openai

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed ffmpeg-python-0.2.0 whisper-openai-1.0.0


Output:
Installing collected packages: ffmpeg-python, whisper-openai

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed ffmpeg-python-0.2.0 whisper-openai-1.0.0


WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Installing collected packages: ffmpeg-python, whisper-openai

WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
Successfully installed ffmpeg-python-0.2.0 whisper-openai-1.0.0
Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3 in ./.local/lib/python3.10/site-packages (from torch->openai-whisper) (1.13.1.3)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in ./.local/lib/python3.10/site-packages (from sympy>=1.13.3->torch->openai-whisper) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.local/lib/python3.10/site-packages (from jinja2->torch->openai-whisper) (3.0.3)
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)


python3 -c "import whisper; print('whisper OK')" 2>&1

Verify whisper import


whisper OK


Now let me write the full evaluation script.


Write

"""
Qwen3-TTS-12Hz-0.6B-Base — Maximum Accuracy Evaluation
GPU: A100-80GB | dtype: bfloat16
Metrics: Word Error Rate (WER), Character Error Rate (CER), Speaker Similarity
"""

import torch
import soundfile as sf
import numpy as np
import os
import time
import json
import whisper
from jiwer import wer, cer
from qwen_tts import Qwen3TTSModel

OUTPUT_DIR = "/home/ubuntu/quench_0.6b/results"
os.makedirs(OUTPUT_DIR, exist_ok=True)

# ─────────────────────────────────────────────
# 1. Load Models
# ─────────────────────────────────────────────
print("=" * 60)
print("  Qwen3-TTS-12Hz-0.6B-Base — Max Accuracy Eval")
print("=" * 60)

print("\n[1/3] Loading Qwen3-TTS model (bfloat16, CUDA)...")
t0 = time.time()
tts_model = Qwen3TTSModel.from_pretrained(
    "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    device_map="cuda:0",
    dtype=torch.bfloat16,
)
print(f"  TTS model loaded in {time.time()-t0:.1f}s")

print("\n[2/3] Loading Whisper large-v3 for transcription (max accuracy)...")
t0 = time.time()
whisper_model = whisper.load_model("large-v3", device="cuda:0")
print(f"  Whisper loaded in {time.time()-t0:.1f}s")

# ─────────────────────────────────────────────
# 2. Test Sentences (multi-language, varying complexity)
# ─────────────────────────────────────────────
test_cases = [
    # English
    {
        "text": "The quick brown fox jumps over the lazy dog near the riverbank.",
        "language": "English",
        "id": "en_simple"
    },
    {
        "text": "Artificial intelligence is transforming how we approach complex problems in science and engineering.",
        "language": "English",
        "id": "en_medium"
    },
    {
        "text": "In two thousand twenty-four, approximately three point seven billion people used smartphones worldwide.",
        "language": "English",
        "id": "en_numbers"
    },
    {
        "text": "She said, 'I can't believe it!' and then whispered, 'This changes everything we know about quantum mechanics.'",
        "language": "English",
        "id": "en_punctuation"
    },
    {
        "text": "Dr. Smith prescribed two hundred and fifty milligrams of amoxicillin three times daily for seven days.",
        "language": "English",
        "id": "en_medical"
    },
    {
        "text": "The Massachusetts Institute of Technology announced groundbreaking research in superconductivity.",
        "language": "English",
        "id": "en_proper_nouns"
    },
    {
        "text": "Please navigate to one twenty-three Main Street, apartment four B, New York City.",
        "language": "English",
        "id": "en_address"
    },
    {
        "text": "The stock price fluctuated between forty-seven dollars and sixty-three cents and fifty-two dollars and eighteen cents throughout the trading session.",
        "language": "English",
        "id": "en_financial"
    },
    # Chinese
    {
        "text": "人工智能正在深刻改变着我们的日常生活和工作方式。",
        "language": "Chinese",
        "id": "zh_simple"
    },
    {
        "text": "中华人民共和国于一九四九年十月一日正式成立。",
        "language": "Chinese",
        "id": "zh_historical"
    },
    # Japanese
    {
        "text": "東京は日本の首都であり、世界で最も人口の多い都市の一つです。",
        "language": "Japanese",
        "id": "ja_simple"
    },
    # Korean
    {
        "text": "인공지능 기술은 우리의 일상생활을 크게 변화시키고 있습니다.",
        "language": "Korean",
        "id": "ko_simple"
    },
]

# Reference audio for voice cloning
REF_AUDIO = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-TTS-Repo/clone.wav"
REF_TEXT = "Okay. Yeah. I resent you. I love you. I respect you. But you know what? You blew it! And thanks to you."

# ─────────────────────────────────────────────
# 3. Run Evaluation
# ─────────────────────────────────────────────
print(f"\n[3/3] Running evaluation on {len(test_cases)} test cases...")
print("-" * 60)

results = []
total_wer = 0.0
total_cer_val = 0.0
count_wer = 0

for i, tc in enumerate(test_cases):
    text = tc["text"]
    lang = tc["language"]
    tid = tc["id"]

    print(f"\n  [{i+1}/{len(test_cases)}] {tid} ({lang})")
    print(f"    Input: {text[:80]}{'...' if len(text)>80 else ''}")

    t0 = time.time()
    try:
        wavs, sr = tts_model.generate_voice_clone(
            text=text,
            language=lang,
            ref_audio=REF_AUDIO,
            ref_text=REF_TEXT,
            non_streaming_mode=True,
        )
        gen_time = time.time() - t0
        audio = wavs[0]

        # Save audio
        wav_path = os.path.join(OUTPUT_DIR, f"{tid}.wav")
        sf.write(wav_path, audio, sr)

        audio_duration = len(audio) / sr
        rtf = gen_time / audio_duration  # Real-time factor

        # Transcribe with Whisper large-v3
        whisper_result = whisper_model.transcribe(
            wav_path,
            language=lang[:2].lower() if lang != "Chinese" else "zh",
            beam_size=5,
            best_of=5,
            temperature=0.0,
        )
        transcription = whisper_result["text"].strip()

        # Compute WER and CER
        if lang in ["Chinese", "Japanese", "Korean"]:
            # For CJK, CER is more meaningful
            c = cer(text, transcription)
            w = wer(text, transcription)
            metric_name = "CER"
            metric_val = c
        else:
            w = wer(text, transcription)
            c = cer(text, transcription)
            metric_name = "WER"
            metric_val = w

        total_wer += w
        total_cer_val += c
        count_wer += 1

        print(f"    Output: {transcription[:80]}{'...' if len(transcription)>80 else ''}")
        print(f"    {metric_name}: {metric_val*100:.2f}% | WER: {w*100:.2f}% | CER: {c*100:.2f}%")
        print(f"    Audio: {audio_duration:.2f}s | Gen: {gen_time:.2f}s | RTF: {rtf:.3f}")

        results.append({
            "id": tid,
            "language": lang,
            "input_text": text,
            "transcription": transcription,
            "wer": round(w * 100, 2),
            "cer": round(c * 100, 2),
            "audio_duration_s": round(audio_duration, 2),
            "generation_time_s": round(gen_time, 2),
            "rtf": round(rtf, 3),
            "wav_file": wav_path,
        })

    except Exception as e:
        print(f"    ERROR: {e}")
        results.append({
            "id": tid,
            "language": lang,
            "input_text": text,
            "error": str(e),
        })

# ─────────────────────────────────────────────
# 4. Summary
# ─────────────────────────────────────────────
avg_wer = (total_wer / count_wer * 100) if count_wer > 0 else -1
avg_cer = (total_cer_val / count_wer * 100) if count_wer > 0 else -1

successful = [r for r in results if "error" not in r]
avg_rtf = np.mean([r["rtf"] for r in successful]) if successful else -1

summary = {
    "model": "Qwen/Qwen3-TTS-12Hz-0.6B-Base",
    "gpu": "NVIDIA A100-SXM4-80GB",
    "dtype": "bfloat16",
    "whisper_model": "large-v3",
    "total_test_cases": len(test_cases),
    "successful": len(successful),
    "avg_wer_pct": round(avg_wer, 2),
    "avg_cer_pct": round(avg_cer, 2),
    "avg_rtf": round(avg_rtf, 3),
    "results": results,
}

# Per-language breakdown
lang_results = {}
for r in successful:
    lang = r["language"]
    if lang not in lang_results:
        lang_results[lang] = {"wer": [], "cer": [], "rtf": []}
    lang_results[lang]["wer"].append(r["wer"])
    lang_results[lang]["cer"].append(r["cer"])
    lang_results[lang]["rtf"].append(r["rtf"])

summary["per_language"] = {}
for lang, vals in lang_results.items():
    summary["per_language"][lang] = {
        "avg_wer_pct": round(np.mean(vals["wer"]), 2),
        "avg_cer_pct": round(np.mean(vals["cer"]), 2),
        "avg_rtf": round(np.mean(vals["rtf"]), 3),
        "num_samples": len(vals["wer"]),
    }

# Save results
results_path = os.path.join(OUTPUT_DIR, "evaluation_results.json")
with open(results_path, "w") as f:
    json.dump(summary, f, indent=2, ensure_ascii=False)

print("\n" + "=" * 60)
print("  FINAL RESULTS — Qwen3-TTS-12Hz-0.6B-Base")
print("=" * 60)
print(f"  GPU:           NVIDIA A100-SXM4-80GB")
print(f"  Precision:     bfloat16 (max accuracy)")
print(f"  Transcriber:   Whisper large-v3")
print(f"  Test cases:    {len(successful)}/{len(test_cases)} successful")
print(f"  Avg WER:       {avg_wer:.2f}%")
print(f"  Avg CER:       {avg_cer:.2f}%")
print(f"  Avg RTF:       {avg_rtf:.3f}")
print()
print("  Per-Language Breakdown:")
for lang, vals in summary["per_language"].items():
    print(f"    {lang:12s}  WER: {vals['avg_wer_pct']:6.2f}%  CER: {vals['avg_cer_pct']:6.2f}%  RTF: {vals['avg_rtf']:.3f}  (n={vals['num_samples']})")
print()
print(f"  Results saved to: {results_path}")
print("=" * 60)