<retrieval_status>timeout</retrieval_status>

<task_id>bc6z1aaiw</task_id>

<task_type>local_bash</task_type>

<status>running</status>

<output>
============================================
  Qwen3-0.6B Full Accuracy Benchmark
  GPU: NVIDIA A100-SXM4-80GB
  dtype: bfloat16 (full precision, no quant)
============================================
2026-03-21:10:35:20 INFO     [_cli.run:376] Selected Tasks: ['hellaswag', 'arc_easy', 'arc_challenge', 'winogrande', 'piqa', 'mmlu', 'truthfulqa_mc2', 'gsm8k']
2026-03-21:10:35:21 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-03-21:10:35:21 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': 'Qwen/Qwen3-0.6B', 'dtype': 'bfloat16', 'trust_remote_code': True}
2026-03-21:10:35:23 INFO     [models.huggingface:161] Using device 'cuda:0'
2026-03-21:10:35:24 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 7.02kB [00:00, 27.1MB/s]

Downloading data:   0%|          | 0.00/24.4M [00:00<?, ?B/s]
Downloading data:  43%|████▎     | 10.5M/24.4M [00:00<00:00, 46.6MB/s]
Downloading data: 100%|██████████| 24.4M/24.4M [00:00<00:00, 85.9MB/s]

Downloading data:   0%|          | 0.00/6.11M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.11M/6.11M [00:00<00:00, 68.7MB/s]

Downloading data:   0%|          | 0.00/6.32M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.32M/6.32M [00:00<00:00, 64.5MB/s]

Generating train split:   0%|          | 0/39905 [00:00<?, ? examples/s]
Generating train split:  70%|███████   | 28000/39905 [00:00<00:00, 266674.49 examples/s]
Generating train split: 100%|██████████| 39905/39905 [00:00<00:00, 226653.28 examples/s]

Generating test split:   0%|          | 0/10003 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 226416.32 examples/s]

Generating validation split:   0%|          | 0/10042 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:00<00:00, 216312.05 examples/s]

Map:   0%|          | 0/39905 [00:00<?, ? examples/s]
Map:   3%|▎         | 1000/39905 [00:00<00:13, 2934.46 examples/s]
Map:   5%|▌         | 2032/39905 [00:00<00:07, 5109.09 examples/s]
Map:   8%|▊         | 3125/39905 [00:00<00:05, 6833.81 examples/s]
Map:  11%|█         | 4228/39905 [00:00<00:04, 8081.15 examples/s]
Map:  13%|█▎        | 5326/39905 [00:00<00:03, 8943.96 examples/s]
Map:  16%|█▌        | 6420/39905 [00:00<00:03, 9537.21 examples/s]
Map:  19%|█▉        | 7646/39905 [00:00<00:03, 10051.79 examples/s]
Map:  22%|██▏       | 8755/39905 [00:01<00:03, 10352.90 examples/s]
Map:  25%|██▍       | 9857/39905 [00:01<00:02, 10546.45 examples/s]
Map:  27%|██▋       | 10947/39905 [00:01<00:02, 10649.55 examples/s]
Map:  32%|███▏      | 12651/39905 [00:01<00:02, 10572.07 examples/s]
Map:  34%|███▍      | 13732/39905 [00:01<00:02, 10632.99 examples/s]
Map:  37%|███▋      | 14819/39905 [00:01<00:02, 10693.86 examples/s]
Map:  40%|████      | 16000/39905 [00:01<00:03, 6572.81 examples/s]
Map:  43%|████▎     | 17000/39905 [00:02<00:03, 7124.51 examples/s]
Map:  45%|████▌     | 18000/39905 [00:02<00:02, 7647.02 examples/s]
Map:  48%|████▊     | 19000/39905 [00:02<00:02, 8013.93 examples/s]
Map:  50%|█████     | 20000/39905 [00:02<00:02, 8410.05 examples/s]
Map:  53%|█████▎    | 21000/39905 [00:02<00:02, 8716.28 examples/s]
Map:  55%|█████▌    | 22000/39905 [00:02<00:02, 8941.50 examples/s]
Map:  58%|█████▊    | 23000/39905 [00:02<00:01, 8952.72 examples/s]
Map:  60%|██████    | 24000/39905 [00:02<00:01, 9107.57 examples/s]
Map:  63%|██████▎   | 25000/39905 [00:02<00:01, 9226.00 examples/s]
Map:  65%|██████▌   | 26000/39905 [00:03<00:01, 9319.35 examples/s]
Map:  68%|██████▊   | 27000/39905 [00:03<00:01, 9351.83 examples/s]
Map:  70%|███████   | 28000/39905 [00:03<00:01, 9396.54 examples/s]
Map:  73%|███████▎  | 29000/39905 [00:03<00:01, 9432.13 examples/s]
Map:  75%|███████▌  | 30000/39905 [00:03<00:01, 9482.60 examples/s]
Map:  78%|███████▊  | 31029/39905 [00:03<00:01, 5608.07 examples/s]
Map:  80%|████████  | 32000/39905 [00:03<00:01, 6374.99 examples/s]
Map:  83%|████████▎ | 33000/39905 [00:04<00:00, 7012.16 examples/s]
Map:  85%|████████▌ | 34000/39905 [00:04<00:00, 7621.28 examples/s]
Map:  88%|████████▊ | 35000/39905 [00:04<00:00, 8090.19 examples/s]
Map:  90%|█████████ | 36000/39905 [00:04<00:00, 8475.80 examples/s]
Map:  93%|█████████▎| 37000/39905 [00:04<00:00, 8705.30 examples/s]
Map:  95%|█████████▌| 38000/39905 [00:04<00:00, 8934.10 examples/s]
Map:  98%|█████████▊| 39000/39905 [00:04<00:00, 9103.01 examples/s]
Map: 100%|██████████| 39905/39905 [00:04<00:00, 8393.35 examples/s]

Map:   0%|          | 0/10042 [00:00<?, ? examples/s]
Map:  10%|▉         | 1000/10042 [00:00<00:00, 9065.59 examples/s]
Map:  21%|██        | 2078/10042 [00:00<00:00, 10030.81 examples/s]
Map:  32%|███▏      | 3175/10042 [00:00<00:00, 10446.18 examples/s]
Map:  47%|████▋     | 4708/10042 [00:00<00:00, 10328.88 examples/s]
Map:  60%|█████▉    | 6000/10042 [00:00<00:00, 5318.52 examples/s]
Map:  70%|██████▉   | 7000/10042 [00:01<00:00, 6100.03 examples/s]
Map:  80%|███████▉  | 8000/10042 [00:01<00:00, 6819.77 examples/s]
Map:  90%|████████▉ | 9000/10042 [00:01<00:00, 7442.93 examples/s]
Map: 100%|█████████▉| 10000/10042 [00:01<00:00, 7978.02 examples/s]
Map: 100%|██████████| 10042/10042 [00:01<00:00, 7525.67 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 39.0MB/s]

Downloading data:   0%|          | 0.00/331k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.54MB/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.53MB/s]

Downloading data:   0%|          | 0.00/346k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.08MB/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.07MB/s]

Downloading data:   0%|          | 0.00/86.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 86.1k/86.1k [00:00<00:00, 1.07MB/s]

Generating train split:   0%|          | 0/2251 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 2251/2251 [00:00<00:00, 365675.60 examples/s]

Generating test split:   0%|          | 0/2376 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 2376/2376 [00:00<00:00, 514356.97 examples/s]

Generating validation split:   0%|          | 0/570 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 570/570 [00:00<00:00, 293632.19 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 33.3MB/s]

Downloading data:   0%|          | 0.00/190k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 190k/190k [00:00<00:00, 2.09MB/s]

Downloading data:   0%|          | 0.00/204k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 204k/204k [00:00<00:00, 2.31MB/s]

Downloading data:   0%|          | 0.00/55.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 55.7k/55.7k [00:00<00:00, 661kB/s]

Generating train split:   0%|          | 0/1119 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 1119/1119 [00:00<00:00, 252307.61 examples/s]

Generating test split:   0%|          | 0/1172 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1172/1172 [00:00<00:00, 347499.24 examples/s]

Generating validation split:   0%|          | 0/299 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 299/299 [00:00<00:00, 171488.70 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 11.2kB [00:00, 8.07MB/s]

Downloading data:   0%|          | 0.00/2.06M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.3MB/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.1MB/s]

Downloading data:   0%|          | 0.00/118k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]

Downloading data:   0%|          | 0.00/85.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 1.19MB/s]

Generating train split:   0%|          | 0/40398 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 1264347.22 examples/s]

Generating test split:   0%|          | 0/1767 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 644071.88 examples/s]

Generating validation split:   0%|          | 0/1267 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 621832.81 examples/s]

Downloading data:   0%|          | 0.00/2.64M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 21.0MB/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 20.7MB/s]

Downloading data:   0%|          | 0.00/300k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.76MB/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.74MB/s]

Downloading data:   0%|          | 0.00/496k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.14MB/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.12MB/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 16113 examples [00:00, 871946.38 examples/s]

Generating validation split: 0 examples [00:00, ? examples/s]
Generating validation split: 1838 examples [00:00, 654203.22 examples/s]

Generating test split: 0 examples [00:00, ? examples/s]
Generating test split: 3084 examples [00:00, 676422.82 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 110MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 43.0MB/s]

Downloading data:   0%|          | 0.00/18.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 268kB/s]

Downloading data:   0%|          | 0.00/4.94k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 70.0kB/s]

Downloading data:   0%|          | 0.00/3.30k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 46.0kB/s]

Generating test split:   0%|          | 0/171 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 171/171 [00:00<00:00, 73758.33 examples/s]

Generating validation split:   0%|          | 0/19 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 13837.78 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4454.44 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 109MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 294MB/s]

Downloading data:   0%|          | 0.00/1.04M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 14.5MB/s]

Downloading data:   0%|          | 0.00/116k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 1.59MB/s]

Downloading data:   0%|          | 0.00/15.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 215kB/s]

Generating test split:   0%|          | 0/1534 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 190368.14 examples/s]

Generating validation split:   0%|          | 0/170 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 86933.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3685.03 examples/s]

Downloading data:   0%|          | 0.00/54.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 801kB/s]

Downloading data:   0%|          | 0.00/9.89k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 145kB/s]

Downloading data:   0%|          | 0.00/4.62k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 67.2kB/s]

Generating test split:   0%|          | 0/324 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 324/324 [00:00<00:00, 148535.85 examples/s]

Generating validation split:   0%|          | 0/35 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 28750.61 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3903.13 examples/s]

Downloading data:   0%|          | 0.00/48.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 48.6k/48.6k [00:00<00:00, 724kB/s]

Downloading data:   0%|          | 0.00/9.15k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.15k/9.15k [00:00<00:00, 144kB/s]

Downloading data:   0%|          | 0.00/4.20k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.20k/4.20k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/311 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 311/311 [00:00<00:00, 123795.06 examples/s]

Generating validation split:   0%|          | 0/34 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 34/34 [00:00<00:00, 26988.33 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3915.52 examples/s]

Downloading data:   0%|          | 0.00/89.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 1.32MB/s]

Downloading data:   0%|          | 0.00/14.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 216kB/s]

Downloading data:   0%|          | 0.00/5.14k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 74.2kB/s]

Generating test split:   0%|          | 0/895 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 895/895 [00:00<00:00, 307596.04 examples/s]

Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 67356.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4344.63 examples/s]

Downloading data:   0%|          | 0.00/60.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 687kB/s]

Downloading data:   0%|          | 0.00/10.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 160kB/s]

Downloading data:   0%|          | 0.00/4.41k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 60.8kB/s]

Generating test split:   0%|          | 0/346 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 346/346 [00:00<00:00, 122116.22 examples/s]

Generating validation split:   0%|          | 0/38 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 24404.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4163.49 examples/s]

Downloading data:   0%|          | 0.00/23.0k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.0k/23.0k [00:00<00:00, 320kB/s]

Downloading data:   0%|          | 0.00/6.52k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.52k/6.52k [00:00<00:00, 85.0kB/s]

Downloading data:   0%|          | 0.00/4.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.12k/4.12k [00:00<00:00, 58.1kB/s]

Generating test split:   0%|          | 0/163 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 163/163 [00:00<00:00, 73096.50 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 12130.06 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4532.42 examples/s]

Downloading data:   0%|          | 0.00/23.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.3k/23.3k [00:00<00:00, 338kB/s]

Downloading data:   0%|          | 0.00/6.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.21k/6.21k [00:00<00:00, 98.4kB/s]

Downloading data:   0%|          | 0.00/4.05k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.05k/4.05k [00:00<00:00, 52.8kB/s]

Generating test split:   0%|          | 0/108 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 108/108 [00:00<00:00, 38008.46 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 7080.62 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3219.95 examples/s]

Downloading data:   0%|          | 0.00/29.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 29.5k/29.5k [00:00<00:00, 411kB/s]

Downloading data:   0%|          | 0.00/7.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.12k/7.12k [00:00<00:00, 106kB/s]

Downloading data:   0%|          | 0.00/4.96k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.96k/4.96k [00:00<00:00, 68.0kB/s]

Generating test split:   0%|          | 0/121 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 121/121 [00:00<00:00, 51113.99 examples/s]

Generating validation split:   0%|          | 0/13 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 13/13 [00:00<00:00, 10030.53 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3499.34 examples/s]

Downloading data:   0%|          | 0.00/202k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 202k/202k [00:00<00:00, 2.94MB/s]

Downloading data:   0%|          | 0.00/38.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 38.5k/38.5k [00:00<00:00, 521kB/s]

Downloading data:   0%|          | 0.00/10.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.2k/10.2k [00:00<00:00, 150kB/s]

Generating test split:   0%|          | 0/237 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 237/237 [00:00<00:00, 73285.91 examples/s]

Generating validation split:   0%|          | 0/26 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 18464.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3556.91 examples/s]

Downloading data:   0%|          | 0.00/155k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 155k/155k [00:00<00:00, 2.25MB/s]

Downloading data:   0%|          | 0.00/27.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 419kB/s]

Downloading data:   0%|          | 0.00/17.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 17.8k/17.8k [00:00<00:00, 258kB/s]

Generating test split:   0%|          | 0/204 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 204/204 [00:00<00:00, 79720.30 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 13615.86 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4526.55 examples/s]

Downloading data:   0%|          | 0.00/142k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 142k/142k [00:00<00:00, 1.96MB/s]

Downloading data:   0%|          | 0.00/31.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 31.6k/31.6k [00:00<00:00, 464kB/s]

Downloading data:   0%|          | 0.00/22.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.2k/22.2k [00:00<00:00, 311kB/s]

Generating test split:   0%|          | 0/165 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 165/165 [00:00<00:00, 63919.84 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 11865.07 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4293.04 examples/s]

Downloading data:   0%|          | 0.00/21.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 21.5k/21.5k [00:00<00:00, 318kB/s]

Downloading data:   0%|          | 0.00/6.56k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.56k/6.56k [00:00<00:00, 87.1kB/s]

Downloading data:   0%|          | 0.00/4.81k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 41.0kB/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 40.9kB/s]

Generating test split:   0%|          | 0/126 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 58350.70 examples/s]

Generating validation split:   0%|          | 0/14 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 10374.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4641.77 examples/s]

Downloading data:   0%|          | 0.00/19.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 292kB/s]

Downloading data:   0%|          | 0.00/5.27k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 74.2kB/s]

Downloading data:   0%|          | 0.00/4.22k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 60.5kB/s]

Generating test split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 49061.93 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 9145.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3454.38 examples/s]

Downloading data:   0%|          | 0.00/43.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 658kB/s]

Downloading data:   0%|          | 0.00/8.36k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 118kB/s]

Downloading data:   0%|          | 0.00/4.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 66.6kB/s]

Generating test split:   0%|          | 0/201 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 201/201 [00:00<00:00, 96140.39 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 18443.87 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3890.10 examples/s]

Downloading data:   0%|          | 0.00/114k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 1.65MB/s]

Downloading data:   0%|          | 0.00/18.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 285kB/s]

Downloading data:   0%|          | 0.00/7.49k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 106kB/s]

Generating test split:   0%|          | 0/245 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 245/245 [00:00<00:00, 98998.50 examples/s]

Generating validation split:   0%|          | 0/27 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 21278.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3775.93 examples/s]

Downloading data:   0%|          | 0.00/20.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 297kB/s]

Downloading data:   0%|          | 0.00/6.45k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 93.1kB/s]

Downloading data:   0%|          | 0.00/4.43k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 63.4kB/s]

Generating test split:   0%|          | 0/110 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 110/110 [00:00<00:00, 59089.84 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 10082.46 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4160.19 examples/s]

Downloading data:   0%|          | 0.00/133k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 1.83MB/s]

Downloading data:   0%|          | 0.00/22.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/4.69k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 70.8kB/s]

Generating test split:   0%|          | 0/612 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 612/612 [00:00<00:00, 245778.82 examples/s]

Generating validation split:   0%|          | 0/69 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 52042.25 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4654.13 examples/s]

Downloading data:   0%|          | 0.00/23.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.2k/23.2k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/5.26k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.26k/5.26k [00:00<00:00, 83.7kB/s]

Downloading data:   0%|          | 0.00/4.08k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.08k/4.08k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/131 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 131/131 [00:00<00:00, 65147.48 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 8817.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4652.07 examples/s]

Downloading data:
</output>

Run full benchmark suite for Qwen3-0.6B on A100

Output:
============================================
  Qwen3-0.6B Full Accuracy Benchmark
  GPU: NVIDIA A100-SXM4-80GB
  dtype: bfloat16 (full precision, no quant)
============================================
2026-03-21:10:35:20 INFO     [_cli.run:376] Selected Tasks: ['hellaswag', 'arc_easy', 'arc_challenge', 'winogrande', 'piqa', 'mmlu', 'truthfulqa_mc2', 'gsm8k']
2026-03-21:10:35:21 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-03-21:10:35:21 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': 'Qwen/Qwen3-0.6B', 'dtype': 'bfloat16', 'trust_remote_code': True}
2026-03-21:10:35:23 INFO     [models.huggingface:161] Using device 'cuda:0'
2026-03-21:10:35:24 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 7.02kB [00:00, 27.1MB/s]

Downloading data:   0%|          | 0.00/24.4M [00:00<?, ?B/s]
Downloading data:  43%|████▎     | 10.5M/24.4M [00:00<00:00, 46.6MB/s]
Downloading data: 100%|██████████| 24.4M/24.4M [00:00<00:00, 85.9MB/s]

Downloading data:   0%|          | 0.00/6.11M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.11M/6.11M [00:00<00:00, 68.7MB/s]

Downloading data:   0%|          | 0.00/6.32M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.32M/6.32M [00:00<00:00, 64.5MB/s]

Generating train split:   0%|          | 0/39905 [00:00<?, ? examples/s]
Generating train split:  70%|███████   | 28000/39905 [00:00<00:00, 266674.49 examples/s]
Generating train split: 100%|██████████| 39905/39905 [00:00<00:00, 226653.28 examples/s]

Generating test split:   0%|          | 0/10003 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 226416.32 examples/s]

Generating validation split:   0%|          | 0/10042 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:00<00:00, 216312.05 examples/s]

Map:   0%|          | 0/39905 [00:00<?, ? examples/s]
Map:   3%|▎         | 1000/39905 [00:00<00:13, 2934.46 examples/s]
Map:   5%|▌         | 2032/39905 [00:00<00:07, 5109.09 examples/s]
Map:   8%|▊         | 3125/39905 [00:00<00:05, 6833.81 examples/s]
Map:  11%|█         | 4228/39905 [00:00<00:04, 8081.15 examples/s]
Map:  13%|█▎        | 5326/39905 [00:00<00:03, 8943.96 examples/s]
Map:  16%|█▌        | 6420/39905 [00:00<00:03, 9537.21 examples/s]
Map:  19%|█▉        | 7646/39905 [00:00<00:03, 10051.79 examples/s]
Map:  22%|██▏       | 8755/39905 [00:01<00:03, 10352.90 examples/s]
Map:  25%|██▍       | 9857/39905 [00:01<00:02, 10546.45 examples/s]
Map:  27%|██▋       | 10947/39905 [00:01<00:02, 10649.55 examples/s]
Map:  32%|███▏      | 12651/39905 [00:01<00:02, 10572.07 examples/s]
Map:  34%|███▍      | 13732/39905 [00:01<00:02, 10632.99 examples/s]
Map:  37%|███▋      | 14819/39905 [00:01<00:02, 10693.86 examples/s]
Map:  40%|████      | 16000/39905 [00:01<00:03, 6572.81 examples/s]
Map:  43%|████▎     | 17000/39905 [00:02<00:03, 7124.51 examples/s]
Map:  45%|████▌     | 18000/39905 [00:02<00:02, 7647.02 examples/s]
Map:  48%|████▊     | 19000/39905 [00:02<00:02, 8013.93 examples/s]
Map:  50%|█████     | 20000/39905 [00:02<00:02, 8410.05 examples/s]
Map:  53%|█████▎    | 21000/39905 [00:02<00:02, 8716.28 examples/s]
Map:  55%|█████▌    | 22000/39905 [00:02<00:02, 8941.50 examples/s]
Map:  58%|█████▊    | 23000/39905 [00:02<00:01, 8952.72 examples/s]
Map:  60%|██████    | 24000/39905 [00:02<00:01, 9107.57 examples/s]
Map:  63%|██████▎   | 25000/39905 [00:02<00:01, 9226.00 examples/s]
Map:  65%|██████▌   | 26000/39905 [00:03<00:01, 9319.35 examples/s]
Map:  68%|██████▊   | 27000/39905 [00:03<00:01, 9351.83 examples/s]
Map:  70%|███████   | 28000/39905 [00:03<00:01, 9396.54 examples/s]
Map:  73%|███████▎  | 29000/39905 [00:03<00:01, 9432.13 examples/s]
Map:  75%|███████▌  | 30000/39905 [00:03<00:01, 9482.60 examples/s]
Map:  78%|███████▊  | 31029/39905 [00:03<00:01, 5608.07 examples/s]
Map:  80%|████████  | 32000/39905 [00:03<00:01, 6374.99 examples/s]
Map:  83%|████████▎ | 33000/39905 [00:04<00:00, 7012.16 examples/s]
Map:  85%|████████▌ | 34000/39905 [00:04<00:00, 7621.28 examples/s]
Map:  88%|████████▊ | 35000/39905 [00:04<00:00, 8090.19 examples/s]
Map:  90%|█████████ | 36000/39905 [00:04<00:00, 8475.80 examples/s]
Map:  93%|█████████▎| 37000/39905 [00:04<00:00, 8705.30 examples/s]
Map:  95%|█████████▌| 38000/39905 [00:04<00:00, 8934.10 examples/s]
Map:  98%|█████████▊| 39000/39905 [00:04<00:00, 9103.01 examples/s]
Map: 100%|██████████| 39905/39905 [00:04<00:00, 8393.35 examples/s]

Map:   0%|          | 0/10042 [00:00<?, ? examples/s]
Map:  10%|▉         | 1000/10042 [00:00<00:00, 9065.59 examples/s]
Map:  21%|██        | 2078/10042 [00:00<00:00, 10030.81 examples/s]
Map:  32%|███▏      | 3175/10042 [00:00<00:00, 10446.18 examples/s]
Map:  47%|████▋     | 4708/10042 [00:00<00:00, 10328.88 examples/s]
Map:  60%|█████▉    | 6000/10042 [00:00<00:00, 5318.52 examples/s]
Map:  70%|██████▉   | 7000/10042 [00:01<00:00, 6100.03 examples/s]
Map:  80%|███████▉  | 8000/10042 [00:01<00:00, 6819.77 examples/s]
Map:  90%|████████▉ | 9000/10042 [00:01<00:00, 7442.93 examples/s]
Map: 100%|█████████▉| 10000/10042 [00:01<00:00, 7978.02 examples/s]
Map: 100%|██████████| 10042/10042 [00:01<00:00, 7525.67 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 39.0MB/s]

Downloading data:   0%|          | 0.00/331k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.54MB/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.53MB/s]

Downloading data:   0%|          | 0.00/346k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.08MB/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.07MB/s]

Downloading data:   0%|          | 0.00/86.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 86.1k/86.1k [00:00<00:00, 1.07MB/s]

Generating train split:   0%|          | 0/2251 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 2251/2251 [00:00<00:00, 365675.60 examples/s]

Generating test split:   0%|          | 0/2376 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 2376/2376 [00:00<00:00, 514356.97 examples/s]

Generating validation split:   0%|          | 0/570 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 570/570 [00:00<00:00, 293632.19 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 33.3MB/s]

Downloading data:   0%|          | 0.00/190k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 190k/190k [00:00<00:00, 2.09MB/s]

Downloading data:   0%|          | 0.00/204k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 204k/204k [00:00<00:00, 2.31MB/s]

Downloading data:   0%|          | 0.00/55.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 55.7k/55.7k [00:00<00:00, 661kB/s]

Generating train split:   0%|          | 0/1119 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 1119/1119 [00:00<00:00, 252307.61 examples/s]

Generating test split:   0%|          | 0/1172 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1172/1172 [00:00<00:00, 347499.24 examples/s]

Generating validation split:   0%|          | 0/299 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 299/299 [00:00<00:00, 171488.70 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 11.2kB [00:00, 8.07MB/s]

Downloading data:   0%|          | 0.00/2.06M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.3MB/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.1MB/s]

Downloading data:   0%|          | 0.00/118k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]

Downloading data:   0%|          | 0.00/85.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 1.19MB/s]

Generating train split:   0%|          | 0/40398 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 1264347.22 examples/s]

Generating test split:   0%|          | 0/1767 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 644071.88 examples/s]

Generating validation split:   0%|          | 0/1267 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 621832.81 examples/s]

Downloading data:   0%|          | 0.00/2.64M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 21.0MB/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 20.7MB/s]

Downloading data:   0%|          | 0.00/300k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.76MB/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.74MB/s]

Downloading data:   0%|          | 0.00/496k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.14MB/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.12MB/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 16113 examples [00:00, 871946.38 examples/s]

Generating validation split: 0 examples [00:00, ? examples/s]
Generating validation split: 1838 examples [00:00, 654203.22 examples/s]

Generating test split: 0 examples [00:00, ? examples/s]
Generating test split: 3084 examples [00:00, 676422.82 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 110MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 43.0MB/s]

Downloading data:   0%|          | 0.00/18.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 268kB/s]

Downloading data:   0%|          | 0.00/4.94k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 70.0kB/s]

Downloading data:   0%|          | 0.00/3.30k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 46.0kB/s]

Generating test split:   0%|          | 0/171 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 171/171 [00:00<00:00, 73758.33 examples/s]

Generating validation split:   0%|          | 0/19 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 13837.78 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4454.44 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 109MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 294MB/s]

Downloading data:   0%|          | 0.00/1.04M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 14.5MB/s]

Downloading data:   0%|          | 0.00/116k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 1.59MB/s]

Downloading data:   0%|          | 0.00/15.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 215kB/s]

Generating test split:   0%|          | 0/1534 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 190368.14 examples/s]

Generating validation split:   0%|          | 0/170 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 86933.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3685.03 examples/s]

Downloading data:   0%|          | 0.00/54.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 801kB/s]

Downloading data:   0%|          | 0.00/9.89k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 145kB/s]

Downloading data:   0%|          | 0.00/4.62k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 67.2kB/s]

Generating test split:   0%|          | 0/324 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 324/324 [00:00<00:00, 148535.85 examples/s]

Generating validation split:   0%|          | 0/35 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 28750.61 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3903.13 examples/s]

Downloading data:   0%|          | 0.00/48.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 48.6k/48.6k [00:00<00:00, 724kB/s]

Downloading data:   0%|          | 0.00/9.15k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.15k/9.15k [00:00<00:00, 144kB/s]

Downloading data:   0%|          | 0.00/4.20k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.20k/4.20k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/311 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 311/311 [00:00<00:00, 123795.06 examples/s]

Generating validation split:   0%|          | 0/34 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 34/34 [00:00<00:00, 26988.33 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3915.52 examples/s]

Downloading data:   0%|          | 0.00/89.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 1.32MB/s]

Downloading data:   0%|          | 0.00/14.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 216kB/s]

Downloading data:   0%|          | 0.00/5.14k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 74.2kB/s]

Generating test split:   0%|          | 0/895 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 895/895 [00:00<00:00, 307596.04 examples/s]

Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 67356.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4344.63 examples/s]

Downloading data:   0%|          | 0.00/60.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 687kB/s]

Downloading data:   0%|          | 0.00/10.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 160kB/s]

Downloading data:   0%|          | 0.00/4.41k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 60.8kB/s]

Generating test split:   0%|          | 0/346 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 346/346 [00:00<00:00, 122116.22 examples/s]

Generating validation split:   0%|          | 0/38 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 24404.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4163.49 examples/s]

Downloading data:   0%|          | 0.00/23.0k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.0k/23.0k [00:00<00:00, 320kB/s]

Downloading data:   0%|          | 0.00/6.52k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.52k/6.52k [00:00<00:00, 85.0kB/s]

Downloading data:   0%|          | 0.00/4.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.12k/4.12k [00:00<00:00, 58.1kB/s]

Generating test split:   0%|          | 0/163 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 163/163 [00:00<00:00, 73096.50 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 12130.06 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4532.42 examples/s]

Downloading data:   0%|          | 0.00/23.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.3k/23.3k [00:00<00:00, 338kB/s]

Downloading data:   0%|          | 0.00/6.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.21k/6.21k [00:00<00:00, 98.4kB/s]

Downloading data:   0%|          | 0.00/4.05k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.05k/4.05k [00:00<00:00, 52.8kB/s]

Generating test split:   0%|          | 0/108 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 108/108 [00:00<00:00, 38008.46 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 7080.62 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3219.95 examples/s]

Downloading data:   0%|          | 0.00/29.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 29.5k/29.5k [00:00<00:00, 411kB/s]

Downloading data:   0%|          | 0.00/7.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.12k/7.12k [00:00<00:00, 106kB/s]

Downloading data:   0%|          | 0.00/4.96k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.96k/4.96k [00:00<00:00, 68.0kB/s]

Generating test split:   0%|          | 0/121 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 121/121 [00:00<00:00, 51113.99 examples/s]

Generating validation split:   0%|          | 0/13 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 13/13 [00:00<00:00, 10030.53 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3499.34 examples/s]

Downloading data:   0%|          | 0.00/202k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 202k/202k [00:00<00:00, 2.94MB/s]

Downloading data:   0%|          | 0.00/38.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 38.5k/38.5k [00:00<00:00, 521kB/s]

Downloading data:   0%|          | 0.00/10.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.2k/10.2k [00:00<00:00, 150kB/s]

Generating test split:   0%|          | 0/237 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 237/237 [00:00<00:00, 73285.91 examples/s]

Generating validation split:   0%|          | 0/26 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 18464.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3556.91 examples/s]

Downloading data:   0%|          | 0.00/155k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 155k/155k [00:00<00:00, 2.25MB/s]

Downloading data:   0%|          | 0.00/27.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 419kB/s]

Downloading data:   0%|          | 0.00/17.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 17.8k/17.8k [00:00<00:00, 258kB/s]

Generating test split:   0%|          | 0/204 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 204/204 [00:00<00:00, 79720.30 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 13615.86 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4526.55 examples/s]

Downloading data:   0%|          | 0.00/142k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 142k/142k [00:00<00:00, 1.96MB/s]

Downloading data:   0%|          | 0.00/31.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 31.6k/31.6k [00:00<00:00, 464kB/s]

Downloading data:   0%|          | 0.00/22.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.2k/22.2k [00:00<00:00, 311kB/s]

Generating test split:   0%|          | 0/165 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 165/165 [00:00<00:00, 63919.84 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 11865.07 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4293.04 examples/s]

Downloading data:   0%|          | 0.00/21.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 21.5k/21.5k [00:00<00:00, 318kB/s]

Downloading data:   0%|          | 0.00/6.56k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.56k/6.56k [00:00<00:00, 87.1kB/s]

Downloading data:   0%|          | 0.00/4.81k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 41.0kB/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 40.9kB/s]

Generating test split:   0%|          | 0/126 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 58350.70 examples/s]

Generating validation split:   0%|          | 0/14 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 10374.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4641.77 examples/s]

Downloading data:   0%|          | 0.00/19.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 292kB/s]

Downloading data:   0%|          | 0.00/5.27k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 74.2kB/s]

Downloading data:   0%|          | 0.00/4.22k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 60.5kB/s]

Generating test split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 49061.93 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 9145.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3454.38 examples/s]

Downloading data:   0%|          | 0.00/43.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 658kB/s]

Downloading data:   0%|          | 0.00/8.36k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 118kB/s]

Downloading data:   0%|          | 0.00/4.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 66.6kB/s]

Generating test split:   0%|          | 0/201 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 201/201 [00:00<00:00, 96140.39 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 18443.87 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3890.10 examples/s]

Downloading data:   0%|          | 0.00/114k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 1.65MB/s]

Downloading data:   0%|          | 0.00/18.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 285kB/s]

Downloading data:   0%|          | 0.00/7.49k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 106kB/s]

Generating test split:   0%|          | 0/245 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 245/245 [00:00<00:00, 98998.50 examples/s]

Generating validation split:   0%|          | 0/27 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 21278.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3775.93 examples/s]

Downloading data:   0%|          | 0.00/20.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 297kB/s]

Downloading data:   0%|          | 0.00/6.45k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 93.1kB/s]

Downloading data:   0%|          | 0.00/4.43k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 63.4kB/s]

Generating test split:   0%|          | 0/110 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 110/110 [00:00<00:00, 59089.84 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 10082.46 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4160.19 examples/s]

Downloading data:   0%|          | 0.00/133k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 1.83MB/s]

Downloading data:   0%|          | 0.00/22.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/4.69k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 70.8kB/s]

Generating test split:   0%|          | 0/612 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 612/612 [00:00<00:00, 245778.82 examples/s]

Generating validation split:   0%|          | 0/69 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 52042.25 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4654.13 examples/s]

Downloading data:   0%|          | 0.00/23.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.2k/23.2k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/5.26k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.26k/5.26k [00:00<00:00, 83.7kB/s]

Downloading data:   0%|          | 0.00/4.08k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.08k/4.08k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/131 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 131/131 [00:00<00:00, 65147.48 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 8817.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4652.07 examples/s]

Downloading data:


The task is still running (10 min timeout hit). The generate_until step for GSM8K is the bottleneck. Let me keep waiting.


Run full benchmark suite for Qwen3-0.6B on A100


Run full benchmark suite for Qwen3-0.6B on A100

Output:
============================================
  Qwen3-0.6B Full Accuracy Benchmark
  GPU: NVIDIA A100-SXM4-80GB
  dtype: bfloat16 (full precision, no quant)
============================================
2026-03-21:10:35:20 INFO     [_cli.run:376] Selected Tasks: ['hellaswag', 'arc_easy', 'arc_challenge', 'winogrande', 'piqa', 'mmlu', 'truthfulqa_mc2', 'gsm8k']
2026-03-21:10:35:21 INFO     [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2026-03-21:10:35:21 INFO     [evaluator:236] Initializing hf model, with arguments: {'pretrained': 'Qwen/Qwen3-0.6B', 'dtype': 'bfloat16', 'trust_remote_code': True}
2026-03-21:10:35:23 INFO     [models.huggingface:161] Using device 'cuda:0'
2026-03-21:10:35:24 INFO     [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 7.02kB [00:00, 27.1MB/s]

Downloading data:   0%|          | 0.00/24.4M [00:00<?, ?B/s]
Downloading data:  43%|████▎     | 10.5M/24.4M [00:00<00:00, 46.6MB/s]
Downloading data: 100%|██████████| 24.4M/24.4M [00:00<00:00, 85.9MB/s]

Downloading data:   0%|          | 0.00/6.11M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.11M/6.11M [00:00<00:00, 68.7MB/s]

Downloading data:   0%|          | 0.00/6.32M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.32M/6.32M [00:00<00:00, 64.5MB/s]

Generating train split:   0%|          | 0/39905 [00:00<?, ? examples/s]
Generating train split:  70%|███████   | 28000/39905 [00:00<00:00, 266674.49 examples/s]
Generating train split: 100%|██████████| 39905/39905 [00:00<00:00, 226653.28 examples/s]

Generating test split:   0%|          | 0/10003 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 10003/10003 [00:00<00:00, 226416.32 examples/s]

Generating validation split:   0%|          | 0/10042 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 10042/10042 [00:00<00:00, 216312.05 examples/s]

Map:   0%|          | 0/39905 [00:00<?, ? examples/s]
Map:   3%|▎         | 1000/39905 [00:00<00:13, 2934.46 examples/s]
Map:   5%|▌         | 2032/39905 [00:00<00:07, 5109.09 examples/s]
Map:   8%|▊         | 3125/39905 [00:00<00:05, 6833.81 examples/s]
Map:  11%|█         | 4228/39905 [00:00<00:04, 8081.15 examples/s]
Map:  13%|█▎        | 5326/39905 [00:00<00:03, 8943.96 examples/s]
Map:  16%|█▌        | 6420/39905 [00:00<00:03, 9537.21 examples/s]
Map:  19%|█▉        | 7646/39905 [00:00<00:03, 10051.79 examples/s]
Map:  22%|██▏       | 8755/39905 [00:01<00:03, 10352.90 examples/s]
Map:  25%|██▍       | 9857/39905 [00:01<00:02, 10546.45 examples/s]
Map:  27%|██▋       | 10947/39905 [00:01<00:02, 10649.55 examples/s]
Map:  32%|███▏      | 12651/39905 [00:01<00:02, 10572.07 examples/s]
Map:  34%|███▍      | 13732/39905 [00:01<00:02, 10632.99 examples/s]
Map:  37%|███▋      | 14819/39905 [00:01<00:02, 10693.86 examples/s]
Map:  40%|████      | 16000/39905 [00:01<00:03, 6572.81 examples/s]
Map:  43%|████▎     | 17000/39905 [00:02<00:03, 7124.51 examples/s]
Map:  45%|████▌     | 18000/39905 [00:02<00:02, 7647.02 examples/s]
Map:  48%|████▊     | 19000/39905 [00:02<00:02, 8013.93 examples/s]
Map:  50%|█████     | 20000/39905 [00:02<00:02, 8410.05 examples/s]
Map:  53%|█████▎    | 21000/39905 [00:02<00:02, 8716.28 examples/s]
Map:  55%|█████▌    | 22000/39905 [00:02<00:02, 8941.50 examples/s]
Map:  58%|█████▊    | 23000/39905 [00:02<00:01, 8952.72 examples/s]
Map:  60%|██████    | 24000/39905 [00:02<00:01, 9107.57 examples/s]
Map:  63%|██████▎   | 25000/39905 [00:02<00:01, 9226.00 examples/s]
Map:  65%|██████▌   | 26000/39905 [00:03<00:01, 9319.35 examples/s]
Map:  68%|██████▊   | 27000/39905 [00:03<00:01, 9351.83 examples/s]
Map:  70%|███████   | 28000/39905 [00:03<00:01, 9396.54 examples/s]
Map:  73%|███████▎  | 29000/39905 [00:03<00:01, 9432.13 examples/s]
Map:  75%|███████▌  | 30000/39905 [00:03<00:01, 9482.60 examples/s]
Map:  78%|███████▊  | 31029/39905 [00:03<00:01, 5608.07 examples/s]
Map:  80%|████████  | 32000/39905 [00:03<00:01, 6374.99 examples/s]
Map:  83%|████████▎ | 33000/39905 [00:04<00:00, 7012.16 examples/s]
Map:  85%|████████▌ | 34000/39905 [00:04<00:00, 7621.28 examples/s]
Map:  88%|████████▊ | 35000/39905 [00:04<00:00, 8090.19 examples/s]
Map:  90%|█████████ | 36000/39905 [00:04<00:00, 8475.80 examples/s]
Map:  93%|█████████▎| 37000/39905 [00:04<00:00, 8705.30 examples/s]
Map:  95%|█████████▌| 38000/39905 [00:04<00:00, 8934.10 examples/s]
Map:  98%|█████████▊| 39000/39905 [00:04<00:00, 9103.01 examples/s]
Map: 100%|██████████| 39905/39905 [00:04<00:00, 8393.35 examples/s]

Map:   0%|          | 0/10042 [00:00<?, ? examples/s]
Map:  10%|▉         | 1000/10042 [00:00<00:00, 9065.59 examples/s]
Map:  21%|██        | 2078/10042 [00:00<00:00, 10030.81 examples/s]
Map:  32%|███▏      | 3175/10042 [00:00<00:00, 10446.18 examples/s]
Map:  47%|████▋     | 4708/10042 [00:00<00:00, 10328.88 examples/s]
Map:  60%|█████▉    | 6000/10042 [00:00<00:00, 5318.52 examples/s]
Map:  70%|██████▉   | 7000/10042 [00:01<00:00, 6100.03 examples/s]
Map:  80%|███████▉  | 8000/10042 [00:01<00:00, 6819.77 examples/s]
Map:  90%|████████▉ | 9000/10042 [00:01<00:00, 7442.93 examples/s]
Map: 100%|█████████▉| 10000/10042 [00:01<00:00, 7978.02 examples/s]
Map: 100%|██████████| 10042/10042 [00:01<00:00, 7525.67 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 39.0MB/s]

Downloading data:   0%|          | 0.00/331k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.54MB/s]
Downloading data: 100%|██████████| 331k/331k [00:00<00:00, 2.53MB/s]

Downloading data:   0%|          | 0.00/346k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.08MB/s]
Downloading data: 100%|██████████| 346k/346k [00:00<00:00, 3.07MB/s]

Downloading data:   0%|          | 0.00/86.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 86.1k/86.1k [00:00<00:00, 1.07MB/s]

Generating train split:   0%|          | 0/2251 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 2251/2251 [00:00<00:00, 365675.60 examples/s]

Generating test split:   0%|          | 0/2376 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 2376/2376 [00:00<00:00, 514356.97 examples/s]

Generating validation split:   0%|          | 0/570 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 570/570 [00:00<00:00, 293632.19 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 9.00kB [00:00, 33.3MB/s]

Downloading data:   0%|          | 0.00/190k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 190k/190k [00:00<00:00, 2.09MB/s]

Downloading data:   0%|          | 0.00/204k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 204k/204k [00:00<00:00, 2.31MB/s]

Downloading data:   0%|          | 0.00/55.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 55.7k/55.7k [00:00<00:00, 661kB/s]

Generating train split:   0%|          | 0/1119 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 1119/1119 [00:00<00:00, 252307.61 examples/s]

Generating test split:   0%|          | 0/1172 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1172/1172 [00:00<00:00, 347499.24 examples/s]

Generating validation split:   0%|          | 0/299 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 299/299 [00:00<00:00, 171488.70 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 11.2kB [00:00, 8.07MB/s]

Downloading data:   0%|          | 0.00/2.06M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.3MB/s]
Downloading data: 100%|██████████| 2.06M/2.06M [00:00<00:00, 17.1MB/s]

Downloading data:   0%|          | 0.00/118k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]
Downloading data: 100%|██████████| 118k/118k [00:00<00:00, 1.15MB/s]

Downloading data:   0%|          | 0.00/85.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 85.9k/85.9k [00:00<00:00, 1.19MB/s]

Generating train split:   0%|          | 0/40398 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 40398/40398 [00:00<00:00, 1264347.22 examples/s]

Generating test split:   0%|          | 0/1767 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1767/1767 [00:00<00:00, 644071.88 examples/s]

Generating validation split:   0%|          | 0/1267 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 1267/1267 [00:00<00:00, 621832.81 examples/s]

Downloading data:   0%|          | 0.00/2.64M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 21.0MB/s]
Downloading data: 100%|██████████| 2.64M/2.64M [00:00<00:00, 20.7MB/s]

Downloading data:   0%|          | 0.00/300k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.76MB/s]
Downloading data: 100%|██████████| 300k/300k [00:00<00:00, 2.74MB/s]

Downloading data:   0%|          | 0.00/496k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.14MB/s]
Downloading data: 100%|██████████| 496k/496k [00:00<00:00, 4.12MB/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 16113 examples [00:00, 871946.38 examples/s]

Generating validation split: 0 examples [00:00, ? examples/s]
Generating validation split: 1838 examples [00:00, 654203.22 examples/s]

Generating test split: 0 examples [00:00, ? examples/s]
Generating test split: 3084 examples [00:00, 676422.82 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 110MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 43.0MB/s]

Downloading data:   0%|          | 0.00/18.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.9k/18.9k [00:00<00:00, 268kB/s]

Downloading data:   0%|          | 0.00/4.94k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.94k/4.94k [00:00<00:00, 70.0kB/s]

Downloading data:   0%|          | 0.00/3.30k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 3.30k/3.30k [00:00<00:00, 46.0kB/s]

Generating test split:   0%|          | 0/171 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 171/171 [00:00<00:00, 73758.33 examples/s]

Generating validation split:   0%|          | 0/19 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 19/19 [00:00<00:00, 13837.78 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4454.44 examples/s]

Downloading readme: 0.00B [00:00, ?B/s]
Downloading readme: 53.2kB [00:00, 109MB/s]

Downloading metadata: 0.00B [00:00, ?B/s]
Downloading metadata: 138kB [00:00, 294MB/s]

Downloading data:   0%|          | 0.00/1.04M [00:00<?, ?B/s]
Downloading data: 100%|██████████| 1.04M/1.04M [00:00<00:00, 14.5MB/s]

Downloading data:   0%|          | 0.00/116k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 116k/116k [00:00<00:00, 1.59MB/s]

Downloading data:   0%|          | 0.00/15.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 15.1k/15.1k [00:00<00:00, 215kB/s]

Generating test split:   0%|          | 0/1534 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 1534/1534 [00:00<00:00, 190368.14 examples/s]

Generating validation split:   0%|          | 0/170 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 170/170 [00:00<00:00, 86933.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3685.03 examples/s]

Downloading data:   0%|          | 0.00/54.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 54.3k/54.3k [00:00<00:00, 801kB/s]

Downloading data:   0%|          | 0.00/9.89k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.89k/9.89k [00:00<00:00, 145kB/s]

Downloading data:   0%|          | 0.00/4.62k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.62k/4.62k [00:00<00:00, 67.2kB/s]

Generating test split:   0%|          | 0/324 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 324/324 [00:00<00:00, 148535.85 examples/s]

Generating validation split:   0%|          | 0/35 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 35/35 [00:00<00:00, 28750.61 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3903.13 examples/s]

Downloading data:   0%|          | 0.00/48.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 48.6k/48.6k [00:00<00:00, 724kB/s]

Downloading data:   0%|          | 0.00/9.15k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 9.15k/9.15k [00:00<00:00, 144kB/s]

Downloading data:   0%|          | 0.00/4.20k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.20k/4.20k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/311 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 311/311 [00:00<00:00, 123795.06 examples/s]

Generating validation split:   0%|          | 0/34 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 34/34 [00:00<00:00, 26988.33 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3915.52 examples/s]

Downloading data:   0%|          | 0.00/89.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 89.8k/89.8k [00:00<00:00, 1.32MB/s]

Downloading data:   0%|          | 0.00/14.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 14.9k/14.9k [00:00<00:00, 216kB/s]

Downloading data:   0%|          | 0.00/5.14k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.14k/5.14k [00:00<00:00, 74.2kB/s]

Generating test split:   0%|          | 0/895 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 895/895 [00:00<00:00, 307596.04 examples/s]

Generating validation split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 100/100 [00:00<00:00, 67356.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4344.63 examples/s]

Downloading data:   0%|          | 0.00/60.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 60.9k/60.9k [00:00<00:00, 687kB/s]

Downloading data:   0%|          | 0.00/10.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.7k/10.7k [00:00<00:00, 160kB/s]

Downloading data:   0%|          | 0.00/4.41k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.41k/4.41k [00:00<00:00, 60.8kB/s]

Generating test split:   0%|          | 0/346 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 346/346 [00:00<00:00, 122116.22 examples/s]

Generating validation split:   0%|          | 0/38 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 38/38 [00:00<00:00, 24404.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4163.49 examples/s]

Downloading data:   0%|          | 0.00/23.0k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.0k/23.0k [00:00<00:00, 320kB/s]

Downloading data:   0%|          | 0.00/6.52k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.52k/6.52k [00:00<00:00, 85.0kB/s]

Downloading data:   0%|          | 0.00/4.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.12k/4.12k [00:00<00:00, 58.1kB/s]

Generating test split:   0%|          | 0/163 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 163/163 [00:00<00:00, 73096.50 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 12130.06 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4532.42 examples/s]

Downloading data:   0%|          | 0.00/23.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.3k/23.3k [00:00<00:00, 338kB/s]

Downloading data:   0%|          | 0.00/6.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.21k/6.21k [00:00<00:00, 98.4kB/s]

Downloading data:   0%|          | 0.00/4.05k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.05k/4.05k [00:00<00:00, 52.8kB/s]

Generating test split:   0%|          | 0/108 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 108/108 [00:00<00:00, 38008.46 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 7080.62 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3219.95 examples/s]

Downloading data:   0%|          | 0.00/29.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 29.5k/29.5k [00:00<00:00, 411kB/s]

Downloading data:   0%|          | 0.00/7.12k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.12k/7.12k [00:00<00:00, 106kB/s]

Downloading data:   0%|          | 0.00/4.96k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.96k/4.96k [00:00<00:00, 68.0kB/s]

Generating test split:   0%|          | 0/121 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 121/121 [00:00<00:00, 51113.99 examples/s]

Generating validation split:   0%|          | 0/13 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 13/13 [00:00<00:00, 10030.53 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3499.34 examples/s]

Downloading data:   0%|          | 0.00/202k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 202k/202k [00:00<00:00, 2.94MB/s]

Downloading data:   0%|          | 0.00/38.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 38.5k/38.5k [00:00<00:00, 521kB/s]

Downloading data:   0%|          | 0.00/10.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 10.2k/10.2k [00:00<00:00, 150kB/s]

Generating test split:   0%|          | 0/237 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 237/237 [00:00<00:00, 73285.91 examples/s]

Generating validation split:   0%|          | 0/26 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 26/26 [00:00<00:00, 18464.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3556.91 examples/s]

Downloading data:   0%|          | 0.00/155k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 155k/155k [00:00<00:00, 2.25MB/s]

Downloading data:   0%|          | 0.00/27.3k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 27.3k/27.3k [00:00<00:00, 419kB/s]

Downloading data:   0%|          | 0.00/17.8k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 17.8k/17.8k [00:00<00:00, 258kB/s]

Generating test split:   0%|          | 0/204 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 204/204 [00:00<00:00, 79720.30 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 13615.86 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4526.55 examples/s]

Downloading data:   0%|          | 0.00/142k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 142k/142k [00:00<00:00, 1.96MB/s]

Downloading data:   0%|          | 0.00/31.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 31.6k/31.6k [00:00<00:00, 464kB/s]

Downloading data:   0%|          | 0.00/22.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.2k/22.2k [00:00<00:00, 311kB/s]

Generating test split:   0%|          | 0/165 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 165/165 [00:00<00:00, 63919.84 examples/s]

Generating validation split:   0%|          | 0/18 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 18/18 [00:00<00:00, 11865.07 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4293.04 examples/s]

Downloading data:   0%|          | 0.00/21.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 21.5k/21.5k [00:00<00:00, 318kB/s]

Downloading data:   0%|          | 0.00/6.56k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.56k/6.56k [00:00<00:00, 87.1kB/s]

Downloading data:   0%|          | 0.00/4.81k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 41.0kB/s]
Downloading data: 100%|██████████| 4.81k/4.81k [00:00<00:00, 40.9kB/s]

Generating test split:   0%|          | 0/126 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 126/126 [00:00<00:00, 58350.70 examples/s]

Generating validation split:   0%|          | 0/14 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 14/14 [00:00<00:00, 10374.60 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4641.77 examples/s]

Downloading data:   0%|          | 0.00/19.5k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 19.5k/19.5k [00:00<00:00, 292kB/s]

Downloading data:   0%|          | 0.00/5.27k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.27k/5.27k [00:00<00:00, 74.2kB/s]

Downloading data:   0%|          | 0.00/4.22k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.22k/4.22k [00:00<00:00, 60.5kB/s]

Generating test split:   0%|          | 0/100 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 100/100 [00:00<00:00, 49061.93 examples/s]

Generating validation split:   0%|          | 0/11 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 11/11 [00:00<00:00, 9145.16 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3454.38 examples/s]

Downloading data:   0%|          | 0.00/43.9k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 43.9k/43.9k [00:00<00:00, 658kB/s]

Downloading data:   0%|          | 0.00/8.36k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 8.36k/8.36k [00:00<00:00, 118kB/s]

Downloading data:   0%|          | 0.00/4.21k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.21k/4.21k [00:00<00:00, 66.6kB/s]

Generating test split:   0%|          | 0/201 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 201/201 [00:00<00:00, 96140.39 examples/s]

Generating validation split:   0%|          | 0/22 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 22/22 [00:00<00:00, 18443.87 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3890.10 examples/s]

Downloading data:   0%|          | 0.00/114k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 114k/114k [00:00<00:00, 1.65MB/s]

Downloading data:   0%|          | 0.00/18.7k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 18.7k/18.7k [00:00<00:00, 285kB/s]

Downloading data:   0%|          | 0.00/7.49k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 7.49k/7.49k [00:00<00:00, 106kB/s]

Generating test split:   0%|          | 0/245 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 245/245 [00:00<00:00, 98998.50 examples/s]

Generating validation split:   0%|          | 0/27 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 27/27 [00:00<00:00, 21278.88 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 3775.93 examples/s]

Downloading data:   0%|          | 0.00/20.6k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 20.6k/20.6k [00:00<00:00, 297kB/s]

Downloading data:   0%|          | 0.00/6.45k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 6.45k/6.45k [00:00<00:00, 93.1kB/s]

Downloading data:   0%|          | 0.00/4.43k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.43k/4.43k [00:00<00:00, 63.4kB/s]

Generating test split:   0%|          | 0/110 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 110/110 [00:00<00:00, 59089.84 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 10082.46 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4160.19 examples/s]

Downloading data:   0%|          | 0.00/133k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 133k/133k [00:00<00:00, 1.83MB/s]

Downloading data:   0%|          | 0.00/22.1k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 22.1k/22.1k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/4.69k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.69k/4.69k [00:00<00:00, 70.8kB/s]

Generating test split:   0%|          | 0/612 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 612/612 [00:00<00:00, 245778.82 examples/s]

Generating validation split:   0%|          | 0/69 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 69/69 [00:00<00:00, 52042.25 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4654.13 examples/s]

Downloading data:   0%|          | 0.00/23.2k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 23.2k/23.2k [00:00<00:00, 313kB/s]

Downloading data:   0%|          | 0.00/5.26k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 5.26k/5.26k [00:00<00:00, 83.7kB/s]

Downloading data:   0%|          | 0.00/4.08k [00:00<?, ?B/s]
Downloading data: 100%|██████████| 4.08k/4.08k [00:00<00:00, 61.2kB/s]

Generating test split:   0%|          | 0/131 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 131/131 [00:00<00:00, 65147.48 examples/s]

Generating validation split:   0%|          | 0/12 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 12/12 [00:00<00:00, 8817.74 examples/s]

Generating dev split:   0%|          | 0/5 [00:00<?, ? examples/s]
Generating dev split: 100%|██████████| 5/5 [00:00<00:00, 4652.07 examples/s]

Downloading data:


Still grinding through. Let me check the latest tail of the actual output file directly.


Bash

tail -10 /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null

Check latest benchmark progress


<persisted-output>
Output too large (39.3KB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu/aa63ed1d-948a-4d98-914e-be56275a729f/tool-results/b80q1yikx.txt

Preview (first 2KB):

  0%|          | 0/1267 [00:00<?, ?it/s]
100%|██████████| 1267/1267 [00:00<00:00, 52850.10it/s]
2026-03-21:10:37:32 INFO     [api.task:311] Building contexts for arc_challenge on rank 0...

  0%|          | 0/1172 [00:00<?, ?it/s]
  6%|▌         | 72/1172 [00:00<00:01, 715.32it/s]
 12%|█▏        | 145/1172 [00:00<00:01, 721.25it/s]
 19%|█▊        | 218/1172 [00:00<00:01, 723.11it/s]
 25%|██▍       | 291/1172 [00:00<00:01, 723.54it/s]
 31%|███       | 364/1172 [00:00<00:01, 716.41it/s]
 37%|███▋      | 437/1172 [00:00<00:01, 718.94it/s]
 44%|████▎     | 510/1172 [00:00<00:00, 720.58it/s]
 50%|████▉     | 583/1172 [00:00<00:00, 721.56it/s]
 56%|█████▌    | 656/1172 [00:00<00:00, 722.41it/s]
 62%|██████▏   | 729/1172 [00:01<00:00, 722.53it/s]
 68%|██████▊   | 802/1172 [00:01<00:00, 722.08it/s]
 75%|███████▍  | 875/1172 [00:01<00:00, 721.95it/s]
 81%|████████  | 948/1172 [00:01<00:00, 721.94it/s]
 87%|████████▋ | 1021/1172 [00:01<00:00, 722.10it/s]
 93%|█████████▎| 1094/1172 [00:01<00:00, 722.16it/s]
100%|█████████▉| 1167/1172 [00:01<00:00, 721.77it/s]
100%|██████████| 1172/1172 [00:01<00:00, 721.29it/s]
2026-03-21:10:37:34 INFO     [api.task:311] Building contexts for arc_easy on rank 0...
...
</persisted-output>


grep -E "generate_until|loglikelihood|Running|results|acc" /tmp/claude-1001/-home-ubuntu/tasks/bc6z1aaiw.output 2>/dev/null | tail -20

Extract benchmark progress lines