  [3584/6000] (60%) processed
  [3712/6000] (62%) processed
  [3840/6000] (64%) processed


Bash

tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [3712/6000] (62%) processed
  [3840/6000] (64%) processed
  [3968/6000] (66%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [3840/6000] (64%) processed
  [3968/6000] (66%) processed
  [4096/6000] (68%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [3968/6000] (66%) processed
  [4096/6000] (68%) processed
  [4224/6000] (70%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4096/6000] (68%) processed
  [4224/6000] (70%) processed
  [4352/6000] (73%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4224/6000] (70%) processed
  [4352/6000] (73%) processed
  [4480/6000] (75%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4352/6000] (73%) processed
  [4480/6000] (75%) processed
  [4608/6000] (77%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4480/6000] (75%) processed
  [4608/6000] (77%) processed
  [4736/6000] (79%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4608/6000] (77%) processed
  [4736/6000] (79%) processed
  [4864/6000] (81%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4736/6000] (79%) processed
  [4864/6000] (81%) processed
  [4992/6000] (83%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4864/6000] (81%) processed
  [4992/6000] (83%) processed
  [5120/6000] (85%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [4992/6000] (83%) processed
  [5120/6000] (85%) processed
  [5248/6000] (87%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [5120/6000] (85%) processed
  [5248/6000] (87%) processed
  [5376/6000] (90%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [5248/6000] (87%) processed
  [5376/6000] (90%) processed
  [5504/6000] (92%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [5376/6000] (90%) processed
  [5504/6000] (92%) processed
  [5632/6000] (94%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [5504/6000] (92%) processed
  [5632/6000] (94%) processed
  [5760/6000] (96%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check progress


  [5632/6000] (94%) processed
  [5760/6000] (96%) processed
  [5888/6000] (98%) processed


<task-notification>
<task-id>bct7cipzt</task-id>
<tool-use-id>toolu_017imT3XaVyqiXW61TUWhzkq</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output</output-file>
<status>completed</status>
<summary>Background command "Run ckpt-20000 benchmark" completed (exit code 0)</summary>
</task-notification>


tail -20 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bct7cipzt.output 2>&1

Check for final results


Language           N     WER%     CER%   WER-n%   CER-n%
-------------- ----- -------- -------- -------- --------
assamese         500    21.51     9.02    18.11     8.64
bengali          500    24.00     6.14    16.02     4.42
english          500    30.65    18.91    18.95    16.08
gujarati         500    20.97     7.25    16.36     6.37
hindi            500    14.24     4.49     9.93     3.52
kannada          500    73.51    48.00    70.80    48.51
malayalam        500    42.17     8.26    37.22     7.47
marathi          500    26.28     7.57    21.14     6.75
odia             500    26.65     6.90    20.69     5.89
punjabi          500    27.28    13.74    21.29    11.80
tamil            500    39.14    14.75    36.59    13.98
telugu           500    27.59     5.78    22.26     5.21
-------------- ----- -------- -------- -------- --------
OVERALL         6000    30.76    13.05    25.45    12.21
MACRO-AVG         12    31.17    12.57    25.78    11.55
=====================================================================================

Sample analysis saved: /home/ubuntu/training/benchmark_results_gemma3n/ckpt-20000/sample_analysis.json


**ckpt-20000 benchmark done!** Now let me generate its dashboard JSON and show the comparison.


python3 generate_schema_v1.py --src benchmark_results_gemma3n/ckpt-20000 --ckpt ckpt-20000 2>&1

Generate dashboard JSON for ckpt-20000


============================================================
  ckpt-20000
============================================================
  Loaded 6000 predictions
  Overall: wer_raw=30.43 wer_norm=29.39 wer_nc=29.2 snwer=20.94 mer=13.21 cer_norm=12.73
  Diagnosis: recognition-limited


ls -lh /home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-*/ 2>&1

Verify all dashboard files


/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-20000/:
total 12M
-rw-rw-r-- 1 ubuntu ubuntu  64K Mar 29 08:06 error_analysis.json
-rw-rw-r-- 1 ubuntu ubuntu 5.2K Mar 29 08:06 metrics.json
-rw-rw-r-- 1 ubuntu ubuntu  11M Mar 29 08:06 sample_analysis.json

/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-40000/:
total 12M
-rw-rw-r-- 1 ubuntu ubuntu  64K Mar 29 07:53 error_analysis.json
-rw-rw-r-- 1 ubuntu ubuntu 5.2K Mar 29 07:53 metrics.json
-rw-rw-r-- 1 ubuntu ubuntu  11M Mar 29 07:53 sample_analysis.json


All done. Here's the full comparison of the new checkpoints (from the new training data):

### Gemma3n-E2B-ASR: New Training Data (03-28-2026)

| Language | ckpt-20000 WER-n% | ckpt-40000 WER-n% | Change |
|----------|-------------------|-------------------|--------|
| Hindi | 9.93 | **8.90** | -1.03 |
| Bengali | 16.02 | **16.83** | +0.81 |
| Gujarati | 16.36 | **15.39** | -0.97 |
| English | 18.95 | **17.33** | -1.62 |
| Assamese | 18.11 | **18.60** | +0.49 |
| Odia | 20.69 | **22.79** | +2.10 |
| Marathi | 21.14 | **20.96** | -0.18 |
| Punjabi | 21.29 | **17.17** | -4.12 |
| Telugu | 22.26 | **22.11** | -0.15 |
| Tamil | 36.59 | **33.12** | -3.47 |
| Malayalam | 37.22 | **37.61** | +0.39 |
| Kannada | 70.80 | **40.26** | **-30.54** |
| **OVERALL** | **25.45** | **22.35** | **-3.10** |

**Dashboard schema v1 (6 metric tiers):**

| Metric | ckpt-20000 | ckpt-40000 |
|--------|-----------|-----------|
| wer_raw | 30.43 | 27.01 |
| **wer_norm** | **29.39** | **26.14** |
| wer_numcanon | 29.20 | 26.10 |
| space_norm_wer | 20.94 | 19.15 |
| mer | 13.21 | 10.61 |
| cer_norm | 12.73 | 10.21 |

**Key highlights:**
- Massive Kannada improvement: 70.8% → 40.3% (30 point drop!)
- Overall WER_norm: 29.4% → 26.1% — strong improvement from new training data
- Hindi continues to be best at 8.9% WER_norm
- Both checkpoints: `recognition-limited` diagnosis — normalization/formatting not the main issue


<task-notification>
<task-id>bcy99cwa3</task-id>
<tool-use-id>toolu_012r4DRTYktE4FA5pexBWKom</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bcy99cwa3.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-20000 model files" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>bosyl0g8u</task-id>
<tool-use-id>toolu_01Nqnv52vG1yQ8BWc12WTeBo</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bosyl0g8u.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-40000 model files" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>bjyyto5zb</task-id>
<tool-use-id>toolu_0123bcXw57FtMQRpTeF9t3em</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bjyyto5zb.output</output-file>
<status>completed</status>
<summary>Background command "Re-download ckpt-20000 model.safetensors" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>blh4bku6v</task-id>
<tool-use-id>toolu_01GdnVoPWDe8dwWMg2SAu5BA</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/blh4bku6v.output</output-file>
<status>completed</status>
<summary>Background command "Run ckpt-40000 benchmark (BS=128, full GPU)" completed (exit code 0)</summary>
</task-notification>


All background tasks completed successfully. Everything's done:

- Both downloads (ckpt-20000, ckpt-40000) completed
- Both benchmarks (6000 samples each, BS=128) completed
- Dashboard JSONs generated for both with full schema v1 (6 metric tiers + error_analysis)

Files are at:
```
/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/
  ckpt-20000/  metrics.json, sample_analysis.json, error_analysis.json
  ckpt-40000/  metrics.json, sample_analysis.json, error_analysis.json
```


so what happened is, i made a few changes with the training, and just was training with different data. so there are 2 new ckpts of gemma3n. they're 20k and 40k again. can you first cleanup the local,…


55k ckpt is in R2. mind running bench throgh it.


Please run /login · API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"OAuth token has expired. Please obtain a new token or refresh your existing token."},"request_id":"req_011CZYVkbjgBVy6CArUpHccE"}


<command-message>loop</command-message>
<command-name>/loop</command-name>


Usage: /loop [interval] <prompt>

Run a prompt or slash command on a recurring interval.

Intervals: Ns, Nm, Nh, Nd (e.g. 5m, 30m, 2h, 1d). Minimum granularity is 1 minute.
If no interval is specified, defaults to 10m.

Examples:
  /loop 5m /babysit-prs
  /loop 30m check the deploy
  /loop 1h /standup 1
  /loop check the deploy          (defaults to 10m)
  /loop check the deploy every 20m


Please run /login · API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"OAuth token has expired. Please obtain a new token or refresh your existing token."},"request_id":"req_011CZYX2NgB9Mx9LgH455BDt"}


<local-command-caveat>Caveat: The messages below were generated by the user while running local commands. DO NOT respond to these messages or otherwise consider them in your response unless the user explicitly asks you to.</local-command-caveat>


<command-name>/login</command-name>
            <command-message>login</command-message>
            <command-args></command-args>


<local-command-stdout>Login successful</local-command-stdout>


On it. Let me pull the 55k checkpoint and run the full pipeline.


source /home/ubuntu/training/.env && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

Verify 55k ckpt in R2


nvidia-smi --query-compute-apps=pid,name,used_gpu_memory --format=csv 2>&1

Check GPU availability


                           PRE ckpt-20000/
                           PRE ckpt-40000/
                           PRE ckpt-55000/


pid, process_name, used_gpu_memory [MiB]


GPU free. Downloading model files only (skip optimizer).


source /home/ubuntu/training/.env && mkdir -p /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000 && for f in chat_template.jinja config.json generation_config.json model.safetensors processor_config.json tokenizer.json tokenizer_config.json; do
  AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-55000/$f" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/$f" --endpoint-url "$R2_ENDPOINT_URL" --region auto &
done
wait && echo "DONE"

Download ckpt-55000 model files


Command running in background with ID: bw3yrke4y. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bw3yrke4y.output


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check download progress


600M	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check download progress


1.8G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check download progress


2.7G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check download progress


3.7G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


4.7G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


5.6G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


6.4G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


7.2G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


8.3G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


9.2G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


11G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


12G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


13G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check progress


32M	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/


<task-notification>
<task-id>bw3yrke4y</task-id>
<tool-use-id>toolu_016gvvUCvUgJxxrjv9oALPVS</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bw3yrke4y.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-55000 model files" completed (exit code 0)</summary>
</task-notification>


Dropped to 32M — the small files are done, model.safetensors temp file was cleaned. Let me check if it completed.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/ 2>&1

Check files


total 32M
-rw-rw-r-- 1 ubuntu ubuntu  613 Mar 30 01:57 chat_template.jinja
-rw-rw-r-- 1 ubuntu ubuntu 4.5K Mar 30 01:57 config.json
-rw-rw-r-- 1 ubuntu ubuntu  209 Mar 30 01:57 generation_config.json
-rw-rw-r-- 1 ubuntu ubuntu 2.3K Mar 30 01:57 processor_config.json
-rw-rw-r-- 1 ubuntu ubuntu 1004 Mar 30 01:57 tokenizer_config.json
-rw-rw-r-- 1 ubuntu ubuntu  32M Mar 30 01:57 tokenizer.json


model.safetensors is missing — download failed or still in progress. Let me check the background task.


ps aux | grep "s3 cp.*ckpt-55000" | grep -v grep 2>&1

Check if download is still running


(Bash completed with no output)


Download finished but model.safetensors is gone. Must have failed. Let me re-download just that file.


source /home/ubuntu/training/.env && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-55000/model.safetensors" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

Download ckpt-55000 model.safetensors


Command running in background with ID: bvw6r94e3. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bvw6r94e3.output


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check model file


-rw-rw-r-- 1 ubuntu ubuntu 1.3G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


Downloading. ~1GB/min pace. ~24G total, so about 20 min remaining.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check model file


-rw-rw-r-- 1 ubuntu ubuntu 3.0G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 4.7G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 6.1G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 7.4G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 8.8G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 11G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 13G Mar 30 03:04 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


that's too slow. use multipart or parallel downloads


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 14G Mar 30 03:05 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0


Let me kill this and use parallel multipart download.


pkill -f "s3 cp.*ckpt-55000" 2>&1; rm -f /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1; echo "killed and cleaned"

Kill slow download


<task-notification>
<task-id>bvw6r94e3</task-id>
<tool-use-id>toolu_01QeznvBTiRuVvaa46W3KPMW</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bvw6r94e3.output</output-file>
<status>failed</status>
<summary>Background command "Download ckpt-55000 model.safetensors" failed with exit code 144</summary>
</task-notification>


Exit code 144


source /home/ubuntu/training/.env && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws configure set default.s3.max_concurrent_requests 50 && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws configure set default.s3.multipart_chunksize 64MB && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-55000/model.safetensors" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

Download with multipart parallel (50 concurrent, 64MB chunks)


Command running in background with ID: budyqdjms. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/budyqdjms.output


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check multipart download progress


-rw-rw-r-- 1 ubuntu ubuntu  15G Mar 30 03:05 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0
-rw-rw-r-- 1 ubuntu ubuntu 6.2G Mar 30 03:05 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


Old temp file still there plus new one. 6.2G already on the new one — much faster.


rm -f /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.E5CBf5B0 2>&1; ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Clean old temp, check new progress


-rw-rw-r-- 1 ubuntu ubuntu 9.4G Mar 30 03:05 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 11G Mar 30 03:05 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 13G Mar 30 03:06 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 16G Mar 30 03:06 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


~3GB/min now vs 1GB/min before. 3x faster with multipart.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 18G Mar 30 03:06 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 19G Mar 30 03:06 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 22G Mar 30 03:06 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors.eDEda4ED


<task-notification>
<task-id>budyqdjms</task-id>
<tool-use-id>toolu_01T4tG2i5FNQEr9JKpwk1i85</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/budyqdjms.output</output-file>
<status>completed</status>
<summary>Background command "Download with multipart parallel (50 concurrent, 64MB chunks)" completed (exit code 0)</summary>
</task-notification>


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors 2>&1

Check if complete


-rw-rw-r-- 1 ubuntu ubuntu 23G Mar 30 02:22 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000/model.safetensors


Done. 23G model.safetensors downloaded. Starting benchmark.


python3 benchmark_gemma3n_asr.py \
    --checkpoint /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-55000 \
    --checkpoint-name ckpt-55000 \
    --output-dir /home/ubuntu/training/benchmark_results_gemma3n \
    --batch-size 128 2>&1

Run ckpt-55000 benchmark (BS=128)


Command running in background with ID: bfbobditl. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check benchmark start


(Bash completed with no output)


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check benchmark


`torch_dtype` is deprecated! Use `dtype` instead!


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check benchmark


Found the latest cached dataset configuration 'default' at /home/ubuntu/training/datasets/indic-asr-benchmark-6k/BayAreaBoys___indic-asr-benchmark-6k/default/0.0.0/4d4bc89b7c915a6d80a1efbee3a708d58e688c81 (last modified on Thu Mar 26 01:11:34 2026).
WARNING:datasets.packaged_modules.cache.cache:Found the latest cached dataset configuration 'default' at /home/ubuntu/training/datasets/indic-asr-benchmark-6k/BayAreaBoys___indic-asr-benchmark-6k/default/0.0.0/4d4bc89b7c915a6d80a1efbee3a708d58e688c81 (last modified on Thu Mar 26 01:11:34 2026).
The following generation flags are not valid and may be ignored: ['top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


Running inference (batch_size=128)...
  [128/6000] (2%) processed


Running. Let me check at wider intervals — based on ckpt-40000, ~25 minutes total.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


Running inference (batch_size=128)...
  [128/6000] (2%) processed
  [256/6000] (4%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [128/6000] (2%) processed
  [256/6000] (4%) processed
  [384/6000] (6%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [256/6000] (4%) processed
  [384/6000] (6%) processed
  [512/6000] (9%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [384/6000] (6%) processed
  [512/6000] (9%) processed
  [640/6000] (11%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [512/6000] (9%) processed
  [640/6000] (11%) processed
  [768/6000] (13%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [640/6000] (11%) processed
  [768/6000] (13%) processed
  [896/6000] (15%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [768/6000] (13%) processed
  [896/6000] (15%) processed
  [1024/6000] (17%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [896/6000] (15%) processed
  [1024/6000] (17%) processed
  [1152/6000] (19%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1024/6000] (17%) processed
  [1152/6000] (19%) processed
  [1280/6000] (21%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1152/6000] (19%) processed
  [1280/6000] (21%) processed
  [1408/6000] (23%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1280/6000] (21%) processed
  [1408/6000] (23%) processed
  [1536/6000] (26%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1408/6000] (23%) processed
  [1536/6000] (26%) processed
  [1664/6000] (28%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1536/6000] (26%) processed
  [1664/6000] (28%) processed
  [1792/6000] (30%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1664/6000] (28%) processed
  [1792/6000] (30%) processed
  [1920/6000] (32%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [1792/6000] (30%) processed
  [1920/6000] (32%) processed
  [2048/6000] (34%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2048/6000] (34%) processed
  [2176/6000] (36%) processed
  [2304/6000] (38%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2176/6000] (36%) processed
  [2304/6000] (38%) processed
  [2432/6000] (41%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2304/6000] (38%) processed
  [2432/6000] (41%) processed
  [2560/6000] (43%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2432/6000] (41%) processed
  [2560/6000] (43%) processed
  [2688/6000] (45%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2560/6000] (43%) processed
  [2688/6000] (45%) processed
  [2816/6000] (47%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2688/6000] (45%) processed
  [2816/6000] (47%) processed
  [2944/6000] (49%) processed


Almost 50%. About 12 more minutes.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2816/6000] (47%) processed
  [2944/6000] (49%) processed
  [3072/6000] (51%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [2944/6000] (49%) processed
  [3072/6000] (51%) processed
  [3200/6000] (53%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3072/6000] (51%) processed
  [3200/6000] (53%) processed
  [3328/6000] (55%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3200/6000] (53%) processed
  [3328/6000] (55%) processed
  [3456/6000] (58%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3328/6000] (55%) processed
  [3456/6000] (58%) processed
  [3584/6000] (60%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3456/6000] (58%) processed
  [3584/6000] (60%) processed
  [3712/6000] (62%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3584/6000] (60%) processed
  [3712/6000] (62%) processed
  [3840/6000] (64%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3712/6000] (62%) processed
  [3840/6000] (64%) processed
  [3968/6000] (66%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3840/6000] (64%) processed
  [3968/6000] (66%) processed
  [4096/6000] (68%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [3968/6000] (66%) processed
  [4096/6000] (68%) processed
  [4224/6000] (70%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4096/6000] (68%) processed
  [4224/6000] (70%) processed
  [4352/6000] (73%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4224/6000] (70%) processed
  [4352/6000] (73%) processed
  [4480/6000] (75%) processed


75%. Almost there.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4352/6000] (73%) processed
  [4480/6000] (75%) processed
  [4608/6000] (77%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4480/6000] (75%) processed
  [4608/6000] (77%) processed
  [4736/6000] (79%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4608/6000] (77%) processed
  [4736/6000] (79%) processed
  [4864/6000] (81%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4736/6000] (79%) processed
  [4864/6000] (81%) processed
  [4992/6000] (83%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4864/6000] (81%) processed
  [4992/6000] (83%) processed
  [5120/6000] (85%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [4992/6000] (83%) processed
  [5120/6000] (85%) processed
  [5248/6000] (87%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [5120/6000] (85%) processed
  [5248/6000] (87%) processed
  [5376/6000] (90%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [5248/6000] (87%) processed
  [5376/6000] (90%) processed
  [5504/6000] (92%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [5376/6000] (90%) processed
  [5504/6000] (92%) processed
  [5632/6000] (94%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [5504/6000] (92%) processed
  [5632/6000] (94%) processed
  [5760/6000] (96%) processed


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check progress


  [5632/6000] (94%) processed
  [5760/6000] (96%) processed
  [5888/6000] (98%) processed


tail -20 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output 2>&1

Check for final results


Language           N     WER%     CER%   WER-n%   CER-n%
-------------- ----- -------- -------- -------- --------
assamese         500    21.35     9.97    18.21    10.04
bengali          500    19.58     5.28    15.20     4.73
english          500    28.16    15.05    17.61    12.19
gujarati         500    19.61     7.24    15.32     6.76
hindi            500    13.50     4.92     9.16     4.09
kannada          500    38.59    18.16    35.36    17.25
malayalam        500    39.98     8.27    35.38     7.43
marathi          500    24.57     7.87    19.64     7.04
odia             500    30.23    11.20    24.58     9.88
punjabi          500    25.32    13.39    19.73    11.59
tamil            500    34.35    11.47    32.18    10.96
telugu           500    28.14     7.66    23.22     7.15
-------------- ----- -------- -------- -------- --------
OVERALL         6000    26.67    10.11    21.91     9.32
MACRO-AVG         12    26.95    10.04    22.13     9.09
=====================================================================================

Sample analysis saved: /home/ubuntu/training/benchmark_results_gemma3n/ckpt-55000/sample_analysis.json


<task-notification>
<task-id>bfbobditl</task-id>
<tool-use-id>toolu_01MsWUCkDdcmPRYWSrRN8P5g</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bfbobditl.output</output-file>
<status>completed</status>
<summary>Background command "Run ckpt-55000 benchmark (BS=128)" completed (exit code 0)</summary>
</task-notification>


Benchmark done. Now generating dashboard JSONs.


python3 generate_schema_v1.py --src benchmark_results_gemma3n/ckpt-55000 --ckpt ckpt-55000 2>&1

Generate dashboard JSON for ckpt-55000


============================================================
  ckpt-55000
============================================================
  Loaded 6000 predictions
  Overall: wer_raw=26.58 wer_norm=25.64 wer_nc=25.64 snwer=18.68 mer=10.26 cer_norm=9.9
  Diagnosis: recognition-limited


ls -lh /home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-55000/ 2>&1

Verify output files


total 12M
-rw-rw-r-- 1 ubuntu ubuntu  64K Mar 30 03:17 error_analysis.json
-rw-rw-r-- 1 ubuntu ubuntu 5.2K Mar 30 03:17 metrics.json
-rw-rw-r-- 1 ubuntu ubuntu  11M Mar 30 03:17 sample_analysis.json


All done. Here's the ckpt-55000 results and the full trend:

### Gemma3n-E2B-ASR Progression (03-28-2026 training)

| Metric | ckpt-20000 | ckpt-40000 | **ckpt-55000** |
|--------|-----------|-----------|------------|
| **wer_norm** | 29.39 | 26.14 | **25.64** |
| wer_raw | 30.43 | 27.01 | 26.58 |
| space_norm_wer | 20.94 | 19.15 | 18.68 |
| mer | 13.21 | 10.61 | 10.26 |
| cer_norm | 12.73 | 10.21 | 9.90 |

**Per-language WER_norm at ckpt-55000:**
| Language | WER_norm% |
|----------|----------|
| Hindi | **9.16** |
| Bengali | 15.20 |
| Gujarati | 15.32 |
| English | 17.61 |
| Assamese | 18.21 |
| Punjabi | 19.73 |
| Marathi | 19.64 |
| Telugu | 23.22 |
| Odia | 24.58 |
| Tamil | 32.18 |
| Kannada | 35.36 |
| Malayalam | 35.38 |

Model is still improving steadily — overall WER_norm dropped from 29.4% → 26.1% → **25.6%**. Dashboard files placed at `/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-55000/`.


<task-notification>
<task-id>bw3yrke4y</task-id>
<tool-use-id>toolu_016gvvUCvUgJxxrjv9oALPVS</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bw3yrke4y.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-55000 model files" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>bvw6r94e3</task-id>
<tool-use-id>toolu_01QeznvBTiRuVvaa46W3KPMW</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bvw6r94e3.output</output-file>
<status>failed</status>
<summary>Background command "Download ckpt-55000 model.safetensors" failed with exit code 144</summary>
</task-notification>


<task-notification>
<task-id>budyqdjms</task-id>
<tool-use-id>toolu_01T4tG2i5FNQEr9JKpwk1i85</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/budyqdjms.output</output-file>
<status>completed</status>
<summary>Background command "Download with multipart parallel (50 concurrent, 64MB chunks)" completed (exit code 0)</summary>
</task-notification>


All background tasks completed. The failed one (`bvw6r94e3`) was the slow download I killed intentionally before switching to multipart. Everything else succeeded — ckpt-55000 benchmark and dashboard JSONs are done and placed at `/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-55000/`.


55k ckpt is in R2. mind running bench throgh it.


in exactly 30-40mins, there is going to be a checkpoint of 70k in R2. that is like 45% of total training epoch. put a sleep timer for 40mins, or a bg agent or one-time cron to trigger you in next 40mins, and when it's time, check R2 for ckpt and pull it and run benchmark on that and add to dashboard too.


echo "Waiting 40 minutes for ckpt-70000 to appear in R2..." && sleep 2400 && source /home/ubuntu/training/.env && echo "=== Checking R2 ===" && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

Sleep 40min then check R2 for ckpt-70000


Command running in background with ID: bbr0xflgj. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bbr0xflgj.output


Set a 40-minute timer running in background. When it fires, it'll check R2 for ckpt-70000. I'll be notified when it completes and will then pull the checkpoint, run the benchmark, and update the dashboard.


<task-notification>
<task-id>bbr0xflgj</task-id>
<tool-use-id>toolu_01Q6V9b73KEiEUJ5AvWseE53</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bbr0xflgj.output</output-file>
<status>completed</status>
<summary>Background command "Sleep 40min then check R2 for ckpt-70000" completed (exit code 0)</summary>
</task-notification>


Timer fired. Let me check the result.


Check R2 listing


Waiting 40 minutes for ckpt-70000 to appear in R2...
=== Checking R2 ===
                           PRE ckpt-20000/
                           PRE ckpt-40000/
                           PRE ckpt-55000/
                           PRE ckpt-70000/


ckpt-70000 is there. Pulling and benchmarking now.


source /home/ubuntu/training/.env && aws configure set default.s3.max_concurrent_requests 50 && aws configure set default.s3.multipart_chunksize 64MB && mkdir -p /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000 && for f in chat_template.jinja config.json generation_config.json processor_config.json tokenizer.json tokenizer_config.json; do
  AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/$f" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/$f" --endpoint-url "$R2_ENDPOINT_URL" --region auto &
done
# Download model.safetensors with multipart
AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/model.safetensors" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors" --endpoint-url "$R2_ENDPOINT_URL" --region auto &
wait && echo "DONE"

Download ckpt-70000 (multipart parallel)


Command running in background with ID: bjm3bidp5. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bjm3bidp5.output


<task-notification>
<task-id>bjm3bidp5</task-id>
<tool-use-id>toolu_01Rskscec23cjeCbMB2y8434</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bjm3bidp5.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-70000 (multipart parallel)" completed (exit code 0)</summary>
</task-notification>


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/ 2>&1

Check download progress


32M	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/ 2>&1

Check progress


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check model file


Error: Exit code 2
ls: cannot access '/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors*': No such file or directory


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/ 2>&1

Check all files


total 32M
-rw-rw-r-- 1 ubuntu ubuntu  613 Mar 30 16:23 chat_template.jinja
-rw-rw-r-- 1 ubuntu ubuntu 4.5K Mar 30 16:23 config.json
-rw-rw-r-- 1 ubuntu ubuntu  209 Mar 30 16:23 generation_config.json
-rw-rw-r-- 1 ubuntu ubuntu 1004 Mar 30 16:23 tokenizer_config.json
-rw-rw-r-- 1 ubuntu ubuntu  32M Mar 30 16:23 tokenizer.json


Small files done, model.safetensors still downloading. Let me wait.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1 || echo "model not started yet"

Check model download


ls: cannot access '/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors*': No such file or directory
model not started yet


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1 || echo "not yet"; ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json 2>&1 || echo "processor missing too"

Check files


ls: cannot access '/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors*': No such file or directory
not yet
ls: cannot access '/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json': No such file or directory
processor missing too


Upload to R2 may still be in progress. Let me check what's actually in R2 for this checkpoint.


source /home/ubuntu/training/.env && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

List ckpt-70000 files in R2


2026-03-30 16:23:34        613 chat_template.jinja
2026-03-30 16:23:34       4565 config.json
2026-03-30 16:23:34        209 generation_config.json
2026-03-30 16:23:37   33442428 tokenizer.json
2026-03-30 16:23:34       1004 tokenizer_config.json
2026-03-30 16:23:34       5649 training_args.bin


model.safetensors and processor_config.json aren't in R2 yet — upload is still in progress. The training agent is still pushing. Let me wait and poll every 2 minutes.


pkill -f "s3 cp.*ckpt-70000" 2>/dev/null; echo "killed stale downloads"

Kill stale downloads that would fail


Exit code 144


# Poll R2 every 2 minutes until model.safetensors appears
source /home/ubuntu/training/.env
for i in $(seq 1 30); do
  count=$(AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/model.safetensors" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1 | wc -l)
  if [ "$count" -gt 0 ]; then
    echo "model.safetensors FOUND in R2"
    AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1
    break
  fi
  echo "Poll $i: model.safetensors not yet in R2, waiting 2min..."
  sleep 120
done

Poll R2 every 2min for model.safetensors


Command running in background with ID: b3w3pkmiw. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output


Polling R2 every 2 minutes for model.safetensors to appear. Once it's there, I'll download, benchmark, and push to dashboard. I'll be notified when the poll finds it.


Check poll status


Poll 1: model.safetensors not yet in R2, waiting 2min...


Still waiting. I'll check back when the background task completes.


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


Check poll status


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll status


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll status


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll status


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll status


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


Poll 1: model.safetensors not yet in R2, waiting 2min...
Poll 2: model.safetensors not yet in R2, waiting 2min...


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


2 polls done (4 min). The model.safetensors upload (~24GB) can take 15-20 min from the training machine. I'll wait for the background task to find it.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


Poll 1: model.safetensors not yet in R2, waiting 2min...
Poll 2: model.safetensors not yet in R2, waiting 2min...
Poll 3: model.safetensors not yet in R2, waiting 2min...


3 polls (6 min). Still uploading. I'll keep checking periodically.


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


<task-notification>
<task-id>b3w3pkmiw</task-id>
<tool-use-id>toolu_012oQacYaKp5ttTsqZ6fY5h3</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output</output-file>
<status>completed</status>
<summary>Background command "Poll R2 every 2min for model.safetensors" completed (exit code 0)</summary>
</task-notification>


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output 2>&1

Check poll


2026-03-30 16:51:55       1465 scheduler.pt
2026-03-30 16:23:37   33442428 tokenizer.json
2026-03-30 16:23:34       1004 tokenizer_config.json
2026-03-30 16:51:55    1313182 trainer_state.json
2026-03-30 16:23:34       5649 training_args.bin


Wait — the poll found new files (scheduler.pt, trainer_state.json appeared at 16:51) but model.safetensors still not there. The upload is progressing but model weights are last. Let me check the full output.


Check full poll output


Poll 1: model.safetensors not yet in R2, waiting 2min...
Poll 2: model.safetensors not yet in R2, waiting 2min...
Poll 3: model.safetensors not yet in R2, waiting 2min...
model.safetensors FOUND in R2
2026-03-30 16:23:34        613 chat_template.jinja
2026-03-30 16:23:34       4565 config.json
2026-03-30 16:23:34        209 generation_config.json
2026-03-30 16:51:54 23907545024 model.safetensors
2026-03-30 16:51:56       2262 processor_config.json
2026-03-30 16:51:56      16389 rng_state_0.pth
2026-03-30 16:51:55      16389 rng_state_1.pth
2026-03-30 16:51:55      16389 rng_state_2.pth
2026-03-30 16:51:56      16389 rng_state_3.pth
2026-03-30 16:51:55      16389 rng_state_4.pth
2026-03-30 16:51:56      16389 rng_state_5.pth
2026-03-30 16:51:55      16389 rng_state_6.pth
2026-03-30 16:51:55      16389 rng_state_7.pth
2026-03-30 16:51:55       1465 scheduler.pt
2026-03-30 16:23:37   33442428 tokenizer.json
2026-03-30 16:23:34       1004 tokenizer_config.json
2026-03-30 16:51:55    1313182 trainer_state.json
2026-03-30 16:23:34       5649 training_args.bin


model.safetensors is in R2 (23.9GB, uploaded at 16:51). Downloading now and running the full pipeline.


source /home/ubuntu/training/.env && for f in model.safetensors processor_config.json; do
  AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/$f" "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/$f" --endpoint-url "$R2_ENDPOINT_URL" --region auto &
done
wait && echo "DOWNLOAD DONE" && python3 /home/ubuntu/training/benchmark_gemma3n_asr.py --checkpoint /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000 --checkpoint-name ckpt-70000 --output-dir /home/ubuntu/training/benchmark_results_gemma3n --batch-size 128 && echo "BENCHMARK DONE" && python3 /home/ubuntu/training/generate_schema_v1.py --src /home/ubuntu/training/benchmark_results_gemma3n/ckpt-70000 --ckpt ckpt-70000 && echo "DASHBOARD DONE"

Download + benchmark + dashboard for ckpt-70000


Command running in background with ID: bmxvx7yzg. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output


Full pipeline launched in background: download → benchmark (BS=128, 6000 samples) → dashboard JSON generation. I'll be notified when it completes.


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1

Check pipeline progress


<persisted-output>
Output too large (887.6KB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tool-results/bu7plxy3f.txt

Preview (first 2KB):
Completed 2.2 KiB/2.2 KiB (8.1 KiB/s) with 1 file(s) remaining
download: s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/processor_config.json to checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json
Completed 256.0 KiB/22.3 GiB (552.5 KiB/s) with 1 file(s) remaining
Completed 512.0 KiB/22.3 GiB (1.0 MiB/s) with 1 file(s) remaining
Completed 768.0 KiB/22.3 GiB (1.4 MiB/s) with 1 file(s) remaining
Completed 1.0 MiB/22.3 GiB (1.9 MiB/s) with 1 file(s) remaining
Completed 1.2 MiB/22.3 GiB (2.3 MiB/s) with 1 file(s) remaining
Completed 1.5 MiB/22.3 GiB (2.8 MiB/s) with 1 file(s) remaining
Completed 1.8 MiB/22.3 GiB (3.1 MiB/s) with 1 file(s) remaining
Completed 2.0 MiB/22.3 GiB (3.5 MiB/s) with 1 file(s) remaining
Completed 2.2 MiB/22.3 GiB (3.9 MiB/s) with 1 file(s) remaining
Completed 2.5 MiB/22.3 GiB (4.3 MiB/s) with 1 file(s) remaining
Completed 2.8 MiB/22.3 GiB (4.7 MiB/s) with 1 file(s) remaining
Completed 3.0 MiB/22.3 GiB (5.1 MiB/s) with 1 file(s) remaining
Completed 3.2 MiB/22.3 GiB (5.5 MiB/s) with 1 file(s) remaining
Completed 3.5 MiB/22.3 GiB (5.9 MiB/s) with 1 file(s) remaining
Completed 3.8 MiB/22.3 GiB (6.4 MiB/s) with 1 file(s) remaining
Completed 4.0 MiB/22.3 GiB (6.6 MiB/s) with 1 file(s) remaining
Completed 4.2 MiB/22.3 GiB (7.0 MiB/s) with 1 file(s) remaining
Completed 4.5 MiB/22.3 GiB (7.4 MiB/s) with 1 file(s) remaining
Completed 4.8 MiB/22.3 GiB (7.8 MiB/s) with 1 file(s) remaining
Completed 5.0 MiB/22.3 GiB (8.1 MiB/s) with 1 file(s) remaining
Completed 5.2 MiB/22.3 GiB (8.5 MiB/s) with 1 file(s) remaining
Completed 5.5 MiB/22.3 GiB (8.9 MiB/s) with 1 file(s) remaining
Completed 5.8 MiB/22.3 GiB (9.3 MiB/s) with 1 file(s) remaining
Completed 6.0 MiB/22.3 GiB (9.6 MiB/s) with 1 file(s) remaining
Completed 6.2 MiB/22.3 GiB (10.0 MiB/s) with 1 file(s) remaining
Completed 6.5 MiB/22.3 GiB (10.4 MiB/s) with 1 file(s) remaining
Completed 6.8 MiB/22
...
</persisted-output>

Completed 2.2 KiB/2.2 KiB (8.1 KiB/s) with 1 file(s) remaining
download: s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/processor_config.json to checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json
Completed 256.0 KiB/22.3 GiB (552.5 KiB/s) with 1 file(s) remaining
Completed 512.0 KiB/22.3 GiB (1.0 MiB/s) with 1 file(s) remaining
Completed 768.0 KiB/22.3 GiB (1.4 MiB/s) with 1 file(s) remaining
Completed 1.0 MiB/22.3 GiB (1.9 MiB/s) with 1 file(s) remaining
Completed 1.2 MiB/22.3 GiB (2.3 MiB/s) with 1 file(s) remaining
Completed 1.5 MiB/22.3 GiB (2.8 MiB/s) with 1 file(s) remaining
Completed 1.8 MiB/22.3 GiB (3.1 MiB/s) with 1 file(s) remaining
Completed 2.0 MiB/22.3 GiB (3.5 MiB/s) with 1 file(s) remaining
Completed 2.2 MiB/22.3 GiB (3.9 MiB/s) with 1 file(s) remaining
Completed 2.5 MiB/22.3 GiB (4.3 MiB/s) with 1 file(s) remaining
Completed 2.8 MiB/22.3 GiB (4.7 MiB/s) with 1 file(s) remaining
Completed 3.0 MiB/22.3 GiB (5.1 MiB/s) with 1 file(s) remaining
Completed 3.2 MiB/22.3 GiB (5.5 MiB/s) with 1 file(s) remaining
Completed 3.5 MiB/22.3 GiB (5.9 MiB/s) with 1 file(s) remaining
Completed 3.8 MiB/22.3 GiB (6.4 MiB/s) with 1 file(s) remaining
Completed 4.0 MiB/22.3 GiB (6.6 MiB/s) with 1 file(s) remaining
Completed 4.2 MiB/22.3 GiB (7.0 MiB/s) with 1 file(s) remaining
Completed 4.5 MiB/22.3 GiB (7.4 MiB/s) with 1 file(s) remaining
Completed 4.8 MiB/22.3 GiB (7.8 MiB/s) with 1 file(s) remaining
Completed 5.0 MiB/22.3 GiB (8.1 MiB/s) with 1 file(s) remaining
Completed 5.2 MiB/22.3 GiB (8.5 MiB/s) with 1 file(s) remaining
Completed 5.5 MiB/22.3 GiB (8.9 MiB/s) with 1 file(s) remaining
Completed 5.8 MiB/22.3 GiB (9.3 MiB/s) with 1 file(s) remaining
Completed 6.0 MiB/22.3 GiB (9.6 MiB/s) with 1 file(s) remaining
Completed 6.2 MiB/22.3 GiB (10.0 MiB/s) with 1 file(s) remaining
Completed 6.5 MiB/22.3 GiB (10.4 MiB/s) with 1 file(s) remaining
Completed 6.8 MiB/22.3 GiB (10.8 MiB/s) with 1 file(s) remaining
Completed 7.0 MiB/22.3 GiB (11.2 MiB/s) with 1 file(s) remaining
Completed 7.2 MiB/22.3 GiB (11.6 MiB/s) with 1 file(s) remaining
Completed 7.5 MiB/22.3 GiB (12.0 MiB/s) with 1 file(s) remaining
Completed 7.8 MiB/22.3 GiB (12.3 MiB/s) with 1 file(s) remaining
Completed 8.0 MiB/22.3 GiB (12.7 MiB/s) with 1 file(s) remaining
Completed 8.2 MiB/22.3 GiB (13.1 MiB/s) with 1 file(s) remaining
Completed 8.5 MiB/22.3 GiB (13.4 MiB/s) with 1 file(s) remaining
Completed 8.8 MiB/22.3 GiB (13.8 MiB/s) with 1 file(s) remaining
Completed 9.0 MiB/22.3 GiB (14.2 MiB/s) with 1 file(s) remaining
Completed 9.2 MiB/22.3 GiB (14.5 MiB/s) with 1 file(s) remaining
Completed 9.5 MiB/22.3 GiB (14.8 MiB/s) with 1 file(s) remaining
Completed 9.8 MiB/22.3 GiB (15.2 MiB/s) with 1 file(s) remaining
Completed 10.0 MiB/22.3 GiB (15.6 MiB/s) with 1 file(s) remaining
Completed 10.2 MiB/22.3 GiB (15.9 MiB/s) with 1 file(s) remaining
Completed 10.5 MiB/22.3 GiB (16.3 MiB/s) with 1 file(s) remaining
Completed 10.8 MiB/22.3 GiB (16.6 MiB/s) with 1 file(s) remaining
Completed 11.0 MiB/22.3 GiB (17.0 MiB/s) with 1 file(s) remaining
Completed 11.2 MiB/22.3 GiB (17.3 MiB/s) with 1 file(s) remaining
Completed 11.5 MiB/22.3 GiB (17.6 MiB/s) with 1 file(s) remaining
Completed 11.8 MiB/22.3 GiB (18.0 MiB/s) with 1 file(s) remaining
Completed 12.0 MiB/22.3 GiB (18.3 MiB/s) with 1 file(s) remaining
Completed 12.2 MiB/22.3 GiB (18.7 MiB/s) with 1 file(s) remaining
Completed 12.5 MiB/22.3 GiB (19.0 MiB/s) with 1 file(s) remaining
Completed 12.8 MiB/22.3 GiB (19.4 MiB/s) with 1 file(s) remaining
Completed 13.0 MiB/22.3 GiB (19.7 MiB/s) with 1 file(s) remaining
Completed 13.2 MiB/22.3 GiB (20.1 MiB/s) with 1 file(s) remaining
Completed 13.5 MiB/22.3 GiB (20.3 MiB/s) with 1 file(s) remaining
Completed 13.8 MiB/22.3 GiB (20.6 MiB/s) with 1 file(s) remaining
Completed 14.0 MiB/22.3 GiB (21.0 MiB/s) with 1 file(s) remaining
Completed 14.2 MiB/22.3 GiB (21.3 MiB/s) with 1 file(s) remaining
Completed 14.5 MiB/22.3 GiB (21.6 MiB/s) with 1 file(s) remaining
Completed 14.8 MiB/22.3 GiB (21.9 MiB/s) with 1 file(s) remaining
Completed 15.0 MiB/22.3 GiB (22.2 MiB/s) with 1 file(s) remaining
Completed 15.2 MiB/22.3 GiB (22.5 MiB/s) with 1 file(s) remaining
Completed 15.5 MiB/22.3 GiB (22.9 MiB/s) with 1 file(s) remaining
Completed 15.8 MiB/22.3 GiB (23.1 MiB/s) with 1 file(s) remaining
Completed 16.0 MiB/22.3 GiB (23.4 MiB/s) with 1 file(s) remaining
Completed 16.2 MiB/22.3 GiB (23.7 MiB/s) with 1 file(s) remaining
Completed 16.5 MiB/22.3 GiB (24.0 MiB/s) with 1 file(s) remaining
Completed 16.8 MiB/22.3 GiB (24.3 MiB/s) with 1 file(s) remaining
Completed 17.0 MiB/22.3 GiB (24.7 MiB/s) with 1 file(s) remaining
Completed 17.2 MiB/22.3 GiB (25.0 MiB/s) with 1 file(s) remaining
Completed 17.5 MiB/22.3 GiB (25.2 MiB/s) with 1 file(s) remaining
Completed 17.8 MiB/22.3 GiB (25.6 MiB/s) with 1 file(s) remaining
Completed 18.0 MiB/22.3 GiB (25.9 MiB/s) with 1 file(s) remaining
Completed 18.2 MiB/22.3 GiB (26.2 MiB/s) with 1 file(s) remaining
Completed 18.5 MiB/22.3 GiB (26.6 MiB/s) with 1 file(s) remaining
Completed 18.8 MiB/22.3 GiB (26.9 MiB/s) with 1 file(s) remaining
Completed 19.0 MiB/22.3 GiB (27.2 MiB/s) with 1 file(s) remaining
Completed 19.2 MiB/22.3 GiB (27.5 MiB/s) with 1 file(s) remaining
Completed 19.5 MiB/22.3 GiB (27.9 MiB/s) with 1 file(s) remaining
Completed 19.8 MiB/22.3 GiB (28.2 MiB/s) with 1 file(s) remaining
Completed 20.0 MiB/22.3 GiB (28.5 MiB/s) with 1 file(s) remaining
Completed 20.2 MiB/22.3 GiB (28.5 MiB/s) with 1 file(s) remaining
Completed 20.5 MiB/22.3 GiB (28.8 MiB/s) with 1 file(s) remaining
Completed 20.8 MiB/22.3 GiB (29.0 MiB/s) with 1 file(s) remaining
Completed 21.0 MiB/22.3 GiB (29.1 MiB/s) with 1 file(s) remaining
Completed 21.2 MiB/22.3 GiB (29.4 MiB/s) with 1 file(s) remaining
Completed 21.5 MiB/22.3 GiB (29.7 MiB/s) with 1 file(s) remaining
Completed 21.8 MiB/22.3 GiB (29.9 MiB/s) with 1 file(s) remaining
Completed 22.0 MiB/22.3 GiB (30.2 MiB/s) with 1 file(s) remaining
Completed 22.2 MiB/22.3 GiB (30.5 MiB/s) with 1 file(s) remaining
Completed 22.5 MiB/22.3 GiB (30.8 MiB/s) with 1 file(s) remaining
Completed 22.8 MiB/22.3 GiB (31.1 MiB/s) with 1 file(s) remaining
Completed 23.0 MiB/22.3 GiB (31.3 MiB/s) with 1 file(s) remaining
Completed 23.2 MiB/22.3 GiB (31.6 MiB/s) with 1 file(s) remaining
Completed 23.5 MiB/22.3 GiB (31.9 MiB/s) with 1 file(s) remaining
Completed 23.8 MiB/22.3 GiB (32.3 MiB/s) with 1 file(s) remaining
Completed 24.0 MiB/22.3 GiB (32.5 MiB/s) with 1 file(s) remaining
Completed 24.2 MiB/22.3 GiB (32.8 MiB/s) with 1 file(s) remaining
Completed 24.5 MiB/22.3 GiB (33.0 MiB/s) with 1 file(s) remaining
Completed 24.8 MiB/22.3 GiB (33.3 MiB/s) with 1 file(s) remaining
Completed 25.0 MiB/22.3 GiB (33.5 MiB/s) with 1 file(s) remaining
Completed 25.2 MiB/22.3 GiB (33.9 MiB/s) with 1 file(s) remaining
Completed 25.5 MiB/22.3 GiB (33.7 MiB/s) with 1 file(s) remaining
Completed 25.8 MiB/22.3 GiB (34.0 MiB/s) with 1 file(s) remaining
Completed 26.0 MiB/22.3 GiB (34.3 MiB/s) with 1 file(s) remaining
Completed 26.2 MiB/22.3 GiB (34.6 MiB/s) with 1 file(s) remaining
Completed 26.5 MiB/22.3 GiB (34.9 MiB/s) with 1 file(s) remaining
Completed 26.8 MiB/22.3 GiB (35.2 MiB/s) with 1 file(s) remaining
Completed 27.0 MiB/22.3 GiB (35.4 MiB/s) with 1 file(s) remaining
Completed 27.2 MiB/22.3 GiB (35.5 MiB/s) with 1 file(s) remaining
Completed 27.5 MiB/22.3 GiB (35.8 MiB/s) with 1 file(s) remaining
Completed 27.8 MiB/22.3 GiB (36.1 MiB/s) with 1 file(s) remaining
Completed 28.0 MiB/22.3 GiB (36.3 MiB/s) with 1 file(s) remaining
Completed 28.2 MiB/22.3 GiB (36.7 MiB/s) with 1 file(s) remaining
Completed 28.5 MiB/22.3 GiB (37.0 MiB/s) with 1 file(s) remaining
Completed 28.8 MiB/22.3 GiB (37.2 MiB/s) with 1 file(s) remaining
Completed 29.0 MiB/22.3 GiB (37.4 MiB/s) with 1 file(s) remaining
Completed 29.2 MiB/22.3 GiB (37.7 MiB/s) with 1 file(s) remaining
Completed 29.5 MiB/22.3 GiB (38.0 MiB/s) with 1 file(s) remaining
Completed 29.8 MiB/22.3 GiB (38.3 MiB/s) with 1 file(s) remaining
Completed 30.0 MiB/22.3 GiB (38.6 MiB/s) with 1 file(s) remaining
Completed 30.2 MiB/22.3 GiB (38.8 MiB/s) with 1 file(s) remaining
Completed 30.5 MiB/22.3 GiB (39.1 MiB/s) with 1 file(s) remaining
Completed 30.8 MiB/22.3 GiB (39.2 MiB/s) with 1 file(s) remaining
Completed 31.0 MiB/22.3 GiB (39.5 MiB/s) with 1 file(s) remaining
Completed 31.2 MiB/22.3 GiB (39.7 MiB/s) with 1 file(s) remaining
Completed 31.5 MiB/22.3 GiB (40.0 MiB/s) with 1 file(s) remaining
Completed 31.8 MiB/22.3 GiB (40.3 MiB/s) with 1 file(s) remaining
Completed 32.0 MiB/22.3 GiB (40.6 MiB/s) with 1 file(s) remaining
Completed 32.2 MiB/22.3 GiB (40.8 MiB/s) with 1 file(s) remaining
Completed 32.5 MiB/22.3 GiB (41.1 MiB/s) with 1 file(s) remaining
Completed 32.8 MiB/22.3 GiB (41.4 MiB/s) with 1 file(s) remaining
Completed 33.0 MiB/22.3 GiB (41.6 MiB/s) with 1 file(s) remaining
Completed 33.2 MiB/22.3 GiB (41.8 MiB/s) with 1 file(s) remaining
Completed 33.5 MiB/22.3 GiB (42.1 MiB/s) with 1 file(s) remaining
Completed 33.8 MiB/22.3 GiB (42.3 MiB/s) with 1 file(s) remaining
Completed 34.0 MiB/22.3 GiB (42.6 MiB/s) with 1 file(s) remaining
Completed 34.2 MiB/22.3 GiB (42.9 MiB/s) with 1 file(s) remaining
Completed 34.5 MiB/22.3 GiB (43.2 MiB/s) with 1 file(s) remaining
Completed 34.8 MiB/22.3 GiB (43.5 MiB/s) with 1 file(s) remaining
Completed 35.0 MiB/22.3 GiB (43.6 MiB/s) with 1 file(s) remaining
Completed 35.2 MiB/22.3 GiB (43.8 MiB/s) with 1 file(s) remaining
Completed 35.5 MiB/22.3 GiB (43.9 MiB/s) with 1 file(s) remaining
Completed 35.8 MiB/22.3 GiB (44.1 MiB/s) with 1 file(s) remaining
Completed 36.0 MiB/22.3 GiB (44.4 MiB/s) with 1 file(s) remaining
Completed 36.2 MiB/22.3 GiB (44.7 MiB/s) with 1 file(s) remaining
Completed 36.5 MiB/22.3 GiB (45.0 MiB/s) with 1 file(s) remaining
Completed 36.8 MiB/22.3 GiB (45.2 MiB/s) with 1 file(s) remaining
Completed 37.0 MiB/22.3 GiB (45.5 MiB/s) with 1 file(s) remaining
Completed 37.2 MiB/22.3 GiB (45.7 MiB/s) with 1 file(s) remaining
Completed 37.5 MiB/22.3 GiB (45.9 MiB/s) with 1 file(s) remaining
Completed 37.8 MiB/22.3 GiB (46.1 MiB/s) with 1 file(s) remaining
Completed 38.0 MiB/22.3 GiB (46.1 MiB/s) with 1 file(s) remaining
Completed 38.2 MiB/22.3 GiB (46.3 MiB/s) with 1 file(s) remaining
Completed 38.5 MiB/22.3 GiB (46.5 MiB/s) with 1 file(s) remaining
Completed 38.8 MiB/22.3 GiB (46.8 MiB/s) with 1 file(s) remaining
Completed 39.0 MiB/22.3 GiB (47.1 MiB/s) with 1 file(s) remaining
Completed 39.2 MiB/22.3 GiB (47.4 MiB/s) with 1 file(s) remaining
Completed 39.5 MiB/22.3 GiB (47.6 MiB/s) with 1 file(s) remaining
Completed 39.8 MiB/22.3 GiB (47.9 MiB/s) with 1 file(s) remaining
Completed 40.0 MiB/22.3 GiB (48.2 MiB/s) with 1 file(s) remaining
Completed 40.2 MiB/22.3 GiB (48.4 MiB/s) with 1 file(s) remaining
Completed 40.5 MiB/22.3 GiB (48.7 MiB/s) with 1 file(s) remaining
Completed 40.8 MiB/22.3 GiB (49.0 MiB/s) with 1 file(s) remaining
Completed 41.0 MiB/22.3 GiB (49.2 MiB/s) with 1 file(s) remaining
Completed 41.2 MiB/22.3 GiB (49.4 MiB/s) with 1 file(s) remaining
Completed 41.5 MiB/22.3 GiB (49.6 MiB/s) with 1 file(s) remaining
Completed 41.8 MiB/22.3 GiB (49.9 MiB/s) with 1 file(s) remaining
Completed 42.0 MiB/22.3 GiB (50.1 MiB/s) with 1 file(s) remaining
Completed 42.2 MiB/22.3 GiB (50.4 MiB/s) with 1 file(s) remaining
Completed 42.5 MiB/22.3 GiB (50.6 MiB/s) with 1 file(s) remaining
Completed 42.8 MiB/22.3 GiB (50.8 MiB/s) with 1 file(s) remaining
Completed 43.0 MiB/22.3 GiB (51.1 MiB/s) with 1 file(s) remaining
Completed 43.2 MiB/22.3 GiB (51.2 MiB/s) with 1 file(s) remaining
Completed 43.5 MiB/22.3 GiB (51.4 MiB/s) with 1 file(s) remaining
Completed 43.8 MiB/22.3 GiB (51.7 MiB/s) with 1 file(s) remaining
Completed 44.0 MiB/22.3 GiB (52.0 MiB/s) with 1 file(s) remaining
Completed 44.2 MiB/22.3 GiB (52.3 MiB/s) with 1 file(s) remaining
Completed 44.5 MiB/22.3 GiB (52.5 MiB/s) with 1 file(s) remaining
Completed 44.8 MiB/22.3 GiB (52.8 MiB/s) with 1 file(s) remaining
Completed 45.0 MiB/22.3 GiB (53.0 MiB/s) with 1 file(s) remaining
Completed 45.2 MiB/22.3 GiB (53.1 MiB/s) with 1 file(s) remaining
Completed 45.5 MiB/22.3 GiB (53.3 MiB/s) with 1 file(s) remaining
Completed 45.8 MiB/22.3 GiB (53.5 MiB/s) with 1 file(s) remaining
Completed 46.0 MiB/22.3 GiB (53.8 MiB/s) with 1 file(s) remaining
Completed 46.2 MiB/22.3 GiB (54.0 MiB/s) with 1 file(s) remaining
Completed 46.5 MiB/22.3 GiB (54.1 MiB/s) with 1 file(s) remaining
Completed 46.8 MiB/22.3 GiB (54.2 MiB/s) with 1 file(s) remaining
Completed 47.0 MiB/22.3 GiB (54.4 MiB/s) with 1 file(s) remaining
Completed 47.2 MiB/22.3 GiB (54.7 MiB/s) with 1 file(s) remaining
Completed 47.5 MiB/22.3 GiB (55.0 MiB/s) with 1 file(s) remaining
Completed 47.8 MiB/22.3 GiB (55.2 MiB/s) with 1 file(s) remaining
Completed 48.0 MiB/22.3 GiB (55.4 MiB/s) with 1 file(s) remaining
Completed 48.2 MiB/22.3 GiB (55.6 MiB/s) with 1 file(s) remaining
Completed 48.5 MiB/22.3 GiB (55.9 MiB/s) with 1 file(s) remaining
Completed 48.8 MiB/22.3 GiB (56.2 MiB/s) with 1 file(s) remaining
Completed 49.0 MiB/22.3 GiB (56.4 MiB/s) with 1 file(s) remaining
Completed 49.2 MiB/22.3 GiB (56.5 MiB/s) with 1 file(s) remaining
Completed 49.5 MiB/22.3 GiB (56.8 MiB/s) with 1 file(s) remaining
Completed 49.8 MiB/22.3 GiB (57.0 MiB/s) with 1 file(s) remaining
Completed 50.0 MiB/22.3 GiB (57.2 MiB/s) with 1 file(s) remaining
Completed 50.2 MiB/22.3 GiB (57.5 MiB/s) with 1 file(s) remaining
Completed 50.5 MiB/22.3 GiB (57.7 MiB/s) with 1 file(s) remaining
Completed 50.8 MiB/22.3 GiB (57.9 MiB/s) with 1 file(s) remaining
Completed 51.0 MiB/22.3 GiB (58.2 MiB/s) with 1 file(s) remaining
Completed 51.2 MiB/22.3 GiB (58.4 MiB/s) with 1 file(s) remaining
Completed 51.5 MiB/22.3 GiB (58.6 MiB/s) with 1 file(s) remaining
Completed 51.8 MiB/22.3 GiB (58.8 MiB/s) with 1 file(s) remaining
Completed 52.0 MiB/22.3 GiB (58.9 MiB/s) with 1 file(s) remaining
Completed 52.2 MiB/22.3 GiB (59.1 MiB/s) with 1 file(s) remaining
Completed 52.5 MiB/22.3 GiB (59.4 MiB/s) with 1 file(s) remaining
Completed 52.8 MiB/22.3 GiB (59.6 MiB/s) with 1 file(s) remaining
Completed 53.0 MiB/22.3 GiB (59.8 MiB/s) with 1 file(s) remaining
Completed 53.2 MiB/22.3 GiB (59.9 MiB/s) with 1 file(s) remaining
Completed 53.5 MiB/22.3 GiB (60.1 MiB/s) with 1 file(s) remaining
Completed 53.8 MiB/22.3 GiB (60.3 MiB/s) with 1 file(s) remaining
Completed 54.0 MiB/22.3 GiB (60.6 MiB/s) with 1 file(s) remaining
Completed 54.2 MiB/22.3 GiB (60.8 MiB/s) with 1 file(s) remaining
Completed 54.5 MiB/22.3 GiB (61.0 MiB/s) with 1 file(s) remaining
Completed 54.8 MiB/22.3 GiB (61.3 MiB/s) with 1 file(s) remaining
Completed 55.0 MiB/22.3 GiB (61.5 MiB/s) with 1 file(s) remaining
Completed 55.2 MiB/22.3 GiB (61.6 MiB/s) with 1 file(s) remaining
Completed 55.5 MiB/22.3 GiB (61.8 MiB/s) with 1 file(s) remaining
Completed 55.8 MiB/22.3 GiB (62.0 MiB/s) with 1 file(s) remaining
Completed 56.0 MiB/22.3 GiB (62.2 MiB/s) with 1 file(s) remaining
Completed 56.2 MiB/22.3 GiB (62.5 MiB/s) with 1 file(s) remaining
Completed 56.5 MiB/22.3 GiB (62.7 MiB/s) with 1 file(s) remaining
Completed 56.8 MiB/22.3 GiB (62.8 MiB/s) with 1 file(s) remaining
Completed 57.0 MiB/22.3 GiB (62.9 MiB/s) with 1 file(s) remaining
Completed 57.2 MiB/22.3 GiB (63.2 MiB/s) with 1 file(s) remaining
Completed 57.5 MiB/22.3 GiB (63.4 MiB/s) with 1 file(s) remaining
Completed 57.8 MiB/22.3 GiB (63.7 MiB/s) with 1 file(s) remaining
Completed 58.0 MiB/22.3 GiB (63.9 MiB/s) with 1 file(s) remaining
Completed 58.2 MiB/22.3 GiB (63.9 MiB/s) with 1 file(s) remaining
Completed 58.5 MiB/22.3 GiB (64.1 MiB/s) with 1 file(s) remaining
Completed 58.8 MiB/22.3 GiB (64.3 MiB/s) with 1 file(s) remaining
Completed 59.0 MiB/22.3 GiB (64.6 MiB/s) with 1 file(s) remaining
Completed 59.2 MiB/22.3 GiB (64.8 MiB/s) with 1 file(s) remaining
Completed 59.5 MiB/22.3 GiB (65.0 MiB/s) with 1 file(s) remaining
Completed 59.8 MiB/22.3 GiB (65.2 MiB/s) with 1 file(s) remaining
Completed 60.0 MiB/22.3 GiB (65.5 MiB/s) with 1 file(s) remaining
Completed 60.2 MiB/22.3 GiB (65.7 MiB/s) with 1 file(s) remaining
Completed 60.5 MiB/22.3 GiB (65.9 MiB/s) with 1 file(s) remaining
Completed 60.8 MiB/22.3 GiB (66.2 MiB/s) with 1 file(s) remaining
Completed 61.0 MiB/22.3 GiB (66.4 MiB/s) with 1 file(s) remaining
Completed 61.2 MiB/22.3 GiB (66.6 MiB/s) with 1 file(s) remaining
Completed 61.5 MiB/22.3 GiB (66.9 MiB/s) with 1 file(s) remaining
Completed 61.8 MiB/22.3 GiB (67.1 MiB/s) with 1 file(s) remaining
Completed 62.0 MiB/22.3 GiB (67.3 MiB/s) with 1 file(s) remaining
Completed 62.2 MiB/22.3 GiB (67.5 MiB/s) with 1 file(s) remaining
Completed 62.5 MiB/22.3 GiB (67.8 MiB/s) with 1 file(s) remaining
Completed 62.8 MiB/22.3 GiB (67.9 MiB/s) with 1 file(s) remaining
Completed 63.0 MiB/22.3 GiB (67.9 MiB/s) with 1 file(s) remaining
Completed 63.2 MiB/22.3 GiB (68.1 MiB/s) with 1 file(s) remaining
Completed 63.5 MiB/22.3 GiB (68.4 MiB/s) with 1 file(s) remaining
Completed 63.8 MiB/22.3 GiB (68.6 MiB/s) with 1 file(s) remaining
Completed 64.0 MiB/22.3 GiB (68.8 MiB/s) with 1 file(s) remaining
Completed 64.2 MiB/22.3 GiB (69.0 MiB/s) with 1 file(s) remaining
Completed 64.5 MiB/22.3 GiB (69.2 MiB/s) with 1 file(s) remaining
Completed 64.8 MiB/22.3 GiB (69.4 MiB/s) with 1 file(s) remaining
Completed 65.0 MiB/22.3 GiB (69.6 MiB/s) with 1 file(s) remaining
Completed 65.2 MiB/22.3 GiB (69.8 MiB/s) with 1 file(s) remaining
Completed 65.5 MiB/22.3 GiB (70.1 MiB/s) with 1 file(s) remaining
Completed 65.8 MiB/22.3 GiB (70.3 MiB/s) with 1 file(s) remaining
Completed 66.0 MiB/22.3 GiB (70.5 MiB/s) with 1 file(s) remaining
Completed 66.2 MiB/22.3 GiB (70.7 MiB/s) with 1 file(s) remaining
Completed 66.5 MiB/22.3 GiB (70.9 MiB/s) with 1 file(s) remaining
Completed 66.8 MiB/22.3 GiB (71.0 MiB/s) with 1 file(s) remaining
Completed 67.0 MiB/22.3 GiB (71.2 MiB/s) with 1 file(s) remaining
Completed 67.2 MiB/22.3 GiB (71.4 MiB/s) with 1 file(s) remaining
Completed 67.5 MiB/22.3 GiB (71.6 MiB/s) with 1 file(s) remaining
Completed 67.8 MiB/22.3 GiB (71.9 MiB/s) with 1 file(s) remaining
Completed 68.0 MiB/22.3 GiB (72.1 MiB/s) with 1 file(s) remaining
Completed 68.2 MiB/22.3 GiB (72.2 MiB/s) with 1 file(s) remaining
Completed 68.5 MiB/22.3 GiB (72.3 MiB/s) with 1 file(s) remaining
Completed 68.8 MiB/22.3 GiB (72.6 MiB/s) with 1 file(s) remaining
Completed 69.0 MiB/22.3 GiB (72.8 MiB/s) with 1 file(s) remaining
Completed 69.2 MiB/22.3 GiB (73.0 MiB/s) with 1 file(s) remaining
Completed 69.5 MiB/22.3 GiB (73.2 MiB/s) with 1 file(s) remaining
Completed 69.8 MiB/22.3 GiB (73.3 MiB/s) with 1 file(s) remaining
Completed 70.0 MiB/22.3 GiB (73.5 MiB/s) with 1 file(s) remaining
Completed 70.2 MiB/22.3 GiB (73.7 MiB/s) with 1 file(s) remaining
Completed 70.5 MiB/22.3 GiB (73.9 MiB/s) with 1 file(s) remaining
Completed 70.8 MiB/22.3 GiB (74.1 MiB/s) with 1 file(s) remaining
Completed 71.0 MiB/22.3 GiB (74.3 MiB/s) with 1 file(s) remaining
Completed 71.2 MiB/22.3 GiB (74.5 MiB/s) with 1 file(s) remaining
Completed 71.5 MiB/22.3 GiB (74.7 MiB/s) with 1 file(s) remaining
Completed 71.8 MiB/22.3 GiB (74.9 MiB/s) with 1 file(s) remaining
Completed 72.0 MiB/22.3 GiB (75.1 MiB/s) with 1 file(s) remaining
Completed 72.2 MiB/22.3 GiB (75.2 MiB/s) with 1 file(s) remaining
Completed 72.5 MiB/22.3 GiB (75.4 MiB/s) with 1 file(s) remaining
Completed 72.8 MiB/22.3 GiB (75.6 MiB/s) with 1 file(s) remaining
Completed 73.0 MiB/22.3 GiB (75.9 MiB/s) with 1 file(s) remaining
Completed 73.2 MiB/22.3 GiB (76.1 MiB/s) with 1 file(s) remaining
Completed 73.5 MiB/22.3 GiB (76.3 MiB/s) with 1 file(s) remaining
Completed 73.8 MiB/22.3 GiB (76.5 MiB/s) with 1 file(s) remaining
Completed 74.0 MiB/22.3 GiB (76.7 MiB/s) with 1 file(s) remaining
Completed 74.2 MiB/22.3 GiB (76.9 MiB/s) with 1 file(s) remaining
Completed 74.5 MiB/22.3 GiB (77.1 MiB/s) with 1 file(s) remaining
Completed 74.8 MiB/22.3 GiB (77.3 MiB/s) with 1 file(s) remaining
Completed 75.0 MiB/22.3 GiB (77.5 MiB/s) with 1 file(s) remaining
Completed 75.2 MiB/22.3 GiB (77.7 MiB/s) with 1 file(s) remaining
Completed 75.5 MiB/22.3 GiB (77.9 MiB/s) with 1 file(s) remaining
Completed 75.8 MiB/22.3 GiB (78.2 MiB/s) with 1 file(s) remaining
Completed 76.0 MiB/22.3 GiB (78.4 MiB/s) with 1 file(s) remaining
Completed 76.2 MiB/22.3 GiB (78.6 MiB/s) with 1 file(s) remaining
Completed 76.5 MiB/22.3 GiB (78.8 MiB/s) with 1 file(s) remaining
Completed 76.8 MiB/22.3 GiB (79.0 MiB/s) with 1 file(s) remaining
Completed 77.0 MiB/22.3 GiB (79.2 MiB/s) with 1 file(s) remaining
Completed 77.2 MiB/22.3 GiB (79.4 MiB/s) with 1 file(s) remaining
Completed 77.5 MiB/22.3 GiB (79.6 MiB/s) with 1 file(s) remaining
Completed 77.8 MiB/22.3 GiB (79.8 MiB/s) with 1 file(s) remaining
Completed 78.0 MiB/22.3 GiB (80.0 MiB/s) with 1 file(s) remaining
Completed 78.2 MiB/22.3 GiB (80.2 MiB/s) with 1 file(s) remaining
Completed 78.5 MiB/22.3 GiB (80.4 MiB/s) with 1 file(s) remaining
Completed 78.8 MiB/22.3 GiB (80.3 MiB/s) with 1 file(s) remaining
Completed 79.0 MiB/22.3 GiB (80.5 MiB/s) with 1 file(s) remaining
Completed 79.2 MiB/22.3 GiB (80.7 MiB/s) with 1 file(s) remaining
Completed 79.5 MiB/22.3 GiB (80.8 MiB/s) with 1 file(s) remaining
Completed 79.8 MiB/22.3 GiB (80.9 MiB/s) with 1 file(s) remaining
Completed 80.0 MiB/22.3 GiB (81.1 MiB/s) with 1 file(s) remaining
Completed 80.2 MiB/22.3 GiB (81.3 MiB/s) with 1 file(s) remaining
Completed 80.5 MiB/22.3 GiB (81.4 MiB/s) with 1 file(s) remaining
Completed 80.8 MiB/22.3 GiB (81.7 MiB/s) with 1 file(s) remaining
Completed 81.0 MiB/22.3 GiB (81.8 MiB/s) with 1 file(s) remaining
Completed 81.2 MiB/22.3 GiB (82.0 MiB/s) with 1 file(s) remaining
Completed 81.5 MiB/22.3 GiB (82.2 MiB/s) with 1 file(s) remaining
Completed 81.8 MiB/22.3 GiB (82.4 MiB/s) with 1 file(s) remaining
Completed 82.0 MiB/22.3 GiB (82.6 MiB/s) with 1 file(s) remaining
Completed 82.2 MiB/22.3 GiB (82.8 MiB/s) with 1 file(s) remaining
Completed 82.5 MiB/22.3 GiB (83.0 MiB/s) with 1 file(s) remaining
Completed 82.8 MiB/22.3 GiB (83.3 MiB/s) with 1 file(s) remaining
Completed 83.0 MiB/22.3 GiB (83.4 MiB/s) with 1 file(s) remaining
Completed 83.2 MiB/22.3 GiB (83.5 MiB/s) with 1 file(s) remaining
Completed 83.5 MiB/22.3 GiB (83.7 MiB/s) with 1 file(s) remaining
Completed 83.8 MiB/22.3 GiB (83.9 MiB/s) with 1 file(s) remaining
Completed 84.0 MiB/22.3 GiB (84.2 MiB/s) with 1 file(s) remaining
Completed 84.2 MiB/22.3 GiB (84.4 MiB/s) with 1 file(s) remaining
Completed 84.5 MiB/22.3 GiB (84.6 MiB/s) with 1 file(s) remaining
Completed 84.8 MiB/22.3 GiB (84.8 MiB/s) with 1 file(s) remaining
Completed 85.0 MiB/22.3 GiB (85.0 MiB/s) with 1 file(s) remaining
Completed 85.2 MiB/22.3 GiB (85.2 MiB/s) with 1 file(s) remaining
Completed 85.5 MiB/22.3 GiB (85.4 MiB/s) with 1 file(s) remaining
Completed 85.8 MiB/22.3 GiB (85.6 MiB/s) with 1 file(s) remaining
Completed 86.0 MiB/22.3 GiB (85.8 MiB/s) with 1 file(s) remaining
Completed 86.2 MiB/22.3 GiB (85.9 MiB/s) with 1 file(s) remaining
Completed 86.5 MiB/22.3 GiB (86.1 MiB/s) with 1 file(s) remaining
Completed 86.8 MiB/22.3 GiB (86.3 MiB/s) with 1 file(s) remaining
Completed 87.0 MiB/22.3 GiB (86.5 MiB/s) with 1 file(s) remaining
Completed 87.2 MiB/22.3 GiB (86.6 MiB/s) with 1 file(s) remaining
Completed 87.5 MiB/22.3 GiB (86.8 MiB/s) with 1 file(s) remaining
Completed 87.8 MiB/22.3 GiB (87.0 MiB/s) with 1 file(s) remaining
Completed 88.0 MiB/22.3 GiB (87.2 MiB/s) with 1 file(s) remaining
Completed 88.2 MiB/22.3 GiB (87.4 MiB/s) with 1 file(s) remaining
Completed 88.5 MiB/22.3 GiB (87.6 MiB/s) with 1 file(s) remaining
Completed 88.8 MiB/22.3 GiB (87.8 MiB/s) with 1 file(s) remaining
Completed 89.0 MiB/22.3 GiB (87.9 MiB/s) with 1 file(s) remaining
Completed 89.2 MiB/22.3 GiB (88.1 MiB/s) with 1 file(s) remaining
Completed 89.5 MiB/22.3 GiB (88.1 MiB/s) with 1 file(s) remaining
Completed 89.8 MiB/22.3 GiB (88.3 MiB/s) with 1 file(s) remaining
Completed 90.0 MiB/22.3 GiB (88.5 MiB/s) with 1 file(s) remaining
Completed 90.2 MiB/22.3 GiB (88.7 MiB/s) with 1 file(s) remaining
Completed 90.5 MiB/22.3 GiB (88.9 MiB/s) with 1 file(s) remaining
Completed 90.8 MiB/22.3 GiB (89.1 MiB/s) with 1 file(s) remaining
Completed 91.0 MiB/22.3 GiB (89.3 MiB/s) with 1 file(s) remaining
Completed 91.2 MiB/22.3 GiB (89.6 MiB/s) with 1 file(s) remaining
Completed 91.5 MiB/22.3 GiB (89.8 MiB/s) with 1 file(s) remaining
Completed 91.8 MiB/22.3 GiB (90.0 MiB/s) with 1 file(s) remaining
Completed 92.0 MiB/22.3 GiB (90.0 MiB/s) with 1 file(s) remaining
Completed 92.2 MiB/22.3 GiB (90.2 MiB/s) with 1 file(s) remaining
Completed 92.5 MiB/22.3 GiB (90.4 MiB/s) with 1 file(s) remaining
Completed 92.8 MiB/22.3 GiB (90.6 MiB/s) with 1 file(s) remaining
Completed 93.0 MiB/22.3 GiB (90.8 MiB/s) with 1 file(s) remaining
Completed 93.2 MiB/22.3 GiB (91.0 MiB/s) with 1 file(s) remaining
Completed 93.5 MiB/22.3 GiB (91.1 MiB/s) with 1 file(s) remaining
Completed 93.8 MiB/22.3 GiB (91.2 MiB/s) with 1 file(s) remaining
Completed 94.0 MiB/22.3 GiB (91.4 MiB/s) with 1 file(s) remaining
Completed 94.2 MiB/22.3 GiB (91.5 MiB/s) with 1 file(s) remaining
Completed 94.5 MiB/22.3 GiB (91.7 MiB/s) with 1 file(s) remaining
Completed 94.8 MiB/22.3 GiB (91.9 MiB/s) with 1 file(s) remaining
Completed 95.0 MiB/22.3 GiB (92.1 MiB/s) with 1 file(s) remaining
Completed 95.2 MiB/22.3 GiB (92.3 MiB/s) with 1 file(s) remaining
Completed 95.5 MiB/22.3 GiB (92.5 MiB/s) with 1 file(s) remaining
Completed 95.8 MiB/22.3 GiB (92.7 MiB/s) with 1 file(s) remaining
Completed 96.0 MiB/22.3 GiB (92.9 MiB/s) with 1 file(s) remaining
Completed 96.2 MiB/22.3 GiB (93.1 MiB/s) with 1 file(s) remaining
Completed 96.5 MiB/22.3 GiB (93.2 MiB/s) with 1 file(s) remaining
Completed 96.8 MiB/22.3 GiB (93.4 MiB/s) with 1 file(s) remaining
Completed 97.0 MiB/22.3 GiB (93.5 MiB/s) with 1 file(s) remaining
Completed 97.2 MiB/22.3 GiB (93.7 MiB/s) with 1 file(s) remaining
Completed 97.5 MiB/22.3 GiB (93.8 MiB/s) with 1 file(s) remaining
Completed 97.8 MiB/22.3 GiB (94.0 MiB/s) with 1 file(s) remaining
Completed 98.0 MiB/22.3 GiB (94.2 MiB/s) with 1 file(s) remaining
Completed 98.2 MiB/22.3 GiB (94.4 MiB/s) with 1 file(s) remaining
Completed 98.5 MiB/22.3 GiB (94.6 MiB/s) with 1 file(s) remaining
Completed 98.8 MiB/22.3 GiB (94.7 MiB/s) with 1 file(s) remaining
Completed 99.0 MiB/22.3 GiB (95.0 MiB/s) with 1 file(s) remaining
Completed 99.2 MiB/22.3 GiB (95.1 MiB/s) with 1 file(s) remaining
Completed 99.5 MiB/22.3 GiB (95.3 MiB/s) with 1 file(s) remaining
Completed 99.8 MiB/22.3 GiB (95.5 MiB/s) with 1 file(s) remaining
Completed 100.0 MiB/22.3 GiB (95.7 MiB/s) with 1 file(s) remaining
Completed 100.2 MiB/22.3 GiB (95.9 MiB/s) with 1 file(s) remaining
Completed 100.5 MiB/22.3 GiB (96.0 MiB/s) with 1 file(s) remaining
Completed 100.8 MiB/22.3 GiB (96.1 MiB/s) with 1 file(s) remaining
Completed 101.0 MiB/22.3 GiB (96.3 MiB/s) with 1 file(s) remaining
Completed 101.2 MiB/22.3 GiB (96.5 MiB/s) with 1 file(s) remaining
Completed 101.5 MiB/22.3 GiB (96.6 MiB/s) with 1 file(s) remaining
Completed 101.8 MiB/22.3 GiB (96.9 MiB/s) with 1 file(s) remaining
Completed 102.0 MiB/22.3 GiB (97.0 MiB/s) with 1 file(s) remaining
Completed 102.2 MiB/22.3 GiB (97.2 MiB/s) with 1 file(s) remaining
Completed 102.5 MiB/22.3 GiB (97.4 MiB/s) with 1 file(s) remaining
Completed 102.8 MiB/22.3 GiB (97.5 MiB/s) with 1 file(s) remaining
Completed 103.0 MiB/22.3 GiB (97.7 MiB/s) with 1 file(s) remaining
Completed 103.2 MiB/22.3 GiB (97.9 MiB/s) with 1 file(s) remaining
Completed 103.5 MiB/22.3 GiB (98.1 MiB/s) with 1 file(s) remaining
Completed 103.8 MiB/22.3 GiB (98.3 MiB/s) with 1 file(s) remaining
Completed 104.0 MiB/22.3 GiB (98.4 MiB/s) with 1 file(s) remaining
Completed 104.2 MiB/22.3 GiB (98.6 MiB/s) with 1 file(s) remaining
Completed 104.5 MiB/22.3 GiB (98.8 MiB/s) with 1 file(s) remaining
Completed 104.8 MiB/22.3 GiB (99.0 MiB/s) with 1 file(s) remaining
Completed 105.0 MiB/22.3 GiB (99.1 MiB/s) with 1 file(s) remaining
Completed 105.2 MiB/22.3 GiB (99.1 MiB/s) with 1 file(s) remaining
Completed 105.5 MiB/22.3 GiB (99.3 MiB/s) with 1 file(s) remaining
Completed 105.8 MiB/22.3 GiB (99.5 MiB/s) with 1 file(s) remaining
Completed 106.0 MiB/22.3 GiB (99.6 MiB/s) with 1 file(s) remaining
Completed 106.2 MiB/22.3 GiB (99.8 MiB/s) with 1 file(s) remaining
Completed 106.5 MiB/22.3 GiB (100.0 MiB/s) with 1 file(s) remaining
Completed 106.8 MiB/22.3 GiB (100.1 MiB/s) with 1 file(s) remaining
Completed 107.0 MiB/22.3 GiB (100.3 MiB/s) with 1 file(s) remaining
Completed 107.2 MiB/22.3 GiB (100.5 MiB/s) with 1 file(s) remaining
Completed 107.5 MiB/22.3 GiB (100.7 MiB/s) with 1 file(s) remaining
Completed 107.8 MiB/22.3 GiB (100.9 MiB/s) with 1 file(s) remaining
Completed 108.0 MiB/22.3 GiB (101.0 MiB/s) with 1 file(s) remaining
Completed 108.2 MiB/22.3 GiB (101.1 MiB/s) with 1 file(s) remaining
Completed 108.5 MiB/22.3 GiB (101.3 MiB/s) with 1 file(s) remaining
Completed 108.8 MiB/22.3 GiB (101.5 MiB/s) with 1 file(s) remaining
Completed 109.0 MiB/22.3 GiB (101.7 MiB/s) with 1 file(s) remaining
Completed 109.2 MiB/22.3 GiB (102.0 MiB/s) with 1 file(s) remaining
Completed 109.5 MiB/22.3 GiB (102.1 MiB/s) with 1 file(s) remaining
Comp


Downloading at increasing speed. Let me check after a while.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check model download


-rw-rw-r-- 1 ubuntu ubuntu 6.7G Mar 30 16:53 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 9.9G Mar 30 16:53 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 13G Mar 30 16:53 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


13G of 24G. About 4 more minutes for download.


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 16G Mar 30 16:54 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 19G Mar 30 16:54 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors* 2>&1

Check progress


-rw-rw-r-- 1 ubuntu ubuntu 22G Mar 30 16:54 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors.d5F3D2d4


Almost done. Let me wait for the benchmark to start.


grep -c "processed\|DONE\|BENCHMARK\|DASHBOARD" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1; tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1

Check pipeline status