<persisted-output>
Output too large (6MB). Full output saved to: /home/ubuntu/.claude/projects/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tool-results/bhl3jvrs7.txt

Preview (first 2KB):
1
Completed 2.2 KiB/2.2 KiB (8.1 KiB/s) with 1 file(s) remaining
download: s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/processor_config.json to checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json
Completed 256.0 KiB/22.3 GiB (552.5 KiB/s) with 1 file(s) remaining
Completed 512.0 KiB/22.3 GiB (1.0 MiB/s) with 1 file(s) remaining
Completed 768.0 KiB/22.3 GiB (1.4 MiB/s) with 1 file(s) remaining
Completed 1.0 MiB/22.3 GiB (1.9 MiB/s) with 1 file(s) remaining
Completed 1.2 MiB/22.3 GiB (2.3 MiB/s) with 1 file(s) remaining
Completed 1.5 MiB/22.3 GiB (2.8 MiB/s) with 1 file(s) remaining
Completed 1.8 MiB/22.3 GiB (3.1 MiB/s) with 1 file(s) remaining
Completed 2.0 MiB/22.3 GiB (3.5 MiB/s) with 1 file(s) remaining
Completed 2.2 MiB/22.3 GiB (3.9 MiB/s) with 1 file(s) remaining
Completed 2.5 MiB/22.3 GiB (4.3 MiB/s) with 1 file(s) remaining
Completed 2.8 MiB/22.3 GiB (4.7 MiB/s) with 1 file(s) remaining
Completed 3.0 MiB/22.3 GiB (5.1 MiB/s) with 1 file(s) remaining
Completed 3.2 MiB/22.3 GiB (5.5 MiB/s) with 1 file(s) remaining
Completed 3.5 MiB/22.3 GiB (5.9 MiB/s) with 1 file(s) remaining
Completed 3.8 MiB/22.3 GiB (6.4 MiB/s) with 1 file(s) remaining
Completed 4.0 MiB/22.3 GiB (6.6 MiB/s) with 1 file(s) remaining
Completed 4.2 MiB/22.3 GiB (7.0 MiB/s) with 1 file(s) remaining
Completed 4.5 MiB/22.3 GiB (7.4 MiB/s) with 1 file(s) remaining
Completed 4.8 MiB/22.3 GiB (7.8 MiB/s) with 1 file(s) remaining
Completed 5.0 MiB/22.3 GiB (8.1 MiB/s) with 1 file(s) remaining
Completed 5.2 MiB/22.3 GiB (8.5 MiB/s) with 1 file(s) remaining
Completed 5.5 MiB/22.3 GiB (8.9 MiB/s) with 1 file(s) remaining
Completed 5.8 MiB/22.3 GiB (9.3 MiB/s) with 1 file(s) remaining
Completed 6.0 MiB/22.3 GiB (9.6 MiB/s) with 1 file(s) remaining
Completed 6.2 MiB/22.3 GiB (10.0 MiB/s) with 1 file(s) remaining
Completed 6.5 MiB/22.3 GiB (10.4 MiB/s) with 1 file(s) remaining
Completed 6.8 MiB/
...
</persisted-output>

1
Completed 2.2 KiB/2.2 KiB (8.1 KiB/s) with 1 file(s) remaining
download: s3://ptcheckpoints/gemma3n-E2B-asr/03-28-2026/ckpt-70000/processor_config.json to checkpoints/gemma3n-e2b-ckpt-70000/processor_config.json
Completed 256.0 KiB/22.3 GiB (552.5 KiB/s) with 1 file(s) remaining
Completed 512.0 KiB/22.3 GiB (1.0 MiB/s) with 1 file(s) remaining
Completed 768.0 KiB/22.3 GiB (1.4 MiB/s) with 1 file(s) remaining
Completed 1.0 MiB/22.3 GiB (1.9 MiB/s) with 1 file(s) remaining
Completed 1.2 MiB/22.3 GiB (2.3 MiB/s) with 1 file(s) remaining
Completed 1.5 MiB/22.3 GiB (2.8 MiB/s) with 1 file(s) remaining
Completed 1.8 MiB/22.3 GiB (3.1 MiB/s) with 1 file(s) remaining
Completed 2.0 MiB/22.3 GiB (3.5 MiB/s) with 1 file(s) remaining
Completed 2.2 MiB/22.3 GiB (3.9 MiB/s) with 1 file(s) remaining
Completed 2.5 MiB/22.3 GiB (4.3 MiB/s) with 1 file(s) remaining
Completed 2.8 MiB/22.3 GiB (4.7 MiB/s) with 1 file(s) remaining
Completed 3.0 MiB/22.3 GiB (5.1 MiB/s) with 1 file(s) remaining
Completed 3.2 MiB/22.3 GiB (5.5 MiB/s) with 1 file(s) remaining
Completed 3.5 MiB/22.3 GiB (5.9 MiB/s) with 1 file(s) remaining
Completed 3.8 MiB/22.3 GiB (6.4 MiB/s) with 1 file(s) remaining
Completed 4.0 MiB/22.3 GiB (6.6 MiB/s) with 1 file(s) remaining
Completed 4.2 MiB/22.3 GiB (7.0 MiB/s) with 1 file(s) remaining
Completed 4.5 MiB/22.3 GiB (7.4 MiB/s) with 1 file(s) remaining
Completed 4.8 MiB/22.3 GiB (7.8 MiB/s) with 1 file(s) remaining
Completed 5.0 MiB/22.3 GiB (8.1 MiB/s) with 1 file(s) remaining
Completed 5.2 MiB/22.3 GiB (8.5 MiB/s) with 1 file(s) remaining
Completed 5.5 MiB/22.3 GiB (8.9 MiB/s) with 1 file(s) remaining
Completed 5.8 MiB/22.3 GiB (9.3 MiB/s) with 1 file(s) remaining
Completed 6.0 MiB/22.3 GiB (9.6 MiB/s) with 1 file(s) remaining
Completed 6.2 MiB/22.3 GiB (10.0 MiB/s) with 1 file(s) remaining
Completed 6.5 MiB/22.3 GiB (10.4 MiB/s) with 1 file(s) remaining
Completed 6.8 MiB/22.3 GiB (10.8 MiB/s) with 1 file(s) remaining
Completed 7.0 MiB/22.3 GiB (11.2 MiB/s) with 1 file(s) remaining
Completed 7.2 MiB/22.3 GiB (11.6 MiB/s) with 1 file(s) remaining
Completed 7.5 MiB/22.3 GiB (12.0 MiB/s) with 1 file(s) remaining
Completed 7.8 MiB/22.3 GiB (12.3 MiB/s) with 1 file(s) remaining
Completed 8.0 MiB/22.3 GiB (12.7 MiB/s) with 1 file(s) remaining
Completed 8.2 MiB/22.3 GiB (13.1 MiB/s) with 1 file(s) remaining
Completed 8.5 MiB/22.3 GiB (13.4 MiB/s) with 1 file(s) remaining
Completed 8.8 MiB/22.3 GiB (13.8 MiB/s) with 1 file(s) remaining
Completed 9.0 MiB/22.3 GiB (14.2 MiB/s) with 1 file(s) remaining
Completed 9.2 MiB/22.3 GiB (14.5 MiB/s) with 1 file(s) remaining
Completed 9.5 MiB/22.3 GiB (14.8 MiB/s) with 1 file(s) remaining
Completed 9.8 MiB/22.3 GiB (15.2 MiB/s) with 1 file(s) remaining
Completed 10.0 MiB/22.3 GiB (15.6 MiB/s) with 1 file(s) remaining
Completed 10.2 MiB/22.3 GiB (15.9 MiB/s) with 1 file(s) remaining
Completed 10.5 MiB/22.3 GiB (16.3 MiB/s) with 1 file(s) remaining
Completed 10.8 MiB/22.3 GiB (16.6 MiB/s) with 1 file(s) remaining
Completed 11.0 MiB/22.3 GiB (17.0 MiB/s) with 1 file(s) remaining
Completed 11.2 MiB/22.3 GiB (17.3 MiB/s) with 1 file(s) remaining
Completed 11.5 MiB/22.3 GiB (17.6 MiB/s) with 1 file(s) remaining
Completed 11.8 MiB/22.3 GiB (18.0 MiB/s) with 1 file(s) remaining
Completed 12.0 MiB/22.3 GiB (18.3 MiB/s) with 1 file(s) remaining
Completed 12.2 MiB/22.3 GiB (18.7 MiB/s) with 1 file(s) remaining
Completed 12.5 MiB/22.3 GiB (19.0 MiB/s) with 1 file(s) remaining
Completed 12.8 MiB/22.3 GiB (19.4 MiB/s) with 1 file(s) remaining
Completed 13.0 MiB/22.3 GiB (19.7 MiB/s) with 1 file(s) remaining
Completed 13.2 MiB/22.3 GiB (20.1 MiB/s) with 1 file(s) remaining
Completed 13.5 MiB/22.3 GiB (20.3 MiB/s) with 1 file(s) remaining
Completed 13.8 MiB/22.3 GiB (20.6 MiB/s) with 1 file(s) remaining
Completed 14.0 MiB/22.3 GiB (21.0 MiB/s) with 1 file(s) remaining
Completed 14.2 MiB/22.3 GiB (21.3 MiB/s) with 1 file(s) remaining
Completed 14.5 MiB/22.3 GiB (21.6 MiB/s) with 1 file(s) remaining
Completed 14.8 MiB/22.3 GiB (21.9 MiB/s) with 1 file(s) remaining
Completed 15.0 MiB/22.3 GiB (22.2 MiB/s) with 1 file(s) remaining
Completed 15.2 MiB/22.3 GiB (22.5 MiB/s) with 1 file(s) remaining
Completed 15.5 MiB/22.3 GiB (22.9 MiB/s) with 1 file(s) remaining
Completed 15.8 MiB/22.3 GiB (23.1 MiB/s) with 1 file(s) remaining
Completed 16.0 MiB/22.3 GiB (23.4 MiB/s) with 1 file(s) remaining
Completed 16.2 MiB/22.3 GiB (23.7 MiB/s) with 1 file(s) remaining
Completed 16.5 MiB/22.3 GiB (24.0 MiB/s) with 1 file(s) remaining
Completed 16.8 MiB/22.3 GiB (24.3 MiB/s) with 1 file(s) remaining
Completed 17.0 MiB/22.3 GiB (24.7 MiB/s) with 1 file(s) remaining
Completed 17.2 MiB/22.3 GiB (25.0 MiB/s) with 1 file(s) remaining
Completed 17.5 MiB/22.3 GiB (25.2 MiB/s) with 1 file(s) remaining
Completed 17.8 MiB/22.3 GiB (25.6 MiB/s) with 1 file(s) remaining
Completed 18.0 MiB/22.3 GiB (25.9 MiB/s) with 1 file(s) remaining
Completed 18.2 MiB/22.3 GiB (26.2 MiB/s) with 1 file(s) remaining
Completed 18.5 MiB/22.3 GiB (26.6 MiB/s) with 1 file(s) remaining
Completed 18.8 MiB/22.3 GiB (26.9 MiB/s) with 1 file(s) remaining
Completed 19.0 MiB/22.3 GiB (27.2 MiB/s) with 1 file(s) remaining
Completed 19.2 MiB/22.3 GiB (27.5 MiB/s) with 1 file(s) remaining
Completed 19.5 MiB/22.3 GiB (27.9 MiB/s) with 1 file(s) remaining
Completed 19.8 MiB/22.3 GiB (28.2 MiB/s) with 1 file(s) remaining
Completed 20.0 MiB/22.3 GiB (28.5 MiB/s) with 1 file(s) remaining
Completed 20.2 MiB/22.3 GiB (28.5 MiB/s) with 1 file(s) remaining
Completed 20.5 MiB/22.3 GiB (28.8 MiB/s) with 1 file(s) remaining
Completed 20.8 MiB/22.3 GiB (29.0 MiB/s) with 1 file(s) remaining
Completed 21.0 MiB/22.3 GiB (29.1 MiB/s) with 1 file(s) remaining
Completed 21.2 MiB/22.3 GiB (29.4 MiB/s) with 1 file(s) remaining
Completed 21.5 MiB/22.3 GiB (29.7 MiB/s) with 1 file(s) remaining
Completed 21.8 MiB/22.3 GiB (29.9 MiB/s) with 1 file(s) remaining
Completed 22.0 MiB/22.3 GiB (30.2 MiB/s) with 1 file(s) remaining
Completed 22.2 MiB/22.3 GiB (30.5 MiB/s) with 1 file(s) remaining
Completed 22.5 MiB/22.3 GiB (30.8 MiB/s) with 1 file(s) remaining
Completed 22.8 MiB/22.3 GiB (31.1 MiB/s) with 1 file(s) remaining
Completed 23.0 MiB/22.3 GiB (31.3 MiB/s) with 1 file(s) remaining
Completed 23.2 MiB/22.3 GiB (31.6 MiB/s) with 1 file(s) remaining
Completed 23.5 MiB/22.3 GiB (31.9 MiB/s) with 1 file(s) remaining
Completed 23.8 MiB/22.3 GiB (32.3 MiB/s) with 1 file(s) remaining
Completed 24.0 MiB/22.3 GiB (32.5 MiB/s) with 1 file(s) remaining
Completed 24.2 MiB/22.3 GiB (32.8 MiB/s) with 1 file(s) remaining
Completed 24.5 MiB/22.3 GiB (33.0 MiB/s) with 1 file(s) remaining
Completed 24.8 MiB/22.3 GiB (33.3 MiB/s) with 1 file(s) remaining
Completed 25.0 MiB/22.3 GiB (33.5 MiB/s) with 1 file(s) remaining
Completed 25.2 MiB/22.3 GiB (33.9 MiB/s) with 1 file(s) remaining
Completed 25.5 MiB/22.3 GiB (33.7 MiB/s) with 1 file(s) remaining
Completed 25.8 MiB/22.3 GiB (34.0 MiB/s) with 1 file(s) remaining
Completed 26.0 MiB/22.3 GiB (34.3 MiB/s) with 1 file(s) remaining
Completed 26.2 MiB/22.3 GiB (34.6 MiB/s) with 1 file(s) remaining
Completed 26.5 MiB/22.3 GiB (34.9 MiB/s) with 1 file(s) remaining
Completed 26.8 MiB/22.3 GiB (35.2 MiB/s) with 1 file(s) remaining
Completed 27.0 MiB/22.3 GiB (35.4 MiB/s) with 1 file(s) remaining
Completed 27.2 MiB/22.3 GiB (35.5 MiB/s) with 1 file(s) remaining
Completed 27.5 MiB/22.3 GiB (35.8 MiB/s) with 1 file(s) remaining
Completed 27.8 MiB/22.3 GiB (36.1 MiB/s) with 1 file(s) remaining
Completed 28.0 MiB/22.3 GiB (36.3 MiB/s) with 1 file(s) remaining
Completed 28.2 MiB/22.3 GiB (36.7 MiB/s) with 1 file(s) remaining
Completed 28.5 MiB/22.3 GiB (37.0 MiB/s) with 1 file(s) remaining
Completed 28.8 MiB/22.3 GiB (37.2 MiB/s) with 1 file(s) remaining
Completed 29.0 MiB/22.3 GiB (37.4 MiB/s) with 1 file(s) remaining
Completed 29.2 MiB/22.3 GiB (37.7 MiB/s) with 1 file(s) remaining
Completed 29.5 MiB/22.3 GiB (38.0 MiB/s) with 1 file(s) remaining
Completed 29.8 MiB/22.3 GiB (38.3 MiB/s) with 1 file(s) remaining
Completed 30.0 MiB/22.3 GiB (38.6 MiB/s) with 1 file(s) remaining
Completed 30.2 MiB/22.3 GiB (38.8 MiB/s) with 1 file(s) remaining
Completed 30.5 MiB/22.3 GiB (39.1 MiB/s) with 1 file(s) remaining
Completed 30.8 MiB/22.3 GiB (39.2 MiB/s) with 1 file(s) remaining
Completed 31.0 MiB/22.3 GiB (39.5 MiB/s) with 1 file(s) remaining
Completed 31.2 MiB/22.3 GiB (39.7 MiB/s) with 1 file(s) remaining
Completed 31.5 MiB/22.3 GiB (40.0 MiB/s) with 1 file(s) remaining
Completed 31.8 MiB/22.3 GiB (40.3 MiB/s) with 1 file(s) remaining
Completed 32.0 MiB/22.3 GiB (40.6 MiB/s) with 1 file(s) remaining
Completed 32.2 MiB/22.3 GiB (40.8 MiB/s) with 1 file(s) remaining
Completed 32.5 MiB/22.3 GiB (41.1 MiB/s) with 1 file(s) remaining
Completed 32.8 MiB/22.3 GiB (41.4 MiB/s) with 1 file(s) remaining
Completed 33.0 MiB/22.3 GiB (41.6 MiB/s) with 1 file(s) remaining
Completed 33.2 MiB/22.3 GiB (41.8 MiB/s) with 1 file(s) remaining
Completed 33.5 MiB/22.3 GiB (42.1 MiB/s) with 1 file(s) remaining
Completed 33.8 MiB/22.3 GiB (42.3 MiB/s) with 1 file(s) remaining
Completed 34.0 MiB/22.3 GiB (42.6 MiB/s) with 1 file(s) remaining
Completed 34.2 MiB/22.3 GiB (42.9 MiB/s) with 1 file(s) remaining
Completed 34.5 MiB/22.3 GiB (43.2 MiB/s) with 1 file(s) remaining
Completed 34.8 MiB/22.3 GiB (43.5 MiB/s) with 1 file(s) remaining
Completed 35.0 MiB/22.3 GiB (43.6 MiB/s) with 1 file(s) remaining
Completed 35.2 MiB/22.3 GiB (43.8 MiB/s) with 1 file(s) remaining
Completed 35.5 MiB/22.3 GiB (43.9 MiB/s) with 1 file(s) remaining
Completed 35.8 MiB/22.3 GiB (44.1 MiB/s) with 1 file(s) remaining
Completed 36.0 MiB/22.3 GiB (44.4 MiB/s) with 1 file(s) remaining
Completed 36.2 MiB/22.3 GiB (44.7 MiB/s) with 1 file(s) remaining
Completed 36.5 MiB/22.3 GiB (45.0 MiB/s) with 1 file(s) remaining
Completed 36.8 MiB/22.3 GiB (45.2 MiB/s) with 1 file(s) remaining
Completed 37.0 MiB/22.3 GiB (45.5 MiB/s) with 1 file(s) remaining
Completed 37.2 MiB/22.3 GiB (45.7 MiB/s) with 1 file(s) remaining
Completed 37.5 MiB/22.3 GiB (45.9 MiB/s) with 1 file(s) remaining
Completed 37.8 MiB/22.3 GiB (46.1 MiB/s) with 1 file(s) remaining
Completed 38.0 MiB/22.3 GiB (46.1 MiB/s) with 1 file(s) remaining
Completed 38.2 MiB/22.3 GiB (46.3 MiB/s) with 1 file(s) remaining
Completed 38.5 MiB/22.3 GiB (46.5 MiB/s) with 1 file(s) remaining
Completed 38.8 MiB/22.3 GiB (46.8 MiB/s) with 1 file(s) remaining
Completed 39.0 MiB/22.3 GiB (47.1 MiB/s) with 1 file(s) remaining
Completed 39.2 MiB/22.3 GiB (47.4 MiB/s) with 1 file(s) remaining
Completed 39.5 MiB/22.3 GiB (47.6 MiB/s) with 1 file(s) remaining
Completed 39.8 MiB/22.3 GiB (47.9 MiB/s) with 1 file(s) remaining
Completed 40.0 MiB/22.3 GiB (48.2 MiB/s) with 1 file(s) remaining
Completed 40.2 MiB/22.3 GiB (48.4 MiB/s) with 1 file(s) remaining
Completed 40.5 MiB/22.3 GiB (48.7 MiB/s) with 1 file(s) remaining
Completed 40.8 MiB/22.3 GiB (49.0 MiB/s) with 1 file(s) remaining
Completed 41.0 MiB/22.3 GiB (49.2 MiB/s) with 1 file(s) remaining
Completed 41.2 MiB/22.3 GiB (49.4 MiB/s) with 1 file(s) remaining
Completed 41.5 MiB/22.3 GiB (49.6 MiB/s) with 1 file(s) remaining
Completed 41.8 MiB/22.3 GiB (49.9 MiB/s) with 1 file(s) remaining
Completed 42.0 MiB/22.3 GiB (50.1 MiB/s) with 1 file(s) remaining
Completed 42.2 MiB/22.3 GiB (50.4 MiB/s) with 1 file(s) remaining
Completed 42.5 MiB/22.3 GiB (50.6 MiB/s) with 1 file(s) remaining
Completed 42.8 MiB/22.3 GiB (50.8 MiB/s) with 1 file(s) remaining
Completed 43.0 MiB/22.3 GiB (51.1 MiB/s) with 1 file(s) remaining
Completed 43.2 MiB/22.3 GiB (51.2 MiB/s) with 1 file(s) remaining
Completed 43.5 MiB/22.3 GiB (51.4 MiB/s) with 1 file(s) remaining
Completed 43.8 MiB/22.3 GiB (51.7 MiB/s) with 1 file(s) remaining
Completed 44.0 MiB/22.3 GiB (52.0 MiB/s) with 1 file(s) remaining
Completed 44.2 MiB/22.3 GiB (52.3 MiB/s) with 1 file(s) remaining
Completed 44.5 MiB/22.3 GiB (52.5 MiB/s) with 1 file(s) remaining
Completed 44.8 MiB/22.3 GiB (52.8 MiB/s) with 1 file(s) remaining
Completed 45.0 MiB/22.3 GiB (53.0 MiB/s) with 1 file(s) remaining
Completed 45.2 MiB/22.3 GiB (53.1 MiB/s) with 1 file(s) remaining
Completed 45.5 MiB/22.3 GiB (53.3 MiB/s) with 1 file(s) remaining
Completed 45.8 MiB/22.3 GiB (53.5 MiB/s) with 1 file(s) remaining
Completed 46.0 MiB/22.3 GiB (53.8 MiB/s) with 1 file(s) remaining
Completed 46.2 MiB/22.3 GiB (54.0 MiB/s) with 1 file(s) remaining
Completed 46.5 MiB/22.3 GiB (54.1 MiB/s) with 1 file(s) remaining
Completed 46.8 MiB/22.3 GiB (54.2 MiB/s) with 1 file(s) remaining
Completed 47.0 MiB/22.3 GiB (54.4 MiB/s) with 1 file(s) remaining
Completed 47.2 MiB/22.3 GiB (54.7 MiB/s) with 1 file(s) remaining
Completed 47.5 MiB/22.3 GiB (55.0 MiB/s) with 1 file(s) remaining
Completed 47.8 MiB/22.3 GiB (55.2 MiB/s) with 1 file(s) remaining
Completed 48.0 MiB/22.3 GiB (55.4 MiB/s) with 1 file(s) remaining
Completed 48.2 MiB/22.3 GiB (55.6 MiB/s) with 1 file(s) remaining
Completed 48.5 MiB/22.3 GiB (55.9 MiB/s) with 1 file(s) remaining
Completed 48.8 MiB/22.3 GiB (56.2 MiB/s) with 1 file(s) remaining
Completed 49.0 MiB/22.3 GiB (56.4 MiB/s) with 1 file(s) remaining
Completed 49.2 MiB/22.3 GiB (56.5 MiB/s) with 1 file(s) remaining
Completed 49.5 MiB/22.3 GiB (56.8 MiB/s) with 1 file(s) remaining
Completed 49.8 MiB/22.3 GiB (57.0 MiB/s) with 1 file(s) remaining
Completed 50.0 MiB/22.3 GiB (57.2 MiB/s) with 1 file(s) remaining
Completed 50.2 MiB/22.3 GiB (57.5 MiB/s) with 1 file(s) remaining
Completed 50.5 MiB/22.3 GiB (57.7 MiB/s) with 1 file(s) remaining
Completed 50.8 MiB/22.3 GiB (57.9 MiB/s) with 1 file(s) remaining
Completed 51.0 MiB/22.3 GiB (58.2 MiB/s) with 1 file(s) remaining
Completed 51.2 MiB/22.3 GiB (58.4 MiB/s) with 1 file(s) remaining
Completed 51.5 MiB/22.3 GiB (58.6 MiB/s) with 1 file(s) remaining
Completed 51.8 MiB/22.3 GiB (58.8 MiB/s) with 1 file(s) remaining
Completed 52.0 MiB/22.3 GiB (58.9 MiB/s) with 1 file(s) remaining
Completed 52.2 MiB/22.3 GiB (59.1 MiB/s) with 1 file(s) remaining
Completed 52.5 MiB/22.3 GiB (59.4 MiB/s) with 1 file(s) remaining
Completed 52.8 MiB/22.3 GiB (59.6 MiB/s) with 1 file(s) remaining
Completed 53.0 MiB/22.3 GiB (59.8 MiB/s) with 1 file(s) remaining
Completed 53.2 MiB/22.3 GiB (59.9 MiB/s) with 1 file(s) remaining
Completed 53.5 MiB/22.3 GiB (60.1 MiB/s) with 1 file(s) remaining
Completed 53.8 MiB/22.3 GiB (60.3 MiB/s) with 1 file(s) remaining
Completed 54.0 MiB/22.3 GiB (60.6 MiB/s) with 1 file(s) remaining
Completed 54.2 MiB/22.3 GiB (60.8 MiB/s) with 1 file(s) remaining
Completed 54.5 MiB/22.3 GiB (61.0 MiB/s) with 1 file(s) remaining
Completed 54.8 MiB/22.3 GiB (61.3 MiB/s) with 1 file(s) remaining
Completed 55.0 MiB/22.3 GiB (61.5 MiB/s) with 1 file(s) remaining
Completed 55.2 MiB/22.3 GiB (61.6 MiB/s) with 1 file(s) remaining
Completed 55.5 MiB/22.3 GiB (61.8 MiB/s) with 1 file(s) remaining
Completed 55.8 MiB/22.3 GiB (62.0 MiB/s) with 1 file(s) remaining
Completed 56.0 MiB/22.3 GiB (62.2 MiB/s) with 1 file(s) remaining
Completed 56.2 MiB/22.3 GiB (62.5 MiB/s) with 1 file(s) remaining
Completed 56.5 MiB/22.3 GiB (62.7 MiB/s) with 1 file(s) remaining
Completed 56.8 MiB/22.3 GiB (62.8 MiB/s) with 1 file(s) remaining
Completed 57.0 MiB/22.3 GiB (62.9 MiB/s) with 1 file(s) remaining
Completed 57.2 MiB/22.3 GiB (63.2 MiB/s) with 1 file(s) remaining
Completed 57.5 MiB/22.3 GiB (63.4 MiB/s) with 1 file(s) remaining
Completed 57.8 MiB/22.3 GiB (63.7 MiB/s) with 1 file(s) remaining
Completed 58.0 MiB/22.3 GiB (63.9 MiB/s) with 1 file(s) remaining
Completed 58.2 MiB/22.3 GiB (63.9 MiB/s) with 1 file(s) remaining
Completed 58.5 MiB/22.3 GiB (64.1 MiB/s) with 1 file(s) remaining
Completed 58.8 MiB/22.3 GiB (64.3 MiB/s) with 1 file(s) remaining
Completed 59.0 MiB/22.3 GiB (64.6 MiB/s) with 1 file(s) remaining
Completed 59.2 MiB/22.3 GiB (64.8 MiB/s) with 1 file(s) remaining
Completed 59.5 MiB/22.3 GiB (65.0 MiB/s) with 1 file(s) remaining
Completed 59.8 MiB/22.3 GiB (65.2 MiB/s) with 1 file(s) remaining
Completed 60.0 MiB/22.3 GiB (65.5 MiB/s) with 1 file(s) remaining
Completed 60.2 MiB/22.3 GiB (65.7 MiB/s) with 1 file(s) remaining
Completed 60.5 MiB/22.3 GiB (65.9 MiB/s) with 1 file(s) remaining
Completed 60.8 MiB/22.3 GiB (66.2 MiB/s) with 1 file(s) remaining
Completed 61.0 MiB/22.3 GiB (66.4 MiB/s) with 1 file(s) remaining
Completed 61.2 MiB/22.3 GiB (66.6 MiB/s) with 1 file(s) remaining
Completed 61.5 MiB/22.3 GiB (66.9 MiB/s) with 1 file(s) remaining
Completed 61.8 MiB/22.3 GiB (67.1 MiB/s) with 1 file(s) remaining
Completed 62.0 MiB/22.3 GiB (67.3 MiB/s) with 1 file(s) remaining
Completed 62.2 MiB/22.3 GiB (67.5 MiB/s) with 1 file(s) remaining
Completed 62.5 MiB/22.3 GiB (67.8 MiB/s) with 1 file(s) remaining
Completed 62.8 MiB/22.3 GiB (67.9 MiB/s) with 1 file(s) remaining
Completed 63.0 MiB/22.3 GiB (67.9 MiB/s) with 1 file(s) remaining
Completed 63.2 MiB/22.3 GiB (68.1 MiB/s) with 1 file(s) remaining
Completed 63.5 MiB/22.3 GiB (68.4 MiB/s) with 1 file(s) remaining
Completed 63.8 MiB/22.3 GiB (68.6 MiB/s) with 1 file(s) remaining
Completed 64.0 MiB/22.3 GiB (68.8 MiB/s) with 1 file(s) remaining
Completed 64.2 MiB/22.3 GiB (69.0 MiB/s) with 1 file(s) remaining
Completed 64.5 MiB/22.3 GiB (69.2 MiB/s) with 1 file(s) remaining
Completed 64.8 MiB/22.3 GiB (69.4 MiB/s) with 1 file(s) remaining
Completed 65.0 MiB/22.3 GiB (69.6 MiB/s) with 1 file(s) remaining
Completed 65.2 MiB/22.3 GiB (69.8 MiB/s) with 1 file(s) remaining
Completed 65.5 MiB/22.3 GiB (70.1 MiB/s) with 1 file(s) remaining
Completed 65.8 MiB/22.3 GiB (70.3 MiB/s) with 1 file(s) remaining
Completed 66.0 MiB/22.3 GiB (70.5 MiB/s) with 1 file(s) remaining
Completed 66.2 MiB/22.3 GiB (70.7 MiB/s) with 1 file(s) remaining
Completed 66.5 MiB/22.3 GiB (70.9 MiB/s) with 1 file(s) remaining
Completed 66.8 MiB/22.3 GiB (71.0 MiB/s) with 1 file(s) remaining
Completed 67.0 MiB/22.3 GiB (71.2 MiB/s) with 1 file(s) remaining
Completed 67.2 MiB/22.3 GiB (71.4 MiB/s) with 1 file(s) remaining
Completed 67.5 MiB/22.3 GiB (71.6 MiB/s) with 1 file(s) remaining
Completed 67.8 MiB/22.3 GiB (71.9 MiB/s) with 1 file(s) remaining
Completed 68.0 MiB/22.3 GiB (72.1 MiB/s) with 1 file(s) remaining
Completed 68.2 MiB/22.3 GiB (72.2 MiB/s) with 1 file(s) remaining
Completed 68.5 MiB/22.3 GiB (72.3 MiB/s) with 1 file(s) remaining
Completed 68.8 MiB/22.3 GiB (72.6 MiB/s) with 1 file(s) remaining
Completed 69.0 MiB/22.3 GiB (72.8 MiB/s) with 1 file(s) remaining
Completed 69.2 MiB/22.3 GiB (73.0 MiB/s) with 1 file(s) remaining
Completed 69.5 MiB/22.3 GiB (73.2 MiB/s) with 1 file(s) remaining
Completed 69.8 MiB/22.3 GiB (73.3 MiB/s) with 1 file(s) remaining
Completed 70.0 MiB/22.3 GiB (73.5 MiB/s) with 1 file(s) remaining
Completed 70.2 MiB/22.3 GiB (73.7 MiB/s) with 1 file(s) remaining
Completed 70.5 MiB/22.3 GiB (73.9 MiB/s) with 1 file(s) remaining
Completed 70.8 MiB/22.3 GiB (74.1 MiB/s) with 1 file(s) remaining
Completed 71.0 MiB/22.3 GiB (74.3 MiB/s) with 1 file(s) remaining
Completed 71.2 MiB/22.3 GiB (74.5 MiB/s) with 1 file(s) remaining
Completed 71.5 MiB/22.3 GiB (74.7 MiB/s) with 1 file(s) remaining
Completed 71.8 MiB/22.3 GiB (74.9 MiB/s) with 1 file(s) remaining
Completed 72.0 MiB/22.3 GiB (75.1 MiB/s) with 1 file(s) remaining
Completed 72.2 MiB/22.3 GiB (75.2 MiB/s) with 1 file(s) remaining
Completed 72.5 MiB/22.3 GiB (75.4 MiB/s) with 1 file(s) remaining
Completed 72.8 MiB/22.3 GiB (75.6 MiB/s) with 1 file(s) remaining
Completed 73.0 MiB/22.3 GiB (75.9 MiB/s) with 1 file(s) remaining
Completed 73.2 MiB/22.3 GiB (76.1 MiB/s) with 1 file(s) remaining
Completed 73.5 MiB/22.3 GiB (76.3 MiB/s) with 1 file(s) remaining
Completed 73.8 MiB/22.3 GiB (76.5 MiB/s) with 1 file(s) remaining
Completed 74.0 MiB/22.3 GiB (76.7 MiB/s) with 1 file(s) remaining
Completed 74.2 MiB/22.3 GiB (76.9 MiB/s) with 1 file(s) remaining
Completed 74.5 MiB/22.3 GiB (77.1 MiB/s) with 1 file(s) remaining
Completed 74.8 MiB/22.3 GiB (77.3 MiB/s) with 1 file(s) remaining
Completed 75.0 MiB/22.3 GiB (77.5 MiB/s) with 1 file(s) remaining
Completed 75.2 MiB/22.3 GiB (77.7 MiB/s) with 1 file(s) remaining
Completed 75.5 MiB/22.3 GiB (77.9 MiB/s) with 1 file(s) remaining
Completed 75.8 MiB/22.3 GiB (78.2 MiB/s) with 1 file(s) remaining
Completed 76.0 MiB/22.3 GiB (78.4 MiB/s) with 1 file(s) remaining
Completed 76.2 MiB/22.3 GiB (78.6 MiB/s) with 1 file(s) remaining
Completed 76.5 MiB/22.3 GiB (78.8 MiB/s) with 1 file(s) remaining
Completed 76.8 MiB/22.3 GiB (79.0 MiB/s) with 1 file(s) remaining
Completed 77.0 MiB/22.3 GiB (79.2 MiB/s) with 1 file(s) remaining
Completed 77.2 MiB/22.3 GiB (79.4 MiB/s) with 1 file(s) remaining
Completed 77.5 MiB/22.3 GiB (79.6 MiB/s) with 1 file(s) remaining
Completed 77.8 MiB/22.3 GiB (79.8 MiB/s) with 1 file(s) remaining
Completed 78.0 MiB/22.3 GiB (80.0 MiB/s) with 1 file(s) remaining
Completed 78.2 MiB/22.3 GiB (80.2 MiB/s) with 1 file(s) remaining
Completed 78.5 MiB/22.3 GiB (80.4 MiB/s) with 1 file(s) remaining
Completed 78.8 MiB/22.3 GiB (80.3 MiB/s) with 1 file(s) remaining
Completed 79.0 MiB/22.3 GiB (80.5 MiB/s) with 1 file(s) remaining
Completed 79.2 MiB/22.3 GiB (80.7 MiB/s) with 1 file(s) remaining
Completed 79.5 MiB/22.3 GiB (80.8 MiB/s) with 1 file(s) remaining
Completed 79.8 MiB/22.3 GiB (80.9 MiB/s) with 1 file(s) remaining
Completed 80.0 MiB/22.3 GiB (81.1 MiB/s) with 1 file(s) remaining
Completed 80.2 MiB/22.3 GiB (81.3 MiB/s) with 1 file(s) remaining
Completed 80.5 MiB/22.3 GiB (81.4 MiB/s) with 1 file(s) remaining
Completed 80.8 MiB/22.3 GiB (81.7 MiB/s) with 1 file(s) remaining
Completed 81.0 MiB/22.3 GiB (81.8 MiB/s) with 1 file(s) remaining
Completed 81.2 MiB/22.3 GiB (82.0 MiB/s) with 1 file(s) remaining
Completed 81.5 MiB/22.3 GiB (82.2 MiB/s) with 1 file(s) remaining
Completed 81.8 MiB/22.3 GiB (82.4 MiB/s) with 1 file(s) remaining
Completed 82.0 MiB/22.3 GiB (82.6 MiB/s) with 1 file(s) remaining
Completed 82.2 MiB/22.3 GiB (82.8 MiB/s) with 1 file(s) remaining
Completed 82.5 MiB/22.3 GiB (83.0 MiB/s) with 1 file(s) remaining
Completed 82.8 MiB/22.3 GiB (83.3 MiB/s) with 1 file(s) remaining
Completed 83.0 MiB/22.3 GiB (83.4 MiB/s) with 1 file(s) remaining
Completed 83.2 MiB/22.3 GiB (83.5 MiB/s) with 1 file(s) remaining
Completed 83.5 MiB/22.3 GiB (83.7 MiB/s) with 1 file(s) remaining
Completed 83.8 MiB/22.3 GiB (83.9 MiB/s) with 1 file(s) remaining
Completed 84.0 MiB/22.3 GiB (84.2 MiB/s) with 1 file(s) remaining
Completed 84.2 MiB/22.3 GiB (84.4 MiB/s) with 1 file(s) remaining
Completed 84.5 MiB/22.3 GiB (84.6 MiB/s) with 1 file(s) remaining
Completed 84.8 MiB/22.3 GiB (84.8 MiB/s) with 1 file(s) remaining
Completed 85.0 MiB/22.3 GiB (85.0 MiB/s) with 1 file(s) remaining
Completed 85.2 MiB/22.3 GiB (85.2 MiB/s) with 1 file(s) remaining
Completed 85.5 MiB/22.3 GiB (85.4 MiB/s) with 1 file(s) remaining
Completed 85.8 MiB/22.3 GiB (85.6 MiB/s) with 1 file(s) remaining
Completed 86.0 MiB/22.3 GiB (85.8 MiB/s) with 1 file(s) remaining
Completed 86.2 MiB/22.3 GiB (85.9 MiB/s) with 1 file(s) remaining
Completed 86.5 MiB/22.3 GiB (86.1 MiB/s) with 1 file(s) remaining
Completed 86.8 MiB/22.3 GiB (86.3 MiB/s) with 1 file(s) remaining
Completed 87.0 MiB/22.3 GiB (86.5 MiB/s) with 1 file(s) remaining
Completed 87.2 MiB/22.3 GiB (86.6 MiB/s) with 1 file(s) remaining
Completed 87.5 MiB/22.3 GiB (86.8 MiB/s) with 1 file(s) remaining
Completed 87.8 MiB/22.3 GiB (87.0 MiB/s) with 1 file(s) remaining
Completed 88.0 MiB/22.3 GiB (87.2 MiB/s) with 1 file(s) remaining
Completed 88.2 MiB/22.3 GiB (87.4 MiB/s) with 1 file(s) remaining
Completed 88.5 MiB/22.3 GiB (87.6 MiB/s) with 1 file(s) remaining
Completed 88.8 MiB/22.3 GiB (87.8 MiB/s) with 1 file(s) remaining
Completed 89.0 MiB/22.3 GiB (87.9 MiB/s) with 1 file(s) remaining
Completed 89.2 MiB/22.3 GiB (88.1 MiB/s) with 1 file(s) remaining
Completed 89.5 MiB/22.3 GiB (88.1 MiB/s) with 1 file(s) remaining
Completed 89.8 MiB/22.3 GiB (88.3 MiB/s) with 1 file(s) remaining
Completed 90.0 MiB/22.3 GiB (88.5 MiB/s) with 1 file(s) remaining
Completed 90.2 MiB/22.3 GiB (88.7 MiB/s) with 1 file(s) remaining
Completed 90.5 MiB/22.3 GiB (88.9 MiB/s) with 1 file(s) remaining
Completed 90.8 MiB/22.3 GiB (89.1 MiB/s) with 1 file(s) remaining
Completed 91.0 MiB/22.3 GiB (89.3 MiB/s) with 1 file(s) remaining
Completed 91.2 MiB/22.3 GiB (89.6 MiB/s) with 1 file(s) remaining
Completed 91.5 MiB/22.3 GiB (89.8 MiB/s) with 1 file(s) remaining
Completed 91.8 MiB/22.3 GiB (90.0 MiB/s) with 1 file(s) remaining
Completed 92.0 MiB/22.3 GiB (90.0 MiB/s) with 1 file(s) remaining
Completed 92.2 MiB/22.3 GiB (90.2 MiB/s) with 1 file(s) remaining
Completed 92.5 MiB/22.3 GiB (90.4 MiB/s) with 1 file(s) remaining
Completed 92.8 MiB/22.3 GiB (90.6 MiB/s) with 1 file(s) remaining
Completed 93.0 MiB/22.3 GiB (90.8 MiB/s) with 1 file(s) remaining
Completed 93.2 MiB/22.3 GiB (91.0 MiB/s) with 1 file(s) remaining
Completed 93.5 MiB/22.3 GiB (91.1 MiB/s) with 1 file(s) remaining
Completed 93.8 MiB/22.3 GiB (91.2 MiB/s) with 1 file(s) remaining
Completed 94.0 MiB/22.3 GiB (91.4 MiB/s) with 1 file(s) remaining
Completed 94.2 MiB/22.3 GiB (91.5 MiB/s) with 1 file(s) remaining
Completed 94.5 MiB/22.3 GiB (91.7 MiB/s) with 1 file(s) remaining
Completed 94.8 MiB/22.3 GiB (91.9 MiB/s) with 1 file(s) remaining
Completed 95.0 MiB/22.3 GiB (92.1 MiB/s) with 1 file(s) remaining
Completed 95.2 MiB/22.3 GiB (92.3 MiB/s) with 1 file(s) remaining
Completed 95.5 MiB/22.3 GiB (92.5 MiB/s) with 1 file(s) remaining
Completed 95.8 MiB/22.3 GiB (92.7 MiB/s) with 1 file(s) remaining
Completed 96.0 MiB/22.3 GiB (92.9 MiB/s) with 1 file(s) remaining
Completed 96.2 MiB/22.3 GiB (93.1 MiB/s) with 1 file(s) remaining
Completed 96.5 MiB/22.3 GiB (93.2 MiB/s) with 1 file(s) remaining
Completed 96.8 MiB/22.3 GiB (93.4 MiB/s) with 1 file(s) remaining
Completed 97.0 MiB/22.3 GiB (93.5 MiB/s) with 1 file(s) remaining
Completed 97.2 MiB/22.3 GiB (93.7 MiB/s) with 1 file(s) remaining
Completed 97.5 MiB/22.3 GiB (93.8 MiB/s) with 1 file(s) remaining
Completed 97.8 MiB/22.3 GiB (94.0 MiB/s) with 1 file(s) remaining
Completed 98.0 MiB/22.3 GiB (94.2 MiB/s) with 1 file(s) remaining
Completed 98.2 MiB/22.3 GiB (94.4 MiB/s) with 1 file(s) remaining
Completed 98.5 MiB/22.3 GiB (94.6 MiB/s) with 1 file(s) remaining
Completed 98.8 MiB/22.3 GiB (94.7 MiB/s) with 1 file(s) remaining
Completed 99.0 MiB/22.3 GiB (95.0 MiB/s) with 1 file(s) remaining
Completed 99.2 MiB/22.3 GiB (95.1 MiB/s) with 1 file(s) remaining
Completed 99.5 MiB/22.3 GiB (95.3 MiB/s) with 1 file(s) remaining
Completed 99.8 MiB/22.3 GiB (95.5 MiB/s) with 1 file(s) remaining
Completed 100.0 MiB/22.3 GiB (95.7 MiB/s) with 1 file(s) remaining
Completed 100.2 MiB/22.3 GiB (95.9 MiB/s) with 1 file(s) remaining
Completed 100.5 MiB/22.3 GiB (96.0 MiB/s) with 1 file(s) remaining
Completed 100.8 MiB/22.3 GiB (96.1 MiB/s) with 1 file(s) remaining
Completed 101.0 MiB/22.3 GiB (96.3 MiB/s) with 1 file(s) remaining
Completed 101.2 MiB/22.3 GiB (96.5 MiB/s) with 1 file(s) remaining
Completed 101.5 MiB/22.3 GiB (96.6 MiB/s) with 1 file(s) remaining
Completed 101.8 MiB/22.3 GiB (96.9 MiB/s) with 1 file(s) remaining
Completed 102.0 MiB/22.3 GiB (97.0 MiB/s) with 1 file(s) remaining
Completed 102.2 MiB/22.3 GiB (97.2 MiB/s) with 1 file(s) remaining
Completed 102.5 MiB/22.3 GiB (97.4 MiB/s) with 1 file(s) remaining
Completed 102.8 MiB/22.3 GiB (97.5 MiB/s) with 1 file(s) remaining
Completed 103.0 MiB/22.3 GiB (97.7 MiB/s) with 1 file(s) remaining
Completed 103.2 MiB/22.3 GiB (97.9 MiB/s) with 1 file(s) remaining
Completed 103.5 MiB/22.3 GiB (98.1 MiB/s) with 1 file(s) remaining
Completed 103.8 MiB/22.3 GiB (98.3 MiB/s) with 1 file(s) remaining
Completed 104.0 MiB/22.3 GiB (98.4 MiB/s) with 1 file(s) remaining
Completed 104.2 MiB/22.3 GiB (98.6 MiB/s) with 1 file(s) remaining
Completed 104.5 MiB/22.3 GiB (98.8 MiB/s) with 1 file(s) remaining
Completed 104.8 MiB/22.3 GiB (99.0 MiB/s) with 1 file(s) remaining
Completed 105.0 MiB/22.3 GiB (99.1 MiB/s) with 1 file(s) remaining
Completed 105.2 MiB/22.3 GiB (99.1 MiB/s) with 1 file(s) remaining
Completed 105.5 MiB/22.3 GiB (99.3 MiB/s) with 1 file(s) remaining
Completed 105.8 MiB/22.3 GiB (99.5 MiB/s) with 1 file(s) remaining
Completed 106.0 MiB/22.3 GiB (99.6 MiB/s) with 1 file(s) remaining
Completed 106.2 MiB/22.3 GiB (99.8 MiB/s) with 1 file(s) remaining
Completed 106.5 MiB/22.3 GiB (100.0 MiB/s) with 1 file(s) remaining
Completed 106.8 MiB/22.3 GiB (100.1 MiB/s) with 1 file(s) remaining
Completed 107.0 MiB/22.3 GiB (100.3 MiB/s) with 1 file(s) remaining
Completed 107.2 MiB/22.3 GiB (100.5 MiB/s) with 1 file(s) remaining
Completed 107.5 MiB/22.3 GiB (100.7 MiB/s) with 1 file(s) remaining
Completed 107.8 MiB/22.3 GiB (100.9 MiB/s) with 1 file(s) remaining
Completed 108.0 MiB/22.3 GiB (101.0 MiB/s) with 1 file(s) remaining
Completed 108.2 MiB/22.3 GiB (101.1 MiB/s) with 1 file(s) remaining
Completed 108.5 MiB/22.3 GiB (101.3 MiB/s) with 1 file(s) remaining
Completed 108.8 MiB/22.3 GiB (101.5 MiB/s) with 1 file(s) remaining
Completed 109.0 MiB/22.3 GiB (101.7 MiB/s) with 1 file(s) remaining
Completed 109.2 MiB/22.3 GiB (102.0 MiB/s) with 1 file(s) remaining
Completed 109.5 MiB/22.3 GiB (102.1 MiB/s) with 1 file(s) remaining
Co


Still downloading. Let me check again in a moment.


Bash

ls -lh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors 2>&1

Check if download is complete


-rw-rw-r-- 1 ubuntu ubuntu 23G Mar 30 16:51 /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-70000/model.safetensors


Download complete! Benchmark should be running now.


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline progress


DOWNLOAD DONE


Download done, benchmark starting. Let me wait for results.


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline progress


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline progress


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline progress


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline progress


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline


grep "processed\|DONE\|BENCHMARK\|DASHBOARD\|OVERALL" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check pipeline


DOWNLOAD DONE
  [128/6000] (2%) processed

DOWNLOAD DONE
  [128/6000] (2%) processed


Benchmark running. Let me check at wider intervals.


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


DOWNLOAD DONE
  [128/6000] (2%) processed
  [256/6000] (4%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


DOWNLOAD DONE
  [128/6000] (2%) processed
  [256/6000] (4%) processed
  [384/6000] (6%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


DOWNLOAD DONE
  [128/6000] (2%) processed
  [256/6000] (4%) processed
  [384/6000] (6%) processed
  [512/6000] (9%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


  [128/6000] (2%) processed
  [256/6000] (4%) processed
  [384/6000] (6%) processed
  [512/6000] (9%) processed
  [640/6000] (11%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


  [256/6000] (4%) processed
  [384/6000] (6%) processed
  [512/6000] (9%) processed
  [640/6000] (11%) processed
  [768/6000] (13%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


  [384/6000] (6%) processed
  [512/6000] (9%) processed
  [640/6000] (11%) processed
  [768/6000] (13%) processed
  [896/6000] (15%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


  [512/6000] (9%) processed
  [640/6000] (11%) processed
  [768/6000] (13%) processed
  [896/6000] (15%) processed
  [1024/6000] (17%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check pipeline


17%. Running well. About 20 more minutes. I'll wait for the background task to complete.


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [896/6000] (15%) processed
  [1024/6000] (17%) processed
  [1152/6000] (19%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1024/6000] (17%) processed
  [1152/6000] (19%) processed
  [1280/6000] (21%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1152/6000] (19%) processed
  [1280/6000] (21%) processed
  [1408/6000] (23%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1408/6000] (23%) processed
  [1536/6000] (26%) processed
  [1664/6000] (28%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1536/6000] (26%) processed
  [1664/6000] (28%) processed
  [1792/6000] (30%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1664/6000] (28%) processed
  [1792/6000] (30%) processed
  [1920/6000] (32%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1792/6000] (30%) processed
  [1920/6000] (32%) processed
  [2048/6000] (34%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [1920/6000] (32%) processed
  [2048/6000] (34%) processed
  [2176/6000] (36%) processed


36%. About 15 more minutes.

grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2176/6000] (36%) processed
  [2304/6000] (38%) processed
  [2432/6000] (41%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2304/6000] (38%) processed
  [2432/6000] (41%) processed
  [2560/6000] (43%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2432/6000] (41%) processed
  [2560/6000] (43%) processed
  [2688/6000] (45%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2560/6000] (43%) processed
  [2688/6000] (45%) processed
  [2816/6000] (47%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2688/6000] (45%) processed
  [2816/6000] (47%) processed
  [2944/6000] (49%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2816/6000] (47%) processed
  [2944/6000] (49%) processed
  [3072/6000] (51%) processed


51%. About 12 more minutes.


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [2944/6000] (49%) processed
  [3072/6000] (51%) processed
  [3200/6000] (53%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3072/6000] (51%) processed
  [3200/6000] (53%) processed
  [3328/6000] (55%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3200/6000] (53%) processed
  [3328/6000] (55%) processed
  [3456/6000] (58%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3328/6000] (55%) processed
  [3456/6000] (58%) processed
  [3584/6000] (60%) processed


60%. About 10 more minutes.


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3456/6000] (58%) processed
  [3584/6000] (60%) processed
  [3712/6000] (62%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3584/6000] (60%) processed
  [3712/6000] (62%) processed
  [3840/6000] (64%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3712/6000] (62%) processed
  [3840/6000] (64%) processed
  [3968/6000] (66%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3840/6000] (64%) processed
  [3968/6000] (66%) processed
  [4096/6000] (68%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [3968/6000] (66%) processed
  [4096/6000] (68%) processed
  [4224/6000] (70%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4096/6000] (68%) processed
  [4224/6000] (70%) processed
  [4352/6000] (73%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4224/6000] (70%) processed
  [4352/6000] (73%) processed
  [4480/6000] (75%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4352/6000] (73%) processed
  [4480/6000] (75%) processed
  [4608/6000] (77%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4480/6000] (75%) processed
  [4608/6000] (77%) processed
  [4736/6000] (79%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4608/6000] (77%) processed
  [4736/6000] (79%) processed
  [4864/6000] (81%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4736/6000] (79%) processed
  [4864/6000] (81%) processed
  [4992/6000] (83%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4864/6000] (81%) processed
  [4992/6000] (83%) processed
  [5120/6000] (85%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [4992/6000] (83%) processed
  [5120/6000] (85%) processed
  [5248/6000] (87%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [5120/6000] (85%) processed
  [5248/6000] (87%) processed
  [5376/6000] (90%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [5248/6000] (87%) processed
  [5376/6000] (90%) processed
  [5504/6000] (92%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [5376/6000] (90%) processed
  [5504/6000] (92%) processed
  [5632/6000] (94%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [5504/6000] (92%) processed
  [5632/6000] (94%) processed
  [5760/6000] (96%) processed


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


grep "processed\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -3

Check pipeline


  [5632/6000] (94%) processed
  [5760/6000] (96%) processed
  [5888/6000] (98%) processed


grep "processed\|DONE\|OVERALL\|BENCHMARK\|DASHBOARD\|Diagnosis\|wer_norm" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -20

Check for final results


  [3456/6000] (58%) processed
  [3584/6000] (60%) processed
  [3712/6000] (62%) processed
  [3840/6000] (64%) processed
  [3968/6000] (66%) processed
  [4096/6000] (68%) processed
  [4224/6000] (70%) processed
  [4352/6000] (73%) processed
  [4480/6000] (75%) processed
  [4608/6000] (77%) processed
  [4736/6000] (79%) processed
  [4864/6000] (81%) processed
  [4992/6000] (83%) processed
  [5120/6000] (85%) processed
  [5248/6000] (87%) processed
  [5376/6000] (90%) processed
  [5504/6000] (92%) processed
  [5632/6000] (94%) processed
  [5760/6000] (96%) processed
  [5888/6000] (98%) processed


grep "OVERALL\|BENCHMARK\|DASHBOARD\|Diagnosis\|wer_norm\|DONE" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -10

Check for final results


DOWNLOAD DONE
  BENCHMARK RESULTS: ckpt-70000
OVERALL         6000    25.98    10.51    21.46     9.87
BENCHMARK DONE


Get results table


DOWNLOAD DONE
  BENCHMARK RESULTS: ckpt-70000
Language           N     WER%     CER%   WER-n%   CER-n%
-------------- ----- -------- -------- -------- --------
assamese         500    14.25     6.21    11.54     6.13
bengali          500    19.39     5.01    14.74     4.36
english          500    33.57    22.60    23.92    18.59
gujarati         500    17.68     5.91    13.35     5.25
hindi            500    12.31     4.29     8.24     3.43
kannada          500    38.84    19.90    36.06    19.56
malayalam        500    39.13     7.69    34.87     6.89
marathi          500    22.92     6.55    18.36     5.76
odia             500    33.55    15.51    28.09    14.34
punjabi          500    24.41    13.44    18.66    12.08
tamil            500    32.14    10.31    30.10     9.78
telugu           500    29.38     9.83    24.70     9.10
-------------- ----- -------- -------- -------- --------
OVERALL         6000    25.98    10.51    21.46     9.87
MACRO-AVG         12    26.46    10.60    21.89     9.61
BENCHMARK DONE


Benchmark done. Dashboard generation should follow.


grep "DASHBOARD\|Diagnosis\|Overall" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check dashboard generation


(Bash completed with no output)


grep "DASHBOARD\|Diagnosis\|Overall" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output 2>&1 | tail -5

Check dashboard


<task-notification>
<task-id>bmxvx7yzg</task-id>
<tool-use-id>toolu_01DpypvNRRx5dJwKZq7G1rLj</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bmxvx7yzg.output</output-file>
<status>completed</status>
<summary>Background command "Download + benchmark + dashboard for ckpt-70000" completed (exit code 0)</summary>
</task-notification>


Check dashboard


  Overall: wer_raw=25.91 wer_norm=25.03 wer_nc=25.03 snwer=19.03 mer=10.87 cer_norm=10.32
  Diagnosis: recognition-limited
DASHBOARD DONE


All done! ckpt-70000 fully processed — benchmark + dashboard JSON. Here's the complete progression:

### Gemma3n-E2B-ASR Training Curve (03-28-2026)

| Metric | ckpt-20k | ckpt-40k | ckpt-55k | **ckpt-70k** |
|--------|---------|---------|---------|------------|
| **wer_norm** | 29.39 | 26.14 | 25.64 | **25.03** |
| wer_raw | 30.43 | 27.01 | 26.58 | 25.91 |
| cer_norm | 12.73 | 10.21 | 9.90 | 10.32 |

**ckpt-70000 per-language (WER-norm):**
- Hindi: **8.24%** (best ever)
- Assamese: **11.54%** (massive drop from 18.2%)
- Gujarati: **13.35%**
- Bengali: **14.74%**
- Punjabi: **18.66%**
- Marathi: **18.36%**
- English: 23.92%
- Telugu: 24.70%
- Odia: 28.09%
- Tamil: 30.10%
- Malayalam: 34.87%
- Kannada: 36.06%

Training at 45% of epoch. Model still improving — overall wer_norm: 29.4 → 26.1 → 25.6 → **25.0**.

Dashboard files: `/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-70000/`


<task-notification>
<task-id>bjm3bidp5</task-id>
<tool-use-id>toolu_01Rskscec23cjeCbMB2y8434</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bjm3bidp5.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-70000 (multipart parallel)" completed (exit code 0)</summary>
</task-notification>


<task-notification>
<task-id>b3w3pkmiw</task-id>
<tool-use-id>toolu_012oQacYaKp5ttTsqZ6fY5h3</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b3w3pkmiw.output</output-file>
<status>completed</status>
<summary>Background command "Poll R2 every 2min for model.safetensors" completed (exit code 0)</summary>
</task-notification>


All background tasks completed successfully. ckpt-70000 is fully benchmarked and in the dashboard at `/home/ubuntu/training/benchmark_outputs/gemma3n-e2b/ckpt-70000/`.


what's the raw WER excluding english. can you also exclude english fopr the remaining models as well. all details are in /home/ubuntu/training/benchmark_outputs


python3 -c "
import json, os

base = '/home/ubuntu/training/benchmark_outputs'
for model_id in sorted(os.listdir(base)):
    model_dir = os.path.join(base, model_id)
    if not os.path.isdir(model_dir):
        continue
    print(f'\n=== {model_id} ===')
    print(f'{\"Checkpoint\":<16} {\"WER_raw (all)\":>14} {\"WER_raw (no-en)\":>16} {\"WER_norm (all)\":>15} {\"WER_norm (no-en)\":>17}')
    print('-' * 80)
    for ckpt in sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0):
        mf = os.path.join(model_dir, ckpt, 'metrics.json')
        if not os.path.exists(mf):
            continue
        with open(mf) as f:
            m = json.load(f)
        langs = [k for k in m if not k.startswith('__')]
        langs_no_en = [k for k in langs if k != 'english']

        # Overall (all)
        ov = m.get('__overall__', {})
        wer_raw_all = ov.get('wer_raw', ov.get('wer', '-'))
        wer_norm_all = ov.get('wer_norm', ov.get('wer_normalized', '-'))

        # Recompute excluding english — need weighted micro-average
        # We need per-lang n_samples and raw totals, but jiwer corpus-level != simple average
        # Best approx: macro average of per-lang scores excluding english
        raw_no_en = [m[l].get('wer_raw', m[l].get('wer', 0)) for l in langs_no_en]
        norm_no_en = [m[l].get('wer_norm', m[l].get('wer_normalized', 0)) for l in langs_no_en]

        if raw_no_en:
            wer_raw_noen = round(sum(raw_no_en) / len(raw_no_en), 2)
            wer_norm_noen = round(sum(norm_no_en) / len(norm_no_en), 2)
        else:
            wer_raw_noen = '-'
            wer_norm_noen = '-'

        print(f'{ckpt:<16} {str(wer_raw_all):>14} {str(wer_raw_noen):>16} {str(wer_norm_all):>15} {str(wer_norm_noen):>17}')
" 2>&1

Compute WER excluding English for all models


=== elevenlabs-scribe-v1 ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                  25.82            24.41           21.11             20.61

=== elevenlabs-scribe-v2 ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                  21.94            21.43           17.74             17.97

=== gemini-2.5-flash ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                   63.8            68.78           57.71             62.93

=== gemini-3-flash-preview ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                  71.47            76.73           67.68             73.87

=== gemini-3-flash-strict ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                  27.62            27.87           20.12             20.56

=== gemini-3.1-pro ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                   20.5            19.86           18.98              19.6

=== gemma3n-e2b ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
ckpt-20000                30.43            30.84           29.39             30.72
ckpt-40000                27.01             27.3           26.14              27.2
ckpt-55000                26.58            26.73           25.64             26.61
ckpt-70000                25.91            25.74           25.03             25.61

=== parakeet-1.1b ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------

=== parakeet-tdt-1.1b-lang ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
ckpt-60000                38.67            39.75           37.07             39.15
ckpt-80000                38.49            39.58           36.93             39.02
ckpt-120000               35.03            35.72           33.68             35.39
ckpt-200000               34.83            35.58           33.49             35.23
ckpt-260000               33.89            34.36            32.5             33.91

=== qwen3-asr ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
ckpt-24000                40.81            41.77           39.12             41.02
ckpt-72000                 38.2            39.07           36.77             38.47
ckpt-100000               37.59            38.37           35.98             37.69
ckpt-170000                37.2            37.97           35.59             37.28
ckpt-200000               37.28            38.13           35.64             37.36
ckpt-250000               28.57            28.99           27.18             28.43
ckpt-300000               28.37            28.67           26.96             28.12
ckpt-400000               28.59            28.87           27.21             28.36

=== sarvam-saaras-v3 ===
Checkpoint        WER_raw (all)  WER_raw (no-en)  WER_norm (all)  WER_norm (no-en)
--------------------------------------------------------------------------------
baseline                  26.36            27.12           18.26             19.39


Here's the full leaderboard — **WER_raw excluding English** (macro-avg over 11 Indic languages):

| Model | Checkpoint | WER_raw (no-en) | WER_norm (no-en) |
|-------|-----------|----------------|-----------------|
| **gemini-3.1-pro** | baseline | **19.86** | 19.60 |
| elevenlabs-scribe-v2 | baseline | 21.43 | 17.97 |
| elevenlabs-scribe-v1 | baseline | 24.41 | 20.61 |
| **gemma3n-e2b** | ckpt-70000 | **25.74** | 25.61 |
| gemma3n-e2b | ckpt-55000 | 26.73 | 26.61 |
| gemma3n-e2b | ckpt-40000 | 27.30 | 27.20 |
| sarvam-saaras-v3 | baseline | 27.12 | 19.39 |
| gemini-3-flash-strict | baseline | 27.87 | 20.56 |
| qwen3-asr | ckpt-300000 | 28.67 | 28.12 |
| qwen3-asr | ckpt-400000 | 28.87 | 28.36 |
| gemma3n-e2b | ckpt-20000 | 30.84 | 30.72 |
| parakeet-tdt-1.1b | ckpt-260000 | 34.36 | 33.91 |
| qwen3-asr | ckpt-250000 | 28.99 | 28.43 |

**Key observations:**
- **gemma3n-e2b at ckpt-70k** has Indic-only raw WER of **25.74%** — closing in on commercial APIs (ElevenLabs v1 at 24.41%)
- Interestingly, gemma3n's English is relatively weaker (pulls overall up), but its Indic performance is competitive
- sarvam-saaras-v3 has high raw WER (27.12) but very low norm WER (19.39) — heavy formatting/punctuation in outputs
- qwen3-asr plateaued around 28-29% for Indic languages


why is the other models wer_norm very low for other models when compared to our gemma3n-e2b and what about CER. what's the pattern for our consisetent WER-WER_norm


Good question. Let me dig into the normalization deltas and CER across all models.


python3 -c "
import json, os

base = '/home/ubuntu/training/benchmark_outputs'
print(f'{'Model':<28} {'Ckpt':<14} {'WER_raw':>8} {'WER_norm':>9} {'Δ raw→norm':>11} {'CER_norm':>9} {'WER/CER ratio':>14}')
print('=' * 95)

for model_id in sorted(os.listdir(base)):
    model_dir = os.path.join(base, model_id)
    if not os.path.isdir(model_dir):
        continue
    ckpts = sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0)
    # Show only latest checkpoint per model
    for ckpt in ckpts[-1:]:
        mf = os.path.join(model_dir, ckpt, 'metrics.json')
        if not os.path.exists(mf):
            continue
        with open(mf) as f:
            m = json.load(f)
        ov = m.get('__overall__', {})
        wr = ov.get('wer_raw', ov.get('wer', 0))
        wn = ov.get('wer_norm', ov.get('wer_normalized', 0))
        cn = ov.get('cer_norm', ov.get('cer_normalized', ov.get('cer', 0)))
        delta = round(wr - wn, 2) if isinstance(wr, (int,float)) and isinstance(wn, (int,float)) else '-'
        ratio = round(wn / cn, 2) if cn and cn > 0 else '-'
        print(f'{model_id:<28} {ckpt:<14} {wr:>8} {wn:>9} {delta:>+11} {cn:>9} {ratio:>14}')

print()
print('=' * 95)
print()
print('Per-language breakdown for top models (latest ckpt):')
print()

models_to_show = ['gemma3n-e2b', 'sarvam-saaras-v3', 'elevenlabs-scribe-v2', 'gemini-3-flash-strict', 'qwen3-asr']
langs = ['hindi','bengali','tamil','kannada','malayalam','odia','english']

for mid in models_to_show:
    model_dir = os.path.join(base, mid)
    if not os.path.isdir(model_dir):
        continue
    ckpts = sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0)
    ckpt = ckpts[-1]
    mf = os.path.join(model_dir, ckpt, 'metrics.json')
    if not os.path.exists(mf):
        continue
    with open(mf) as f:
        m = json.load(f)
    print(f'--- {mid} ({ckpt}) ---')
    print(f'  {\"Lang\":<12} {\"WER_raw\":>8} {\"WER_norm\":>9} {\"Δ\":>7} {\"CER_norm\":>9} {\"WER_n/CER_n\":>12}')
    for lang in langs:
        if lang not in m:
            continue
        l = m[lang]
        wr = l.get('wer_raw', l.get('wer', 0))
        wn = l.get('wer_norm', l.get('wer_normalized', 0))
        cn = l.get('cer_norm', l.get('cer_normalized', l.get('cer', 0)))
        d = round(wr - wn, 2)
        ratio = round(wn / cn, 2) if cn and cn > 0 else '-'
        print(f'  {lang:<12} {wr:>8} {wn:>9} {d:>+7} {cn:>9} {ratio:>12}')
    print()
" 2>&1

Analyze normalization delta and CER patterns


Error: Exit code 1
  File "<string>", line 5
    print(f'{'Model':<28} {'Ckpt':<14} {'WER_raw':>8} {'WER_norm':>9} {'Δ raw→norm':>11} {'CER_norm':>9} {'WER/CER ratio':>14}')
                                                                             ^
SyntaxError: invalid character '→' (U+2192)


python3 << 'PYEOF'
import json, os

base = '/home/ubuntu/training/benchmark_outputs'

# Part 1: Overall comparison
header = f"{'Model':<28} {'Ckpt':<14} {'WER_raw':>8} {'WER_nrm':>8} {'Delta':>7} {'CER_nrm':>8} {'W/C ratio':>10}"
print(header)
print("=" * 85)

for model_id in sorted(os.listdir(base)):
    model_dir = os.path.join(base, model_id)
    if not os.path.isdir(model_dir):
        continue
    ckpts = sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0)
    for ckpt in ckpts[-1:]:
        mf = os.path.join(model_dir, ckpt, 'metrics.json')
        if not os.path.exists(mf):
            continue
        with open(mf) as f:
            m = json.load(f)
        ov = m.get('__overall__', {})
        wr = ov.get('wer_raw', ov.get('wer', 0))
        wn = ov.get('wer_norm', ov.get('wer_normalized', 0))
        cn = ov.get('cer_norm', ov.get('cer_normalized', ov.get('cer', 0)))
        delta = round(wr - wn, 2) if isinstance(wr, (int,float)) and isinstance(wn, (int,float)) else 0
        ratio = round(wn / cn, 2) if cn and cn > 0 else 0
        print(f"{model_id:<28} {ckpt:<14} {wr:>8} {wn:>8} {delta:>+7} {cn:>8} {ratio:>10}")

print()
print("=" * 85)
print()
print("Per-language: raw-to-norm delta (shows how much punctuation/casing inflates WER)")
print()

models_to_show = ['gemma3n-e2b', 'sarvam-saaras-v3', 'elevenlabs-scribe-v2', 'gemini-3-flash-strict', 'qwen3-asr']
langs_all = ['hindi','bengali','gujarati','tamil','kannada','malayalam','marathi','odia','english','assamese','punjabi','telugu']

for mid in models_to_show:
    model_dir = os.path.join(base, mid)
    if not os.path.isdir(model_dir):
        continue
    ckpts = sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0)
    ckpt = ckpts[-1]
    mf = os.path.join(model_dir, ckpt, 'metrics.json')
    if not os.path.exists(mf):
        continue
    with open(mf) as f:
        m = json.load(f)
    print(f"--- {mid} ({ckpt}) ---")
    print(f"  {'Lang':<12} {'WER_raw':>8} {'WER_nrm':>8} {'Delta':>7} {'CER_nrm':>8} {'W/C':>6}")
    deltas = []
    for lang in langs_all:
        if lang not in m:
            continue
        l = m[lang]
        wr = l.get('wer_raw', l.get('wer', 0))
        wn = l.get('wer_norm', l.get('wer_normalized', 0))
        cn = l.get('cer_norm', l.get('cer_normalized', l.get('cer', 0)))
        d = round(wr - wn, 2)
        ratio = round(wn / cn, 2) if cn and cn > 0 else 0
        deltas.append(d)
        print(f"  {lang:<12} {wr:>8} {wn:>8} {d:>+7} {cn:>8} {ratio:>6}")
    avg_delta = round(sum(deltas) / len(deltas), 2) if deltas else 0
    print(f"  {'AVG DELTA':<12} {'':>8} {'':>8} {avg_delta:>+7}")
    print()

# Part 3: gemma3n checkpoint progression
print("=" * 85)
print("gemma3n-e2b checkpoint progression (all metrics):")
print(f"  {'Ckpt':<14} {'WER_raw':>8} {'WER_nrm':>8} {'Delta':>7} {'CER_nrm':>8} {'W/C':>6} {'Note':>20}")
print(f"  {'-'*14} {'-'*8} {'-'*8} {'-'*7} {'-'*8} {'-'*6} {'-'*20}")
model_dir = os.path.join(base, 'gemma3n-e2b')
for ckpt in sorted(os.listdir(model_dir), key=lambda x: int(x.split('-')[-1]) if x.split('-')[-1].isdigit() else 0):
    mf = os.path.join(model_dir, ckpt, 'metrics.json')
    if not os.path.exists(mf):
        continue
    with open(mf) as f:
        m = json.load(f)
    ov = m.get('__overall__', {})
    wr = ov.get('wer_raw', 0)
    wn = ov.get('wer_norm', 0)
    cn = ov.get('cer_norm', 0)
    d = round(wr - wn, 2)
    ratio = round(wn / cn, 2) if cn > 0 else 0
    note = "consistent ~1pt" if d < 1.5 else "formatting gap"
    print(f"  {ckpt:<14} {wr:>8} {wn:>8} {d:>+7} {cn:>8} {ratio:>6} {note:>20}")
PYEOF

Full normalization and CER analysis


Model                        Ckpt            WER_raw  WER_nrm   Delta  CER_nrm  W/C ratio
=====================================================================================
elevenlabs-scribe-v1         baseline          25.82    21.11   +4.71     6.29       3.36
elevenlabs-scribe-v2         baseline          21.94    17.74    +4.2     5.28       3.36
gemini-2.5-flash             baseline           63.8    57.71   +6.09    63.57       0.91
gemini-3-flash-preview       baseline          71.47    67.68   +3.79     95.0       0.71
gemini-3-flash-strict        baseline          27.62    20.12    +7.5     5.79       3.47
gemini-3.1-pro               baseline           20.5    18.98   +1.52     4.92       3.86
gemma3n-e2b                  ckpt-70000        25.91    25.03   +0.88    10.32       2.43
parakeet-tdt-1.1b-lang       ckpt-260000       33.89     32.5   +1.39     9.51       3.42
qwen3-asr                    ckpt-400000       28.59    27.21   +1.38     8.16       3.33
sarvam-saaras-v3             baseline          26.36    18.26    +8.1     5.01       3.64

=====================================================================================

Per-language: raw-to-norm delta (shows how much punctuation/casing inflates WER)

--- gemma3n-e2b (ckpt-70000) ---
  Lang          WER_raw  WER_nrm   Delta  CER_nrm    W/C
  hindi           12.31    12.31    +0.0     4.29   2.87
  bengali         19.06    18.53   +0.53     4.78   3.88
  gujarati        17.68    17.68    +0.0     5.89    3.0
  tamil           32.14    32.12   +0.02    10.31   3.12
  kannada         38.84    38.84    +0.0    19.88   1.95
  malayalam       39.13    39.13    +0.0     7.68    5.1
  marathi         22.41    22.41    +0.0     6.39   3.51
  odia            33.55    33.35    +0.2    15.48   2.15
  english         33.57    23.92   +9.65    20.69   1.16
  assamese        14.25    14.06   +0.19     6.05   2.32
  punjabi         24.39     23.9   +0.49    13.29    1.8
  telugu          29.38    29.36   +0.02     9.82   2.99
  AVG DELTA                        +0.92

--- sarvam-saaras-v3 (baseline) ---
  Lang          WER_raw  WER_nrm   Delta  CER_nrm    W/C
  hindi           15.73      9.8   +5.93     3.22   3.04
  bengali         23.33    15.65   +7.68     4.91   3.19
  gujarati        23.06    14.71   +8.35      4.2    3.5
  tamil           31.49    24.31   +7.18     4.24   5.73
  kannada         27.17    19.81   +7.36      4.6   4.31
  malayalam        42.5    34.74   +7.76     6.77   5.13
  marathi         27.28    19.66   +7.62     6.59   2.98
  odia            30.43    21.57   +8.86     5.88   3.67
  english         18.75     6.82  +11.93     3.06   2.23
  assamese         27.3     18.0    +9.3     7.59   2.37
  punjabi          21.6    13.58   +8.02     3.64   3.73
  telugu          28.42    21.44   +6.98     3.63   5.91
  AVG DELTA                        +8.08

--- elevenlabs-scribe-v2 (baseline) ---
  Lang          WER_raw  WER_nrm   Delta  CER_nrm    W/C
  hindi           15.56     9.33   +6.23      5.1   1.83
  bengali         11.19    10.44   +0.75     2.17   4.81
  gujarati        17.28     13.1   +4.18     4.13   3.17
  tamil           23.08     21.9   +1.18     4.26   5.14
  kannada         23.15    18.56   +4.59     3.71    5.0
  malayalam       34.17    31.68   +2.49      5.9   5.37
  marathi         18.78    14.83   +3.95     3.68   4.03
  odia            17.44    17.03   +0.41     4.35   3.91
  english         24.83    13.94  +10.89     9.28    1.5
  assamese        32.73    25.11   +7.62    10.73   2.34
  punjabi         17.32    12.35   +4.97     3.87   3.19
  telugu          25.07    23.29   +1.78      5.9   3.95
  AVG DELTA                        +4.09

--- gemini-3-flash-strict (baseline) ---
  Lang          WER_raw  WER_nrm   Delta  CER_nrm    W/C
  hindi           17.06     9.31   +7.75     2.81   3.31
  bengali         19.36    13.65   +5.71     3.89   3.51
  gujarati        22.61     14.8   +7.81     4.53   3.27
  tamil           30.06    23.23   +6.83      4.3    5.4
  kannada         28.62    20.61   +8.01     4.62   4.46
  malayalam       43.29     35.9   +7.39      6.6   5.44
  marathi         26.93    19.24   +7.69     6.38   3.02
  odia            29.14    21.11   +8.03     4.85   4.35
  english         22.73    12.35  +10.38     7.87   1.57
  assamese        37.44    32.05   +5.39    14.28   2.24
  punjabi         22.33    14.79   +7.54     4.47   3.31
  telugu          29.77    21.48   +8.29     3.57   6.02
  AVG DELTA                        +7.57

--- qwen3-asr (ckpt-400000) ---
  Lang          WER_raw  WER_nrm   Delta  CER_nrm    W/C
  hindi           13.93    13.88   +0.05     4.74   2.93
  bengali         23.63    22.87   +0.76      6.5   3.52
  gujarati        20.81    20.54   +0.27     6.16   3.33
  tamil            40.3    39.12   +1.18     9.38   4.17
  kannada         34.66    34.03   +0.63     8.25   4.12
  malayalam       44.23    43.49   +0.74      9.0   4.83
  marathi         29.18    28.95   +0.23     9.07   3.19
  odia            27.61    27.35   +0.26     7.26   3.77
  english         27.25    15.37  +11.88     9.22   1.67
  assamese        29.06    28.91   +0.15    12.34   2.34
  punjabi         21.57    21.32   +0.25     6.75   3.16
  telugu          32.54    31.54    +1.0     6.89   4.58
  AVG DELTA                        +1.45

=====================================================================================
gemma3n-e2b checkpoint progression (all metrics):
  Ckpt            WER_raw  WER_nrm   Delta  CER_nrm    W/C                 Note
  -------------- -------- -------- ------- -------- ------ --------------------
  ckpt-20000        30.43    29.39   +1.04    12.73   2.31      consistent ~1pt
  ckpt-40000        27.01    26.14   +0.87    10.21   2.56      consistent ~1pt
  ckpt-55000        26.58    25.64   +0.94      9.9   2.59      consistent ~1pt
  ckpt-70000        25.91    25.03   +0.88    10.32   2.43      consistent ~1pt


Here's what's going on:

### Why other models have much lower WER_norm vs WER_raw

**The delta tells the story:**

| Model | raw-to-norm Delta | What it means |
|-------|------------------|---------------|
| **sarvam-saaras-v3** | **+8.1** | Outputs heavy punctuation, casing, formatting |
| **gemini-3-flash-strict** | **+7.5** | Same — adds commas, periods, capitalization |
| **elevenlabs-scribe-v2** | **+4.2** | Moderate formatting in outputs |
| **qwen3-asr** | **+1.4** | Minimal formatting (like gemma3n) |
| **gemma3n-e2b** | **+0.9** | Almost no formatting — outputs bare text |

Sarvam and Gemini produce **formatted transcripts** (punctuation, proper casing, dandas) — which inflates raw WER by ~8 points because the benchmark references are plain text. Once you normalize away punctuation/casing, their "real" recognition quality shows through. Our gemma3n outputs nearly raw text already, so normalization barely helps.

**This is not a quality difference — it's a formatting difference.** The fair comparison is **WER_norm** where everyone is on equal footing.

### CER pattern — gemma3n's W/C ratio is low

| Model | WER_norm/CER_norm ratio |
|-------|------------------------|
| gemini-3.1-pro | 3.86 |
| sarvam-saaras-v3 | 3.64 |
| gemini-3-flash-strict | 3.47 |
| parakeet-tdt-1.1b | 3.42 |
| elevenlabs-scribe-v2 | 3.36 |
| qwen3-asr | 3.33 |
| **gemma3n-e2b** | **2.43** |

A typical WER/CER ratio is ~3.3-3.8x (one wrong character = one wrong word, but words avg 3-4 chars). Our gemma3n at **2.43x** is unusually low, meaning:

- **When gemma3n gets a word wrong, it gets MORE characters wrong per word** — the errors are more "destructive" per word
- Or equivalently: **CER is high relative to WER** — the model sometimes produces longer substitutions or completely different words rather than near-miss typos
- Kannada (1.95x) and Punjabi (1.80x) are the worst — these languages likely have word-level confusion rather than character-level errors

**Bottom line:** gemma3n produces clean raw output (good), but when it makes errors they tend to be whole-word substitutions rather than minor character errors. The commercial APIs make more "cosmetic" errors (close to the right word).


i noticed a lot of language mismatches too. i think, can we just remove the samples where the originasl and generated transcriptions's language is mismatched. just wanna see straight up apples-apples coimpairosn of wer and wer_norm and others. also, excludfe the samples where numerics are involved. recompute all metrics to the 70k ckpt excluding - (numerics,language_mismatches)


Smart idea. Let me build a filter that detects language mismatches and numeric content, then recompute clean metrics.


python3 << 'PYEOF'
import json
import re
import unicodedata
from collections import defaultdict

# Load predictions
with open('/home/ubuntu/training/benchmark_results_gemma3n/ckpt-70000/predictions.json') as f:
    preds = json.load(f)

print(f"Total samples: {len(preds)}")

# ── Script detection ────────────────────────────────────────────────────

SCRIPT_RANGES = {
    "devanagari": (0x0900, 0x097F),   # hi, mr, sa
    "bengali":    (0x0980, 0x09FF),   # bn, as
    "gurmukhi":   (0x0A00, 0x0A7F),   # pa
    "gujarati":   (0x0A80, 0x0AFF),   # gu
    "oriya":      (0x0B00, 0x0B7F),   # or
    "tamil":      (0x0B80, 0x0BFF),   # ta
    "telugu":     (0x0C00, 0x0C7F),   # te
    "kannada":    (0x0C80, 0x0CFF),   # kn
    "malayalam":  (0x0D00, 0x0D7F),   # ml
    "latin":      (0x0041, 0x024F),   # en
}

LANG_TO_SCRIPT = {
    "hindi": "devanagari", "marathi": "devanagari",
    "bengali": "bengali", "assamese": "bengali",
    "punjabi": "gurmukhi", "gujarati": "gujarati",
    "odia": "oriya", "tamil": "tamil", "telugu": "telugu",
    "kannada": "kannada", "malayalam": "malayalam",
    "english": "latin",
}

def dominant_script(text):
    """Return the script with the most characters in text."""
    counts = defaultdict(int)
    for ch in text:
        cp = ord(ch)
        for script, (lo, hi) in SCRIPT_RANGES.items():
            if lo <= cp <= hi:
                counts[script] += 1
                break
    if not counts:
        return "unknown"
    return max(counts, key=counts.get)

# ── Numeric detection ───────────────────────────────────────────────────

DIGIT_PATTERN = re.compile(r'\d')
# Devanagari digits, Bengali digits, etc.
INDIC_DIGIT_PATTERN = re.compile(r'[\u0966-\u096F\u09E6-\u09EF\u0A66-\u0A6F\u0AE6-\u0AEF\u0B66-\u0B6F\u0BE6-\u0BEF\u0C66-\u0C6F\u0CE6-\u0CEF\u0D66-\u0D6F]')
# Common number words in English
EN_NUM_WORDS = re.compile(r'\b(zero|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty|thirty|forty|fifty|sixty|seventy|eighty|ninety|hundred|thousand|million|billion|lakh|crore|karod)\b', re.I)

def has_numerics(text):
    """Check if text contains digits or number words."""
    if DIGIT_PATTERN.search(text):
        return True
    if INDIC_DIGIT_PATTERN.search(text):
        return True
    if EN_NUM_WORDS.search(text):
        return True
    return False

# ── Classify each sample ───────────────────────────────────────────────

lang_mismatch = 0
numeric_samples = 0
both = 0
clean = 0

by_lang = defaultdict(lambda: {"total": 0, "mismatch": 0, "numeric": 0, "clean": 0,
                                 "mismatch_examples": [], "numeric_examples": []})

for p in preds:
    lang = p["language"]
    ref = p["reference"]
    hyp = p.get("hypothesis", "")

    expected_script = LANG_TO_SCRIPT.get(lang, "unknown")
    hyp_script = dominant_script(hyp) if hyp.strip() else "empty"

    is_mismatch = (hyp.strip() != "" and hyp_script != expected_script and hyp_script != "unknown")
    is_numeric = has_numerics(ref) or has_numerics(hyp)

    by_lang[lang]["total"] += 1

    if is_mismatch and is_numeric:
        both += 1
        by_lang[lang]["mismatch"] += 1
        by_lang[lang]["numeric"] += 1
    elif is_mismatch:
        lang_mismatch += 1
        by_lang[lang]["mismatch"] += 1
        if len(by_lang[lang]["mismatch_examples"]) < 2:
            by_lang[lang]["mismatch_examples"].append({
                "id": p["id"], "expected": expected_script, "got": hyp_script,
                "ref": ref[:60], "hyp": hyp[:60]
            })
    elif is_numeric:
        numeric_samples += 1
        by_lang[lang]["numeric"] += 1
        if len(by_lang[lang]["numeric_examples"]) < 2:
            by_lang[lang]["numeric_examples"].append({
                "id": p["id"], "ref": ref[:60], "hyp": hyp[:60]
            })
    else:
        clean += 1
        by_lang[lang]["clean"] += 1

excluded = lang_mismatch + numeric_samples + both
print(f"\nExclusion breakdown:")
print(f"  Language mismatch only:  {lang_mismatch}")
print(f"  Numeric only:           {numeric_samples}")
print(f"  Both:                   {both}")
print(f"  Total excluded:         {excluded}")
print(f"  Clean samples:          {clean} ({clean/len(preds)*100:.1f}%)")

print(f"\nPer-language breakdown:")
print(f"  {'Lang':<12} {'Total':>6} {'Mismatch':>9} {'Numeric':>8} {'Clean':>6} {'Clean%':>7}")
print(f"  {'-'*12} {'-'*6} {'-'*9} {'-'*8} {'-'*6} {'-'*7}")
for lang in sorted(by_lang.keys()):
    d = by_lang[lang]
    pct = d["clean"] / d["total"] * 100 if d["total"] > 0 else 0
    print(f"  {lang:<12} {d['total']:>6} {d['mismatch']:>9} {d['numeric']:>8} {d['clean']:>6} {pct:>6.1f}%")

print(f"\nLanguage mismatch examples:")
for lang in sorted(by_lang.keys()):
    for ex in by_lang[lang]["mismatch_examples"]:
        print(f"  [{lang}] {ex['id']}: expected={ex['expected']} got={ex['got']}")
        print(f"    REF: {ex['ref']}")
        print(f"    HYP: {ex['hyp']}")
PYEOF

Analyze language mismatches and numeric content in ckpt-70000


Total samples: 6000

Exclusion breakdown:
  Language mismatch only:  242
  Numeric only:           86
  Both:                   16
  Total excluded:         344
  Clean samples:          5656 (94.3%)

Per-language breakdown:
  Lang          Total  Mismatch  Numeric  Clean  Clean%
  ------------ ------ --------- -------- ------ -------
  assamese        500         0        1    499   99.8%
  bengali         500         5        0    495   99.0%
  english         500        80       97    338   67.6%
  gujarati        500         5        0    495   99.0%
  hindi           500         4        0    496   99.2%
  kannada         500        44        2    455   91.0%
  malayalam       500         2        0    498   99.6%
  marathi         500         1        0    499   99.8%
  odia            500        42        2    456   91.2%
  punjabi         500        27        0    473   94.6%
  tamil           500        22        0    478   95.6%
  telugu          500        26        0    474   94.8%

Language mismatch examples:
  [bengali] 844424930442111-1037-f: expected=bengali got=devanagari
    REF: এ নিয়ে জেলায় মোট করোনা আক্রান্তের সংখ্যা দাঁড়ালো এক শো তে
    HYP: ए निये जिलाई मोट कोरोना आक्रान्तेर सँख्या ताणालो एक सौ तेईस
  [bengali] 844424931066612-99-m: expected=bengali got=devanagari
    REF: রামমন্দির নির্মাণে ট্রাস্টকে সাতষট্টি একর জমি দেওয়ার সিদ্ধা
    HYP: राम मंदिर निर्माणे ट्रस्ट के सातसठ एकर जमीन देवार सिद्धांत प
  [english] en_0005: expected=latin got=gujarati
    REF: Yes, when I had visited Taj Mahal
    HYP: યસ વેન ಐ હેડ ವಿಸಿಟೆಡ್ તાજમહેલ
  [english] en_0008: expected=latin got=devanagari
    REF: main medium of communication has been Gujarati
    HYP: मेन मीडियम ऑफ कम्युनिकेशन हैज बीन गुजराती
  [gujarati] 844424931450651-24-f: expected=gujarati got=devanagari
    REF: નીતિન પટેલે કહ્યું કે શાહીબાગમાં નવી કિડની હોસ્પિટલમાં કોવિડ
    HYP: नितिन पटेल ने कहा कि शाहीबाग में नई किडनी हॉस्पिटल में कोविड
  [gujarati] 844424933342436-1051-m: expected=gujarati got=devanagari
    REF: શિક્ષણ પૂર્ણ કરીને વિરમગામની બાજુમા આવેલ ઉખલોડ ગામમા સરકારી
    HYP: शिक्षण पूर्ण करीने विरमगामने बाजूमा आवेला उखलोळगाममा सरकारी
  [hindi] 844424932335105-384-f: expected=devanagari got=latin
    REF: सोनाक्षी इन दिनों एक सिंगिंग रियलिटी शो जज कर रही हैं
    HYP: Sonakshi In Dino ek singing reality show judge kar rahi hain
  [hindi] 844424930948730-277-f: expected=devanagari got=gurmukhi
    REF: पाक विदेश मंत्री शाह महमूद कुरैशी लंदन दौरे पर पहुंचे थे
    HYP: पाक विदेश मंत्री ਸ਼ਾਹ મહਮੂਦ ਕੁਰੈਸ਼ੀ ਲੰਦਨ ਦੌਰੇ પર પહોંચ્યા હત
  [kannada] 844424930483635-22-f: expected=kannada got=gujarati
    REF: ಗಣೇಶ ಚತುರ್ಥಿ ಒಂದು ತಿಂಗಳು ಇರುವಾಗಲೇ ನಗರಸಭೆ ಪರಿಸರ ಸ್ನೇಹಿ ಮಣ್ಣಿನ
    HYP: ಗಣೇಶ ಚತುರ್ಥी ಒಂದು ತಿಂಗಳು ಇರುವಾಗ ನಗರಸಭೆ परिसर સ્નેહમાં ન ગણેશ
  [kannada] 844424930483652-22-f: expected=kannada got=tamil
    REF: ನಾಲ್ಕೈದು ದಿನದ ಹಿಂದೆ ಮಳೆಯಿಂದಾಗಿ ರಾಡಿಯಾಗಿದ್ದ ಚಾರ್ಮಾಡಿ ಇದೀಗ ತನ್
    HYP: ನಾಲ್ಕೈದು ದಿನದ ಹಿಂದೆ மழೆಯಿಂದ கிராடியಾಗಿದ್ದ சர்மடிக்குத் திகத்
  [malayalam] 844424931007444-763-f: expected=malayalam got=tamil
    REF: ഏക് ദിന് രാത്രേ എന്ന പേരില് ജാഗ്തേ രഹോ ബംഗാളി ഭാഷയിലും പുറത്
    HYP: ഏക് ദിൻ രാത്രി എന്ന പീരൽ ജഗത്ത് ഥോ பெங்காலி பாஷையிலும் புறத்
  [malayalam] 844424931226439-1007-m: expected=malayalam got=tamil
    REF: ഒരാളുടെ മുഖം നോക്കി ബംഗ്ലാദേശി പൌരനാണോയെന്ന് തിരിച്ചറിയാന് സ
    HYP: ഒരാളുടെ முகம் நோக்கி பங்களாதேஷி பௌரனாணோ என்று തിരിച്ചறியான்
  [marathi] 844424933545453-360-m: expected=devanagari got=malayalam
    REF: व्हायरल विडीओझ हे व्हायरल मार्केटिंग बझ्झ मार्केटिंग आणि स्त
    HYP: व्हायरल व्हिडिओज हे व्हायरल मार्केटिंग ബസ് മാർക്കറ്റിംഗ് आणि
  [odia] 844424931549850-824-m: expected=oriya got=devanagari
    REF: ନିର୍ବାଚିତ ପ୍ରତିନିଧିମାନଙ୍କ ନାଁରେ ନିଖୋଜ ହେବା ନେଇ ପୋଷ୍ଟର ଲାଗିବା
    HYP: निर्वाचित प्रतिनिधिमानंक ना रे निखोज हेबार एइ पोस्टर लागिया
  [odia] 844424932876563-97-m: expected=oriya got=bengali
    REF: ବିଭିନ୍ନ ଜାତୀୟ ସାମ୍ବାଦିକ ୟୁନିୟନର ବି ସେ ସକ୍ରିୟ ସଦସ୍ୟ ଥିଲେ
    HYP: বিভিন্ন জাতীয় সাংবাদিক ইউনিয়ন আরও বিশেষ সক্রিয় সদস্য ছিলে
  [punjabi] 844424933629270-359-m: expected=gurmukhi got=devanagari
    REF: ਰਾਵਣ ਸਹਾਰਨਪੁਰ ਦੇ ਧਦਕੁਲੀ ਪਿੰਡ ਪੱਛਮੀ ਉੱਤਰ ਪ੍ਰਦੇਸ਼ ਦਾ ਰਹਿਣ ਵਾਲਾ
    HYP: रावण सहारनपुर दे तत्कुली पिंड पश्चिमे उत्तर प्रदेश दा रहन वा
  [punjabi] 844424933032038-775-f: expected=gurmukhi got=devanagari
    REF: ਪਤਾ ਕਰੋ ਕਿ ਤੁਸੀਂ ਕਿੰਨੀ ਕੁ ਮੱਛੀ ਫੜਨ ਦੀ ਯੋਜਨਾ ਬਣਾਉਂਦੇ ਹੋ
    HYP: पता करो कि तुसीं किन्नी कु मच्छी फड़न दी योजना बनाउंदे हो
  [tamil] 844424931306815-67-f: expected=tamil got=telugu
    REF: தருமபுரி அருகே நீட் தேர்வு எழுத தயாராகி வந்த மாணவன் ஆதித்யா
    HYP: ధర్మపురి అరికే నీట్ తేర్వు ఎళుద తాయారాగి వంద మానవన్ ఆదిత్య త
  [tamil] 844424930439466-781-f: expected=tamil got=telugu
    REF: பரிசளிப்பு விழாவில் வழங்கிக் கௌரவித்துள்ளோம் என்பதனைச் சங்க
    HYP: పరిసలిపు విళావిల్ వళంగి గౌరవిత్తుళ్ళం ఎన్బదై సంఘ నిర్వాగత్తి
  [telugu] 844424932692924-832-f: expected=telugu got=devanagari
    REF: లంబోర్గినీ కార్లను కొనుగోలు చేస్తున్న వారిలో ఎక్కువ మంది పార
    HYP: लंबर केने कारनु कनगोल चेस्तुन्न वारलो एक्कु मंदि पारिश्रमिक
  [telugu] 844424931269950-445-f: expected=telugu got=kannada
    REF: దీంతో అంతకు ముందు వరకు ఉన్న భారీ ల క్షం భారం వారిపై లేకుండా
    HYP: ಇಂತೋ ಅಂದಕು ಮುಂದು ವರಕು ಉನ್ನ ಬಾರೀಲಾಕ್ಷಂ ಬಾರಂ ಬಾರಿಪೈ ಲೇಕುಂಡಾ ಪೋ


Good data. 344 samples excluded (5.7%). Now let me recompute all metrics on the clean set.


python3 << 'PYEOF'
import json
import re
import unicodedata
from collections import defaultdict
import jiwer
import numpy as np

# Load predictions
with open('/home/ubuntu/training/benchmark_results_gemma3n/ckpt-70000/predictions.json') as f:
    preds = json.load(f)

# ── Same script/numeric detection as above ──────────────────────────────

SCRIPT_RANGES = {
    "devanagari": (0x0900, 0x097F), "bengali": (0x0980, 0x09FF),
    "gurmukhi": (0x0A00, 0x0A7F), "gujarati": (0x0A80, 0x0AFF),
    "oriya": (0x0B00, 0x0B7F), "tamil": (0x0B80, 0x0BFF),
    "telugu": (0x0C00, 0x0C7F), "kannada": (0x0C80, 0x0CFF),
    "malayalam": (0x0D00, 0x0D7F), "latin": (0x0041, 0x024F),
}
LANG_TO_SCRIPT = {
    "hindi": "devanagari", "marathi": "devanagari",
    "bengali": "bengali", "assamese": "bengali",
    "punjabi": "gurmukhi", "gujarati": "gujarati",
    "odia": "oriya", "tamil": "tamil", "telugu": "telugu",
    "kannada": "kannada", "malayalam": "malayalam", "english": "latin",
}

def dominant_script(text):
    counts = defaultdict(int)
    for ch in text:
        cp = ord(ch)
        for script, (lo, hi) in SCRIPT_RANGES.items():
            if lo <= cp <= hi:
                counts[script] += 1; break
    return max(counts, key=counts.get) if counts else "unknown"

DIGIT_PAT = re.compile(r'\d')
INDIC_DIGIT = re.compile(r'[\u0966-\u096F\u09E6-\u09EF\u0A66-\u0A6F\u0AE6-\u0AEF\u0B66-\u0B6F\u0BE6-\u0BEF\u0C66-\u0C6F\u0CE6-\u0CEF\u0D66-\u0D6F]')
EN_NUM = re.compile(r'\b(zero|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve|thirteen|fourteen|fifteen|sixteen|seventeen|eighteen|nineteen|twenty|thirty|forty|fifty|sixty|seventy|eighty|ninety|hundred|thousand|million|billion|lakh|crore|karod)\b', re.I)

def has_numerics(text):
    return bool(DIGIT_PAT.search(text) or INDIC_DIGIT.search(text) or EN_NUM.search(text))

# ── Normalization ───────────────────────────────────────────────────────

ZERO_WIDTH = re.compile(r"[\u200b-\u200f\u2028-\u202f\ufeff\u00ad]")
PUNCT = re.compile(r"[!\"#$%&'()*+,\-./:;<=>?@\[\\\]^_`{|}~\u0964\u0965\u2013\u2014\u2015\u2018\u2019\u201c\u201d\u2026\u00ab\u00bb\u0022\u0027]")

def norm_raw(t):
    return unicodedata.normalize("NFC", t).strip()

def norm_std(t):
    t = unicodedata.normalize("NFKC", t)
    t = ZERO_WIDTH.sub("", t)
    t = re.sub(r"\s+", " ", t).strip()
    t = PUNCT.sub("", t)
    t = re.sub(r"\s+", " ", t).strip()
    return t.lower()

# ── Filter samples ──────────────────────────────────────────────────────

clean_preds = []
excluded_preds = []
for p in preds:
    lang = p["language"]
    ref, hyp = p["reference"], p.get("hypothesis", "")
    expected = LANG_TO_SCRIPT.get(lang, "unknown")
    hyp_script = dominant_script(hyp) if hyp.strip() else "empty"
    is_mismatch = (hyp.strip() != "" and hyp_script != expected and hyp_script != "unknown")
    is_numeric = has_numerics(ref) or has_numerics(hyp)
    if is_mismatch or is_numeric:
        excluded_preds.append(p)
    else:
        clean_preds.append(p)

print(f"Clean: {len(clean_preds)} | Excluded: {len(excluded_preds)}")
print()

# ── Compute metrics on clean set ────────────────────────────────────────

by_lang = defaultdict(lambda: {"refs_raw": [], "hyps_raw": [], "refs_norm": [], "hyps_norm": []})

for p in clean_preds:
    lang = p["language"]
    ref, hyp = p["reference"], p.get("hypothesis", "")
    by_lang[lang]["refs_raw"].append(norm_raw(ref))
    by_lang[lang]["hyps_raw"].append(norm_raw(hyp))
    by_lang[lang]["refs_norm"].append(norm_std(ref))
    by_lang[lang]["hyps_norm"].append(norm_std(hyp))

# Also compute on full set for comparison
by_lang_full = defaultdict(lambda: {"refs_raw": [], "hyps_raw": [], "refs_norm": [], "hyps_norm": []})
for p in preds:
    lang = p["language"]
    ref, hyp = p["reference"], p.get("hypothesis", "")
    by_lang_full[lang]["refs_raw"].append(norm_raw(ref))
    by_lang_full[lang]["hyps_raw"].append(norm_raw(hyp))
    by_lang_full[lang]["refs_norm"].append(norm_std(ref))
    by_lang_full[lang]["hyps_norm"].append(norm_std(hyp))

print(f"{'Lang':<12} {'N_cln':>6} {'WER_raw':>8} {'WER_nrm':>8} {'CER_nrm':>8} {'delta':>7} | {'WER_raw*':>9} {'WER_nrm*':>9} {'CER_nrm*':>9} {'Improve':>8}")
print(f"{'-'*12} {'-'*6} {'-'*8} {'-'*8} {'-'*8} {'-'*7} | {'-'*9} {'-'*9} {'-'*9} {'-'*8}")

agg_clean = {"rr": [], "hr": [], "rn": [], "hn": []}
agg_full = {"rr": [], "hr": [], "rn": [], "hn": []}

for lang in sorted(by_lang.keys()):
    c = by_lang[lang]
    f = by_lang_full[lang]
    n = len(c["refs_raw"])

    wr_c = round(jiwer.wer(c["refs_raw"], c["hyps_raw"]) * 100, 2)
    wn_c = round(jiwer.wer(c["refs_norm"], c["hyps_norm"]) * 100, 2)
    cn_c = round(jiwer.cer(c["refs_norm"], c["hyps_norm"]) * 100, 2)
    d_c = round(wr_c - wn_c, 2)

    wr_f = round(jiwer.wer(f["refs_raw"], f["hyps_raw"]) * 100, 2)
    wn_f = round(jiwer.wer(f["refs_norm"], f["hyps_norm"]) * 100, 2)
    cn_f = round(jiwer.cer(f["refs_norm"], f["hyps_norm"]) * 100, 2)

    imp = round(wn_f - wn_c, 2)

    print(f"{lang:<12} {n:>6} {wr_c:>8} {wn_c:>8} {cn_c:>8} {d_c:>+7} | {wr_f:>9} {wn_f:>9} {cn_f:>9} {imp:>+8}")

    agg_clean["rr"].extend(c["refs_raw"]); agg_clean["hr"].extend(c["hyps_raw"])
    agg_clean["rn"].extend(c["refs_norm"]); agg_clean["hn"].extend(c["hyps_norm"])
    agg_full["rr"].extend(f["refs_raw"]); agg_full["hr"].extend(f["hyps_raw"])
    agg_full["rn"].extend(f["refs_norm"]); agg_full["hn"].extend(f["hyps_norm"])

# Overall
wr_c = round(jiwer.wer(agg_clean["rr"], agg_clean["hr"]) * 100, 2)
wn_c = round(jiwer.wer(agg_clean["rn"], agg_clean["hn"]) * 100, 2)
cn_c = round(jiwer.cer(agg_clean["rn"], agg_clean["hn"]) * 100, 2)
d_c = round(wr_c - wn_c, 2)

wr_f = round(jiwer.wer(agg_full["rr"], agg_full["hr"]) * 100, 2)
wn_f = round(jiwer.wer(agg_full["rn"], agg_full["hn"]) * 100, 2)
cn_f = round(jiwer.cer(agg_full["rn"], agg_full["hn"]) * 100, 2)
imp = round(wn_f - wn_c, 2)

print(f"{'-'*12} {'-'*6} {'-'*8} {'-'*8} {'-'*8} {'-'*7} | {'-'*9} {'-'*9} {'-'*9} {'-'*8}")
print(f"{'OVERALL':<12} {len(agg_clean['rr']):>6} {wr_c:>8} {wn_c:>8} {cn_c:>8} {d_c:>+7} | {wr_f:>9} {wn_f:>9} {cn_f:>9} {imp:>+8}")

# No-english clean
rr_ne, hr_ne, rn_ne, hn_ne = [], [], [], []
for lang in sorted(by_lang.keys()):
    if lang == "english": continue
    c = by_lang[lang]
    rr_ne.extend(c["refs_raw"]); hr_ne.extend(c["hyps_raw"])
    rn_ne.extend(c["refs_norm"]); hn_ne.extend(c["hyps_norm"])

wr_ne = round(jiwer.wer(rr_ne, hr_ne) * 100, 2)
wn_ne = round(jiwer.wer(rn_ne, hn_ne) * 100, 2)
cn_ne = round(jiwer.cer(rn_ne, hn_ne) * 100, 2)
d_ne = round(wr_ne - wn_ne, 2)
print(f"{'NO-ENGLISH':<12} {len(rr_ne):>6} {wr_ne:>8} {wn_ne:>8} {cn_ne:>8} {d_ne:>+7} |")

print()
print("Left columns = CLEAN (no mismatches, no numerics)")
print("Right columns (with *) = FULL (all 6000 samples)")
print("Improve = how much WER_norm drops when excluding noisy samples")
PYEOF

Recompute all metrics excluding mismatches and numerics


Clean: 5656 | Excluded: 344

Lang          N_cln  WER_raw  WER_nrm  CER_nrm   delta |  WER_raw*  WER_nrm*  CER_nrm*  Improve
------------ ------ -------- -------- -------- ------- | --------- --------- --------- --------
assamese        499    14.26    14.07     6.05   +0.19 |     14.25     14.06      6.05    -0.01
bengali         495    18.22    17.69     3.93   +0.53 |     19.06     18.53      4.78    +0.84
english         338    19.36     8.48     4.18  +10.88 |     33.57     23.92     20.69   +15.44
gujarati        495    16.75    16.75      5.0    +0.0 |     17.68     17.68      5.89    +0.93
hindi           496    11.85    11.85     3.76    +0.0 |     12.31     12.31      4.29    +0.46
kannada         455    32.91    32.91    13.09    +0.0 |     38.84     38.84     19.88    +5.93
malayalam       498    38.92    38.92     7.49    +0.0 |     39.13     39.13      7.68    +0.21
marathi         499     22.3     22.3     6.24    +0.0 |     22.41     22.41      6.39    +0.11
odia            456    26.53     26.3     8.09   +0.23 |     33.55     33.35     15.48    +7.05
punjabi         473    20.55    20.05     9.18    +0.5 |     24.39      23.9     13.29    +3.85
tamil           478    29.24    29.22     6.86   +0.02 |     32.14     32.12     10.31     +2.9
telugu          474    25.89    25.87     5.97   +0.02 |     29.38     29.36      9.82    +3.49
------------ ------ -------- -------- -------- ------- | --------- --------- --------- --------
OVERALL        5656    22.68    21.94     6.84   +0.74 |     25.91     25.03     10.32    +3.09
NO-ENGLISH     5318    22.88    22.75     6.97   +0.13 |

Left columns = CLEAN (no mismatches, no numerics)
Right columns (with *) = FULL (all 6000 samples)
Improve = how much WER_norm drops when excluding noisy samples