Error: Exit code 1

Downloading data:   0%|          | 0/140 [00:00<?, ?files/s]
Downloading data: 100%|██████████| 140/140 [00:00<00:00, 172351.79files/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 428 examples [00:00, 933.18 examples/s]
Generating train split: 856 examples [00:00, 1515.17 examples/s]
Generating train split: 1284 examples [00:00, 1921.92 examples/s]
Generating train split: 1712 examples [00:00, 2353.05 examples/s]
Generating train split: 2143 examples [00:01, 2231.69 examples/s]
Generating train split: 2571 examples [00:01, 1631.46 examples/s]
Generating train split: 2999 examples [00:01, 1819.51 examples/s]
Generating train split: 3427 examples [00:01, 2033.61 examples/s]
Generating train split: 3855 examples [00:01, 2264.56 examples/s]
Generating train split: 4286 examples [00:02, 2265.20 examples/s]
Generating train split: 4714 examples [00:02, 1618.45 examples/s]
Generating train split: 5142 examples [00:02, 1846.69 examples/s]
Generating train split: 5570 examples [00:02, 2041.08 examples/s]
Generating train split: 5998 examples [00:03, 2201.77 examples/s]
Generating train split: 6857 examples [00:03, 2051.61 examples/s]
Generating train split: 7285 examples [00:03, 2212.66 examples/s]
Generating train split: 7713 examples [00:03, 2398.31 examples/s]
Generating train split: 8141 examples [00:03, 2493.70 examples/s]
Generating train split: 9000 examples [00:04, 2191.02 examples/s]
Generating train split: 9428 examples [00:04, 2333.82 examples/s]
Generating train split: 9856 examples [00:04, 2522.99 examples/s]
Generating train split: 10284 examples [00:04, 2659.79 examples/s]
Generating train split: 11143 examples [00:05, 2266.50 examples/s]
Generating train split: 11571 examples [00:05, 2410.16 examples/s]
Generating train split: 11999 examples [00:05, 2466.04 examples/s]
Generating train split: 12427 examples [00:05, 2601.93 examples/s]
Generating train split: 13286 examples [00:06, 2085.41 examples/s]
Generating train split: 13714 examples [00:06, 2216.51 examples/s]
Generating train split: 14142 examples [00:06, 2377.52 examples/s]
Generating train split: 14570 examples [00:06, 2421.51 examples/s]
Generating train split: 15429 examples [00:07, 1959.75 examples/s]
Generating train split: 15857 examples [00:07, 1985.24 examples/s]
Generating train split: 16285 examples [00:07, 1968.26 examples/s]
Generating train split: 16713 examples [00:07, 1998.30 examples/s]
Generating train split: 17572 examples [00:08, 1835.66 examples/s]
Generating train split: 18000 examples [00:08, 1934.01 examples/s]
Generating train split: 18428 examples [00:08, 1990.97 examples/s]
Generating train split: 18856 examples [00:08, 2074.86 examples/s]
Generating train split: 19715 examples [00:09, 1994.38 examples/s]
Generating train split: 20143 examples [00:09, 2142.34 examples/s]
Generating train split: 20571 examples [00:09, 2269.00 examples/s]
Generating train split: 20999 examples [00:09, 2446.39 examples/s]
Generating train split: 21858 examples [00:10, 2249.27 examples/s]
Generating train split: 22286 examples [00:10, 2427.48 examples/s]
Generating train split: 22714 examples [00:10, 2520.92 examples/s]
Generating train split: 23142 examples [00:10, 2703.14 examples/s]
Generating train split: 24001 examples [00:11, 2114.87 examples/s]
Generating train split: 24429 examples [00:11, 2197.74 examples/s]
Generating train split: 24857 examples [00:11, 2392.80 examples/s]
Generating train split: 25285 examples [00:11, 2475.57 examples/s]
Generating train split: 26144 examples [00:12, 2019.82 examples/s]
Generating train split: 26572 examples [00:12, 2116.34 examples/s]
Generating train split: 27000 examples [00:12, 2209.80 examples/s]
Generating train split: 27428 examples [00:12, 2242.92 examples/s]
Generating train split: 28287 examples [00:13, 1997.20 examples/s]
Generating train split: 28715 examples [00:13, 2039.99 examples/s]
Generating train split: 29143 examples [00:13, 2064.99 examples/s]
Generating train split: 29571 examples [00:13, 2131.92 examples/s]
Generating train split: 30430 examples [00:14, 1923.04 examples/s]
Generating train split: 30858 examples [00:14, 1986.24 examples/s]
Generating train split: 31286 examples [00:14, 2121.49 examples/s]
Generating train split: 31714 examples [00:14, 2409.79 examples/s]
Generating train split: 32573 examples [00:15, 2073.58 examples/s]
Generating train split: 33001 examples [00:15, 2194.51 examples/s]
Generating train split: 33429 examples [00:15, 2246.23 examples/s]
Generating train split: 33857 examples [00:15, 2225.27 examples/s]
Generating train split: 34716 examples [00:16, 2182.80 examples/s]
Generating train split: 35144 examples [00:16, 2389.00 examples/s]
Generating train split: 35572 examples [00:16, 2520.46 examples/s]
Generating train split: 36000 examples [00:16, 2722.46 examples/s]
Generating train split: 36859 examples [00:16, 2464.23 examples/s]
Generating train split: 37287 examples [00:17, 2619.08 examp

... [19992 characters truncated] ...

1:37, 1880.26 examples/s]
Generating train split: 196297 examples [01:37, 1948.83 examples/s]
Generating train split: 196725 examples [01:38, 2031.83 examples/s]
Generating train split: 197584 examples [01:38, 1802.74 examples/s]
Generating train split: 198012 examples [01:38, 1894.12 examples/s]
Generating train split: 198440 examples [01:39, 1996.32 examples/s]
Generating train split: 198868 examples [01:39, 2034.11 examples/s]
Generating train split: 199727 examples [01:39, 1753.09 examples/s]
Generating train split: 200155 examples [01:40, 1802.34 examples/s]
Generating train split: 200583 examples [01:40, 1877.45 examples/s]
Generating train split: 201011 examples [01:40, 2017.45 examples/s]
Generating train split: 201870 examples [01:40, 1804.57 examples/s]
Generating train split: 202298 examples [01:41, 1890.97 examples/s]
Generating train split: 202726 examples [01:41, 2012.33 examples/s]
Generating train split: 203154 examples [01:41, 2051.26 examples/s]
Generating train split: 204013 examples [01:42, 1853.56 examples/s]
Generating train split: 204441 examples [01:42, 1928.20 examples/s]
Generating train split: 204869 examples [01:42, 2032.18 examples/s]
Generating train split: 205297 examples [01:42, 2098.63 examples/s]
Generating train split: 206156 examples [01:43, 1830.87 examples/s]
Generating train split: 206584 examples [01:43, 1936.54 examples/s]
Generating train split: 207012 examples [01:43, 2011.22 examples/s]
Generating train split: 207440 examples [01:43, 1988.27 examples/s]
Generating train split: 208299 examples [01:44, 1754.67 examples/s]
Generating train split: 208727 examples [01:44, 1856.55 examples/s]
Generating train split: 209155 examples [01:44, 1890.26 examples/s]
Generating train split: 209583 examples [01:44, 1959.56 examples/s]
Generating train split: 210442 examples [01:45, 1809.07 examples/s]
Generating train split: 210870 examples [01:45, 1911.75 examples/s]
Generating train split: 211298 examples [01:45, 1980.56 examples/s]
Generating train split: 211726 examples [01:46, 2006.04 examples/s]
Generating train split: 212585 examples [01:46, 1826.55 examples/s]
Generating train split: 213013 examples [01:46, 1918.35 examples/s]
Generating train split: 213441 examples [01:46, 1961.74 examples/s]
Generating train split: 213869 examples [01:47, 2075.70 examples/s]
Generating train split: 214728 examples [01:47, 1985.36 examples/s]
Generating train split: 215156 examples [01:47, 1985.19 examples/s]
Generating train split: 215584 examples [01:47, 2023.69 examples/s]
Generating train split: 216012 examples [01:48, 2055.20 examples/s]
Generating train split: 216871 examples [01:48, 1822.29 examples/s]
Generating train split: 217299 examples [01:48, 1886.20 examples/s]
Generating train split: 217727 examples [01:49, 1966.89 examples/s]
Generating train split: 218155 examples [01:49, 2006.56 examples/s]
Generating train split: 219014 examples [01:49, 1730.94 examples/s]
Generating train split: 219442 examples [01:50, 1815.14 examples/s]
Generating train split: 219870 examples [01:50, 1925.49 examples/s]
Generating train split: 220298 examples [01:50, 1982.59 examples/s]
Generating train split: 221157 examples [01:51, 1814.72 examples/s]
Generating train split: 221585 examples [01:51, 1815.39 examples/s]
Generating train split: 222013 examples [01:51, 1751.66 examples/s]
Generating train split: 222441 examples [01:51, 1866.80 examples/s]
Generating train split: 223300 examples [01:52, 1717.52 examples/s]
Generating train split: 223728 examples [01:52, 1796.35 examples/s]
Generating train split: 224156 examples [01:52, 1863.44 examples/s]
Generating train split: 224584 examples [01:52, 1798.42 examples/s]
Generating train split: 225443 examples [01:53, 1673.13 examples/s]
Generating train split: 225871 examples [01:53, 1743.69 examples/s]
Generating train split: 226299 examples [01:53, 1929.96 examples/s]
Generating train split: 226727 examples [01:53, 2128.49 examples/s]
Generating train split: 227586 examples [01:54, 1874.65 examples/s]
Generating train split: 228014 examples [01:54, 1948.67 examples/s]
Generating train split: 228442 examples [01:54, 1927.66 examples/s]
Generating train split: 228870 examples [01:55, 1694.21 examples/s]
Generating train split: 229729 examples [01:56, 1429.70 examples/s]
Generating train split: 230157 examples [01:56, 1392.99 examples/s]
Generating train split: 230585 examples [01:56, 1437.83 examples/s]
Generating train split: 231013 examples [01:56, 1509.89 examples/s]
Generating train split: 231872 examples [01:57, 1277.68 examples/s]
Generating train split: 232300 examples [01:58, 1189.26 examples/s]
Generating train split: 232728 examples [01:58, 1324.73 examples/s]
Generating train split: 233156 examples [01:58, 1414.65 examples/s]
Generating train split: 234015 examples [01:59, 1400.97 examples/s]
Generating train split: 234443 examples [01:59, 1468.36 examples/s]
Generating train split: 234871 examples [01:59, 1529.82 examples/s]
Generating


Killed — loading 300k samples with audio into RAM is too much. I'll use streaming mode instead. Let me write a proper transcription script.


[Request interrupted by user]


use vllm and use 85% utilisation and max throughput mode and give me an ETA. load few N samples at a time and perform batch wise.


Write

#!/usr/bin/env python3
"""
Transcribe indic-asr-validation-300k with Qwen3-ASR-mixed via vLLM.

Streams dataset in chunks, transcribes with vLLM, saves results incrementally.
Strips <|lang|> tags from reference text before saving.

Usage:
    python transcribe_300k.py \
        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
        --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
        --batch-size 64 --chunk-size 2000
"""

import argparse
import json
import os
import re
import sys
import time

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

import numpy as np
from datasets import load_dataset

sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")
LANG_CODE_TO_NAME = {
    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
}

def strip_lang_tag(text: str) -> str:
    return LANG_TAG_RE.sub("", text).strip()

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--output", required=True, help="Output JSONL path")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/datasets/indic-asr-validation-300k/data")
    parser.add_argument("--batch-size", type=int, default=64, help="Samples per vLLM batch")
    parser.add_argument("--chunk-size", type=int, default=2000, help="Samples loaded into RAM at a time")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
    parser.add_argument("--max-new-tokens", type=int, default=512)
    parser.add_argument("--resume", action="store_true", help="Resume from existing output")
    args = parser.parse_args()

    os.makedirs(os.path.dirname(args.output), exist_ok=True)

    # Count already done if resuming
    done_ids = set()
    if args.resume and os.path.exists(args.output):
        with open(args.output) as f:
            for line in f:
                r = json.loads(line)
                done_ids.add(r["segment_id"])
        print(f"Resuming: {len(done_ids)} samples already transcribed")

    # Load model
    from qwen_asr import Qwen3ASRModel
    print(f"Loading model: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.LLM(
        model=args.checkpoint,
        gpu_memory_utilization=args.gpu_memory_utilization,
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    print(f"Model loaded in {time.time() - t0:.1f}s")

    # Stream dataset
    print(f"Streaming dataset from {args.dataset_dir}")
    ds = load_dataset("parquet", data_dir=args.dataset_dir, split="train", streaming=True)

    total_done = len(done_ids)
    total_audio_sec = 0.0
    t_start = time.time()
    chunk_buf = []
    batch_count = 0

    outfile = open(args.output, "a" if args.resume else "w")

    for sample in ds:
        sid = sample["segment_id"]
        if sid in done_ids:
            continue

        chunk_buf.append(sample)

        if len(chunk_buf) >= args.chunk_size:
            # Process this chunk in batches
            total_done, total_audio_sec, batch_count = process_chunk(
                chunk_buf, model, outfile, args.batch_size,
                total_done, total_audio_sec, batch_count, t_start
            )
            chunk_buf = []

    # Process remaining
    if chunk_buf:
        total_done, total_audio_sec, batch_count = process_chunk(
            chunk_buf, model, outfile, args.batch_size,
            total_done, total_audio_sec, batch_count, t_start
        )

    outfile.close()
    elapsed = time.time() - t_start
    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")
    print(f"Output: {args.output}")

def process_chunk(chunk, model, outfile, batch_size, total_done, total_audio_sec, batch_count, t_start):
    """Process a chunk of samples in batches."""
    for i in range(0, len(chunk), batch_size):
        batch = chunk[i:i + batch_size]

        audio_inputs = []
        metas = []
        for s in batch:
            audio = s["audio"]
            wav = np.array(audio["array"], dtype=np.float32)
            sr = audio["sampling_rate"]
            audio_inputs.append((wav, sr))
            metas.append({
                "segment_id": s["segment_id"],
                "reference": strip_lang_tag(s["text"]),
                "duration": s["duration"],
                "lang": s["lang"],
                "source": s["source"],
            })

        # Force language per sample
        forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

        try:
            transcriptions = model.transcribe(
                audio=audio_inputs,
                language=forced_langs,
                return_time_stamps=False,
            )
        except Exception as e:
            print(f"\n  Batch failed ({e}), trying without forced lang...")
            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs, language=None, return_time_stamps=False,
                )
            except Exception as e2:
                print(f"  Batch FAILED: {e2}, skipping {len(batch)} samples")
                transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in batch]

        for j, t in enumerate(transcriptions):
            result = {
                **metas[j],
                "hypothesis": t.text,
                "detected_language": t.language,
            }
            outfile.write(json.dumps(result, ensure_ascii=False) + "\n")
            total_audio_sec += metas[j]["duration"]

        total_done += len(batch)
        batch_count += 1
        elapsed = time.time() - t_start
        rate = total_done / elapsed if elapsed > 0 else 0
        eta_sec = (300000 - total_done) / rate if rate > 0 else 0
        eta_min = eta_sec / 60

        if batch_count % 5 == 0:
            outfile.flush()
            rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
            print(f"  [{total_done:>7}/~300k] {rate:.0f} samples/s | "
                  f"RTF={rtf:.4f} | ETA={eta_min:.0f}min", flush=True)

    return total_done, total_audio_sec, batch_count

if __name__ == "__main__":
    main()


File created successfully at: /home/ubuntu/training/transcribe_300k.py

#!/usr/bin/env python3
"""
Transcribe indic-asr-validation-300k with Qwen3-ASR-mixed via vLLM.

Streams dataset in chunks, transcribes with vLLM, saves results incrementally.
Strips <|lang|> tags from reference text before saving.

Usage:
    python transcribe_300k.py \
        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
        --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
        --batch-size 64 --chunk-size 2000
"""

import argparse
import json
import os
import re
import sys
import time

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

import numpy as np
from datasets import load_dataset

sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")
LANG_CODE_TO_NAME = {
    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
}

def strip_lang_tag(text: str) -> str:
    return LANG_TAG_RE.sub("", text).strip()

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--output", required=True, help="Output JSONL path")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/datasets/indic-asr-validation-300k/data")
    parser.add_argument("--batch-size", type=int, default=64, help="Samples per vLLM batch")
    parser.add_argument("--chunk-size", type=int, default=2000, help="Samples loaded into RAM at a time")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
    parser.add_argument("--max-new-tokens", type=int, default=512)
    parser.add_argument("--resume", action="store_true", help="Resume from existing output")
    args = parser.parse_args()

    os.makedirs(os.path.dirname(args.output), exist_ok=True)

    # Count already done if resuming
    done_ids = set()
    if args.resume and os.path.exists(args.output):
        with open(args.output) as f:
            for line in f:
                r = json.loads(line)
                done_ids.add(r["segment_id"])
        print(f"Resuming: {len(done_ids)} samples already transcribed")

    # Load model
    from qwen_asr import Qwen3ASRModel
    print(f"Loading model: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.LLM(
        model=args.checkpoint,
        gpu_memory_utilization=args.gpu_memory_utilization,
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    print(f"Model loaded in {time.time() - t0:.1f}s")

    # Stream dataset
    print(f"Streaming dataset from {args.dataset_dir}")
    ds = load_dataset("parquet", data_dir=args.dataset_dir, split="train", streaming=True)

    total_done = len(done_ids)
    total_audio_sec = 0.0
    t_start = time.time()
    chunk_buf = []
    batch_count = 0

    outfile = open(args.output, "a" if args.resume else "w")

    for sample in ds:
        sid = sample["segment_id"]
        if sid in done_ids:
            continue

        chunk_buf.append(sample)

        if len(chunk_buf) >= args.chunk_size:
            # Process this chunk in batches
            total_done, total_audio_sec, batch_count = process_chunk(
                chunk_buf, model, outfile, args.batch_size,
                total_done, total_audio_sec, batch_count, t_start
            )
            chunk_buf = []

    # Process remaining
    if chunk_buf:
        total_done, total_audio_sec, batch_count = process_chunk(
            chunk_buf, model, outfile, args.batch_size,
            total_done, total_audio_sec, batch_count, t_start
        )

    outfile.close()
    elapsed = time.time() - t_start
    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")
    print(f"Output: {args.output}")

def process_chunk(chunk, model, outfile, batch_size, total_done, total_audio_sec, batch_count, t_start):
    """Process a chunk of samples in batches."""
    for i in range(0, len(chunk), batch_size):
        batch = chunk[i:i + batch_size]

        audio_inputs = []
        metas = []
        for s in batch:
            audio = s["audio"]
            wav = np.array(audio["array"], dtype=np.float32)
            sr = audio["sampling_rate"]
            audio_inputs.append((wav, sr))
            metas.append({
                "segment_id": s["segment_id"],
                "reference": strip_lang_tag(s["text"]),
                "duration": s["duration"],
                "lang": s["lang"],
                "source": s["source"],
            })

        # Force language per sample
        forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

        try:
            transcriptions = model.transcribe(
                audio=audio_inputs,
                language=forced_langs,
                return_time_stamps=False,
            )
        except Exception as e:
            print(f"\n  Batch failed ({e}), trying without forced lang...")
            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs, language=None, return_time_stamps=False,
                )
            except Exception as e2:
                print(f"  Batch FAILED: {e2}, skipping {len(batch)} samples")
                transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in batch]

        for j, t in enumerate(transcriptions):
            result = {
                **metas[j],
                "hypothesis": t.text,
                "detected_language": t.language,
            }
            outfile.write(json.dumps(result, ensure_ascii=False) + "\n")
            total_audio_sec += metas[j]["duration"]

        total_done += len(batch)
        batch_count += 1
        elapsed = time.time() - t_start
        rate = total_done / elapsed if elapsed > 0 else 0
        eta_sec = (300000 - total_done) / rate if rate > 0 else 0
        eta_min = eta_sec / 60

        if batch_count % 5 == 0:
            outfile.flush()
            rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
            print(f"  [{total_done:>7}/~300k] {rate:.0f} samples/s | "
                  f"RTF={rtf:.4f} | ETA={eta_min:.0f}min", flush=True)

    return total_done, total_audio_sec, batch_count

if __name__ == "__main__":
    main()


Bash

python3 /home/ubuntu/training/transcribe_300k.py \
    --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
    --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
    --batch-size 64 \
    --chunk-size 2000 \
    --gpu-memory-utilization 0.85 2>&1

Run 300k transcription with vLLM


Command running in background with ID: byv0kb9ce. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1

Check startup


Skipping import of cpp extensions due to incompatible torch version 2.9.1+cu128 for torchao version 0.16.0             Please see https://github.com/pytorch/ao/issues/2919 for more info


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1

Check startup


The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The tokenizer you are loading from '/home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
The tokenizer you are loading from '/home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.


grep "samples/s\|loaded\|Streaming\|DONE\|ETA" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1 | tail -10

Check progress


(Bash completed with no output)


grep "samples/s\|loaded\|Streaming\|DONE\|ETA\|Model loaded" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1 | tail -10

Check progress


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1

Check latest output


(EngineCore_DP0 pid=3324217) INFO 03-31 16:43:46 [qwen3_asr.py:414] thinker_config is None. Initializing thinker model with default values
(EngineCore_DP0 pid=3324217) INFO 03-31 16:43:47 [backends.py:916] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/3689007089/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3324217) INFO 03-31 16:43:47 [backends.py:976] Dynamo bytecode transform time: 7.12 s


Still compiling. Let me wait.


tail -3 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1

Check progress


(EngineCore_DP0 pid=3324217) INFO 03-31 16:43:54 [kv_cache_utils.py:1312] Maximum concurrency for 65,536 tokens per request: 8.55x
(EngineCore_DP0 pid=3324217)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   6%|▌         | 3/51 [00:00<00:01, 28.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  14%|█▎        | 7/51 [00:00<00:01, 29.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  22%|██▏       | 11/51 [00:00<00:01, 29.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  29%|██▉       | 15/51 [00:00<00:01, 30.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  35%|███▌      | 18/51 [00:00<00:01, 29.17it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 22/51 [00:00<00:00, 29.45it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  49%|████▉     | 25/51 [00:00<00:00, 29.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 28/51 [00:00<00:00, 29.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  63%|██████▎   | 32/51 [00:01<00:00, 29.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  69%|██████▊   | 35/51 [00:01<00:00, 29.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  75%|███████▍  | 38/51 [00:01<00:00, 29.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 42/51 [00:01<00:00, 29.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  90%|█████████ | 46/51 [00:01<00:00, 30.02it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  96%|█████████▌| 49/51 [00:01<00:00, 28.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:01<00:00, 29.33it/s]
(EngineCore_DP0 pid=3324217)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   9%|▊         | 3/35 [00:00<00:01, 25.57it/s]


CUDA graph capture in progress. Almost ready.


grep "samples/s\|Model loaded\|Streaming\|ETA" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1 | tail -5

Check inference start


Model loaded in 45.5s
Streaming dataset from /home/ubuntu/training/datasets/indic-asr-validation-300k/data


<task-notification>
<task-id>byv0kb9ce</task-id>
<tool-use-id>toolu_01S1BjRwgh6qbpyVZLL9pYgT</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output</output-file>
<status>failed</status>
<summary>Background command "Run 300k transcription with vLLM" failed with exit code 1</summary>
</task-notification>


grep "samples/s\|ETA" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1 | tail -5

Check throughput


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check how many samples transcribed


0 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl


Still loading the first chunk from streaming. Let me wait.


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check transcription count


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check transcription count


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/byv0kb9ce.output 2>&1

Check latest output


  File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/_ops.py", line 841, in __call__
    return self._op(*args, **kwargs)
RuntimeError: Resource temporarily unavailable
[rank0]:[W331 16:44:03.640304998 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
ERROR 03-31 16:44:03 [core_client.py:616] Engine core proc EngineCore_DP0 died unexpectedly, shutting down client.


Engine crashed — `Resource temporarily unavailable`. Likely too many audio decoding threads alongside vLLM. Let me reduce chunk size and use a smaller batch.


# Kill any leftover processes
pkill -f transcribe_300k 2>/dev/null
# Clear the empty output
> /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl

# Retry with smaller batch and ulimit fix
ulimit -n 65536 2>/dev/null
python3 /home/ubuntu/training/transcribe_300k.py \
    --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
    --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
    --batch-size 32 \
    --chunk-size 1000 \
    --gpu-memory-utilization 0.85 2>&1

Retry with smaller batch, ulimit fix


Command running in background with ID: bitxsc01h. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output


<task-notification>
<task-id>bitxsc01h</task-id>
<tool-use-id>toolu_017Jaw7kAFUsZRBR2RFRLoXb</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output</output-file>
<status>failed</status>
<summary>Background command "Retry with smaller batch, ulimit fix" failed with exit code 144</summary>
</task-notification>


sleep 60 && wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1 && grep "ETA\|samples/s" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output 2>&1 | tail -5

Wait 60s then check progress


tail -5 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output 2>&1

Check status


cat /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output 2>&1 | tail -10

Check status


ps aux | grep transcribe_300k | grep -v grep 2>&1

Check if running


Process died silently. The issue is likely the streaming dataset audio decoding eating too much memory. Let me take a different approach — pre-extract audio from parquet files directly and feed to vLLM in a more controlled way.


# Check what killed it
ls -la /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output 2>&1

Check output file


-rw-rw-r-- 1 ubuntu ubuntu 0 Mar 31 16:45 /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bitxsc01h.output


Zero bytes — crashed before any output. The previous vLLM engine process may still be holding GPU. Let me clean up and try a simpler approach — process one parquet shard at a time.


nvidia-smi --query-compute-apps=pid,name,used_gpu_memory --format=csv 2>&1

Check GPU


pid, process_name, used_gpu_memory [MiB]


GPU is free. The crash was from the previous vLLM instance not cleaning up properly. Let me rewrite to process parquet files one at a time — simpler and more robust.


Write

#!/usr/bin/env python3
"""
Transcribe indic-asr-validation-300k with Qwen3-ASR-mixed via vLLM.

Processes parquet shards one at a time to control memory.
Saves transcriptions as JSONL incrementally with --resume support.

Usage:
    python transcribe_300k.py \
        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
        --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
        --batch-size 32
"""

import argparse
import glob
import json
import os
import re
import sys
import time

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

import numpy as np
import pandas as pd
import soundfile as sf
from io import BytesIO

sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")
LANG_CODE_TO_NAME = {
    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
}

def strip_lang_tag(text: str) -> str:
    return LANG_TAG_RE.sub("", text).strip()

def decode_audio(audio_dict):
    """Decode audio from parquet row. Returns (wav_array, sample_rate)."""
    if isinstance(audio_dict, dict):
        if "array" in audio_dict:
            return np.array(audio_dict["array"], dtype=np.float32), audio_dict.get("sampling_rate", 16000)
        if "bytes" in audio_dict and audio_dict["bytes"]:
            wav, sr = sf.read(BytesIO(audio_dict["bytes"]))
            return wav.astype(np.float32), sr
        if "path" in audio_dict and audio_dict["path"]:
            wav, sr = sf.read(audio_dict["path"])
            return wav.astype(np.float32), sr
    raise ValueError(f"Cannot decode audio: {type(audio_dict)}")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--output", required=True)
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/datasets/indic-asr-validation-300k/data")
    parser.add_argument("--batch-size", type=int, default=32)
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
    parser.add_argument("--max-new-tokens", type=int, default=512)
    parser.add_argument("--resume", action="store_true")
    args = parser.parse_args()

    os.makedirs(os.path.dirname(args.output), exist_ok=True)

    # Get parquet files
    parquet_files = sorted(glob.glob(os.path.join(args.dataset_dir, "*.parquet")))
    print(f"Found {len(parquet_files)} parquet shards")

    # Resume support
    done_ids = set()
    if args.resume and os.path.exists(args.output):
        with open(args.output) as f:
            for line in f:
                done_ids.add(json.loads(line)["segment_id"])
        print(f"Resuming: {len(done_ids)} already done")

    # Load model
    from qwen_asr import Qwen3ASRModel
    print(f"Loading model: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.LLM(
        model=args.checkpoint,
        gpu_memory_utilization=args.gpu_memory_utilization,
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    print(f"Model loaded in {time.time() - t0:.1f}s")

    total_done = len(done_ids)
    total_audio_sec = 0.0
    t_start = time.time()
    outfile = open(args.output, "a" if args.resume else "w")

    for shard_idx, pf in enumerate(parquet_files):
        shard_name = os.path.basename(pf)
        df = pd.read_parquet(pf)
        print(f"\n[Shard {shard_idx+1}/{len(parquet_files)}] {shard_name}: {len(df)} rows")

        # Filter already done
        if done_ids:
            mask = ~df["segment_id"].isin(done_ids)
            df = df[mask].reset_index(drop=True)
            if len(df) == 0:
                print(f"  All done, skipping")
                continue

        # Process in batches
        for batch_start in range(0, len(df), args.batch_size):
            batch_df = df.iloc[batch_start:batch_start + args.batch_size]

            audio_inputs = []
            metas = []
            skip_count = 0
            for _, row in batch_df.iterrows():
                try:
                    wav, sr = decode_audio(row["audio"])
                    audio_inputs.append((wav, sr))
                    metas.append({
                        "segment_id": row["segment_id"],
                        "reference": strip_lang_tag(str(row["text"])),
                        "duration": float(row["duration"]),
                        "lang": row["lang"],
                        "source": row.get("source", ""),
                    })
                except Exception as e:
                    skip_count += 1
                    continue

            if not audio_inputs:
                continue

            forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs,
                    language=forced_langs,
                    return_time_stamps=False,
                )
            except Exception as e:
                print(f"  Batch failed ({e}), trying without forced lang...")
                try:
                    transcriptions = model.transcribe(
                        audio=audio_inputs, language=None, return_time_stamps=False,
                    )
                except Exception as e2:
                    print(f"  FAILED: {e2}, writing empty hyps")
                    transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in metas]

            for j, t in enumerate(transcriptions):
                result = {**metas[j], "hypothesis": t.text, "detected_language": t.language}
                outfile.write(json.dumps(result, ensure_ascii=False) + "\n")
                total_audio_sec += metas[j]["duration"]

            total_done += len(metas)
            elapsed = time.time() - t_start
            rate = total_done / elapsed if elapsed > 0 else 0
            eta_min = (300000 - total_done) / rate / 60 if rate > 0 else 0

            print(f"  [{total_done:>7}/~300k] {rate:.0f} samp/s | ETA={eta_min:.0f}min", end="\r", flush=True)

        outfile.flush()
        elapsed = time.time() - t_start
        rate = total_done / elapsed if elapsed > 0 else 0
        eta_min = (300000 - total_done) / rate / 60 if rate > 0 else 0
        rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
        print(f"\n  Shard done. Total={total_done} | {rate:.0f} samp/s | RTF={rtf:.4f} | ETA={eta_min:.0f}min")

    outfile.close()
    elapsed = time.time() - t_start
    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")

if __name__ == "__main__":
    main()


The file /home/ubuntu/training/transcribe_300k.py has been updated successfully.

#!/usr/bin/env python3
"""
Transcribe indic-asr-validation-300k with Qwen3-ASR-mixed via vLLM.

Processes parquet shards one at a time to control memory.
Saves transcriptions as JSONL incrementally with --resume support.

Usage:
    python transcribe_300k.py \
        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
        --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
        --batch-size 32
"""

import argparse
import glob
import json
import os
import re
import sys
import time

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

import numpy as np
import pandas as pd
import soundfile as sf
from io import BytesIO

sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")
LANG_CODE_TO_NAME = {
    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
}

def strip_lang_tag(text: str) -> str:
    return LANG_TAG_RE.sub("", text).strip()

def decode_audio(audio_dict):
    """Decode audio from parquet row. Returns (wav_array, sample_rate)."""
    if isinstance(audio_dict, dict):
        if "array" in audio_dict:
            return np.array(audio_dict["array"], dtype=np.float32), audio_dict.get("sampling_rate", 16000)
        if "bytes" in audio_dict and audio_dict["bytes"]:
            wav, sr = sf.read(BytesIO(audio_dict["bytes"]))
            return wav.astype(np.float32), sr
        if "path" in audio_dict and audio_dict["path"]:
            wav, sr = sf.read(audio_dict["path"])
            return wav.astype(np.float32), sr
    raise ValueError(f"Cannot decode audio: {type(audio_dict)}")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--output", required=True)
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/datasets/indic-asr-validation-300k/data")
    parser.add_argument("--batch-size", type=int, default=32)
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
    parser.add_argument("--max-new-tokens", type=int, default=512)
    parser.add_argument("--resume", action="store_true")
    args = parser.parse_args()

    os.makedirs(os.path.dirname(args.output), exist_ok=True)

    # Get parquet files
    parquet_files = sorted(glob.glob(os.path.join(args.dataset_dir, "*.parquet")))
    print(f"Found {len(parquet_files)} parquet shards")

    # Resume support
    done_ids = set()
    if args.resume and os.path.exists(args.output):
        with open(args.output) as f:
            for line in f:
                done_ids.add(json.loads(line)["segment_id"])
        print(f"Resuming: {len(done_ids)} already done")

    # Load model
    from qwen_asr import Qwen3ASRModel
    print(f"Loading model: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.LLM(
        model=args.checkpoint,
        gpu_memory_utilization=args.gpu_memory_utilization,
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    print(f"Model loaded in {time.time() - t0:.1f}s")

    total_done = len(done_ids)
    total_audio_sec = 0.0
    t_start = time.time()
    outfile = open(args.output, "a" if args.resume else "w")

    for shard_idx, pf in enumerate(parquet_files):
        shard_name = os.path.basename(pf)
        df = pd.read_parquet(pf)
        print(f"\n[Shard {shard_idx+1}/{len(parquet_files)}] {shard_name}: {len(df)} rows")

        # Filter already done
        if done_ids:
            mask = ~df["segment_id"].isin(done_ids)
            df = df[mask].reset_index(drop=True)
            if len(df) == 0:
                print(f"  All done, skipping")
                continue

        # Process in batches
        for batch_start in range(0, len(df), args.batch_size):
            batch_df = df.iloc[batch_start:batch_start + args.batch_size]

            audio_inputs = []
            metas = []
            skip_count = 0
            for _, row in batch_df.iterrows():
                try:
                    wav, sr = decode_audio(row["audio"])
                    audio_inputs.append((wav, sr))
                    metas.append({
                        "segment_id": row["segment_id"],
                        "reference": strip_lang_tag(str(row["text"])),
                        "duration": float(row["duration"]),
                        "lang": row["lang"],
                        "source": row.get("source", ""),
                    })
                except Exception as e:
                    skip_count += 1
                    continue

            if not audio_inputs:
                continue

            forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs,
                    language=forced_langs,
                    return_time_stamps=False,
                )
            except Exception as e:
                print(f"  Batch failed ({e}), trying without forced lang...")
                try:
                    transcriptions = model.transcribe(
                        audio=audio_inputs, language=None, return_time_stamps=False,
                    )
                except Exception as e2:
                    print(f"  FAILED: {e2}, writing empty hyps")
                    transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in metas]

            for j, t in enumerate(transcriptions):
                result = {**metas[j], "hypothesis": t.text, "detected_language": t.language}
                outfile.write(json.dumps(result, ensure_ascii=False) + "\n")
                total_audio_sec += metas[j]["duration"]

            total_done += len(metas)
            elapsed = time.time() - t_start
            rate = total_done / elapsed if elapsed > 0 else 0
            eta_min = (300000 - total_done) / rate / 60 if rate > 0 else 0

            print(f"  [{total_done:>7}/~300k] {rate:.0f} samp/s | ETA={eta_min:.0f}min", end="\r", flush=True)

        outfile.flush()
        elapsed = time.time() - t_start
        rate = total_done / elapsed if elapsed > 0 else 0
        eta_min = (300000 - total_done) / rate / 60 if rate > 0 else 0
        rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
        print(f"\n  Shard done. Total={total_done} | {rate:.0f} samp/s | RTF={rtf:.4f} | ETA={eta_min:.0f}min")

    outfile.close()
    elapsed = time.time() - t_start
    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")

if __name__ == "__main__":
    main()

-Streams dataset in chunks, transcribes with vLLM, saves results incrementally.

-Strips <|lang|> tags from reference text before saving.

+Processes parquet shards one at a time to control memory.

+Saves transcriptions as JSONL incrementally with --resume support.

-        --batch-size 64 --chunk-size 2000

+    """Decode audio from parquet row. Returns (wav_array, sample_rate)."""

+            return np.array(audio_dict["array"], dtype=np.float32), audio_dict.get("sampling_rate", 16000)

+        if "bytes" in audio_dict and audio_dict["bytes"]:

+            wav, sr = sf.read(BytesIO(audio_dict["bytes"]))

+            return wav.astype(np.float32), sr

+        if "path" in audio_dict and audio_dict["path"]:

+            wav, sr = sf.read(audio_dict["path"])

+    raise ValueError(f"Cannot decode audio: {type(audio_dict)}")

-    parser.add_argument("--output", required=True, help="Output JSONL path")

+    parser.add_argument("--output", required=True)

-    parser.add_argument("--batch-size", type=int, default=64, help="Samples per vLLM batch")

-    parser.add_argument("--chunk-size", type=int, default=2000, help="Samples loaded into RAM at a time")

+    parser.add_argument("--batch-size", type=int, default=32)

-    parser.add_argument("--resume", action="store_true", help="Resume from existing output")

+    parser.add_argument("--resume", action="store_true")

+    parquet_files = sorted(glob.glob(os.path.join(args.dataset_dir, "*.parquet")))

+    print(f"Found {len(parquet_files)} parquet shards")

if args.resume and os.path.exists(args.output):

-                done_ids.add(r["segment_id"])

-        print(f"Resuming: {len(done_ids)} samples already transcribed")

+                done_ids.add(json.loads(line)["segment_id"])

+        print(f"Resuming: {len(done_ids)} already done")

-    print(f"Streaming dataset from {args.dataset_dir}")

-    ds = load_dataset("parquet", data_dir=args.dataset_dir, split="train", streaming=True)

+    for shard_idx, pf in enumerate(parquet_files):

+        shard_name = os.path.basename(pf)

+        print(f"\n[Shard {shard_idx+1}/{len(parquet_files)}] {shard_name}: {len(df)} rows")

+            mask = ~df["segment_id"].isin(done_ids)

+            df = df[mask].reset_index(drop=True)

+                print(f"  All done, skipping")

-        if len(chunk_buf) >= args.chunk_size:

-            # Process this chunk in batches

-            total_done, total_audio_sec, batch_count = process_chunk(

-                chunk_buf, model, outfile, args.batch_size,

-                total_done, total_audio_sec, batch_count, t_start

+        for batch_start in range(0, len(df), args.batch_size):

+            batch_df = df.iloc[batch_start:batch_start + args.batch_size]

-            chunk_buf, model, outfile, args.batch_size,

+            for _, row in batch_df.iterrows():

+                    wav, sr = decode_audio(row["audio"])

+                    audio_inputs.append((wav, sr))

+                        "segment_id": row["segment_id"],

+                        "reference": strip_lang_tag(str(row["text"])),

+                        "duration": float(row["duration"]),

+                        "lang": row["lang"],

+                        "source": row.get("source", ""),

-    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0

-    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")

+            forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

-def process_chunk(chunk, model, outfile, batch_size, total_done, total_audio_sec, batch_count, t_start):

-    """Process a chunk of samples in batches."""

-    for i in range(0, len(chunk), batch_size):

-        batch = chunk[i:i + batch_size]

-            wav = np.array(audio["array"], dtype=np.float32)

-            sr = audio["sampling_rate"]

-            audio_inputs.append((wav, sr))

-                "segment_id": s["segment_id"],

-                "reference": strip_lang_tag(s["text"]),

-                "duration": s["duration"],

-        forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

-            transcriptions = model.transcribe(

-                return_time_stamps=False,

-            print(f"\n  Batch failed ({e}), trying without forced lang...")

-                    audio=audio_inputs, language=None, return_time_stamps=False,

+                    audio=audio_inputs,

+                    language=forced_langs,

+                    return_time_stamps=False,

-                print(f"  Batch FAILED: {e2}, skipping {len(batch)} samples")

-                transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in batch]

+                print(f"  Batch failed ({e}), trying without forced lang...")

+                    transcriptions = model.transcribe(

+                        audio=audio_inputs, language=None, return_time_stamps=False,

+                except Exception as e2:

+                    print(f"  FAILED: {e2}, writing empty hyps")

+                    transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in metas]

-        for j, t in enumerate(transcriptions):

-                "detected_language": t.language,

-            outfile.write(json.dumps(result, ensure_ascii=False) + "\n")

-            total_audio_sec += metas[j]["duration"]

+            for j, t in enumerate(transcriptions):

+                result = {**metas[j], "hypothesis": t.text, "detected_language": t.language}

+                outfile.write(json.dumps(result, ensure_ascii=False) + "\n")

+                total_audio_sec += metas[j]["duration"]

+            elapsed = time.time() - t_start

+            rate = total_done / elapsed if elapsed > 0 else 0

+            eta_min = (300000 - total_done) / rate / 60 if rate > 0 else 0

+            print(f"  [{total_done:>7}/~300k] {rate:.0f} samp/s | ETA={eta_min:.0f}min", end="\r", flush=True)

-        eta_sec = (300000 - total_done) / rate if rate > 0 else 0

+        rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0

+        print(f"\n  Shard done. Total={total_done} | {rate:.0f} samp/s | RTF={rtf:.4f} | ETA={eta_min:.0f}min")

-            print(f"  [{total_done:>7}/~300k] {rate:.0f} samples/s | "

-                  f"RTF={rtf:.4f} | ETA={eta_min:.0f}min", flush=True)

+    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")

-    return total_done, total_audio_sec, batch_count

#!/usr/bin/env python3
"""
Transcribe indic-asr-validation-300k with Qwen3-ASR-mixed via vLLM.

Streams dataset in chunks, transcribes with vLLM, saves results incrementally.
Strips <|lang|> tags from reference text before saving.

Usage:
    python transcribe_300k.py \
        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
        --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
        --batch-size 64 --chunk-size 2000
"""

import argparse
import json
import os
import re
import sys
import time

os.environ.setdefault("VLLM_WORKER_MULTIPROC_METHOD", "spawn")

import numpy as np
from datasets import load_dataset

sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")
LANG_CODE_TO_NAME = {
    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
}

def strip_lang_tag(text: str) -> str:
    return LANG_TAG_RE.sub("", text).strip()

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--checkpoint", required=True)
    parser.add_argument("--output", required=True, help="Output JSONL path")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/datasets/indic-asr-validation-300k/data")
    parser.add_argument("--batch-size", type=int, default=64, help="Samples per vLLM batch")
    parser.add_argument("--chunk-size", type=int, default=2000, help="Samples loaded into RAM at a time")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85)
    parser.add_argument("--max-new-tokens", type=int, default=512)
    parser.add_argument("--resume", action="store_true", help="Resume from existing output")
    args = parser.parse_args()

    os.makedirs(os.path.dirname(args.output), exist_ok=True)

    # Count already done if resuming
    done_ids = set()
    if args.resume and os.path.exists(args.output):
        with open(args.output) as f:
            for line in f:
                r = json.loads(line)
                done_ids.add(r["segment_id"])
        print(f"Resuming: {len(done_ids)} samples already transcribed")

    # Load model
    from qwen_asr import Qwen3ASRModel
    print(f"Loading model: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.LLM(
        model=args.checkpoint,
        gpu_memory_utilization=args.gpu_memory_utilization,
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    print(f"Model loaded in {time.time() - t0:.1f}s")

    # Stream dataset
    print(f"Streaming dataset from {args.dataset_dir}")
    ds = load_dataset("parquet", data_dir=args.dataset_dir, split="train", streaming=True)

    total_done = len(done_ids)
    total_audio_sec = 0.0
    t_start = time.time()
    chunk_buf = []
    batch_count = 0

    outfile = open(args.output, "a" if args.resume else "w")

    for sample in ds:
        sid = sample["segment_id"]
        if sid in done_ids:
            continue

        chunk_buf.append(sample)

        if len(chunk_buf) >= args.chunk_size:
            # Process this chunk in batches
            total_done, total_audio_sec, batch_count = process_chunk(
                chunk_buf, model, outfile, args.batch_size,
                total_done, total_audio_sec, batch_count, t_start
            )
            chunk_buf = []

    # Process remaining
    if chunk_buf:
        total_done, total_audio_sec, batch_count = process_chunk(
            chunk_buf, model, outfile, args.batch_size,
            total_done, total_audio_sec, batch_count, t_start
        )

    outfile.close()
    elapsed = time.time() - t_start
    rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
    print(f"\nDONE: {total_done} samples in {elapsed:.0f}s ({total_audio_sec:.0f}s audio, RTF={rtf:.4f})")
    print(f"Output: {args.output}")

def process_chunk(chunk, model, outfile, batch_size, total_done, total_audio_sec, batch_count, t_start):
    """Process a chunk of samples in batches."""
    for i in range(0, len(chunk), batch_size):
        batch = chunk[i:i + batch_size]

        audio_inputs = []
        metas = []
        for s in batch:
            audio = s["audio"]
            wav = np.array(audio["array"], dtype=np.float32)
            sr = audio["sampling_rate"]
            audio_inputs.append((wav, sr))
            metas.append({
                "segment_id": s["segment_id"],
                "reference": strip_lang_tag(s["text"]),
                "duration": s["duration"],
                "lang": s["lang"],
                "source": s["source"],
            })

        # Force language per sample
        forced_langs = [LANG_CODE_TO_NAME.get(m["lang"]) for m in metas]

        try:
            transcriptions = model.transcribe(
                audio=audio_inputs,
                language=forced_langs,
                return_time_stamps=False,
            )
        except Exception as e:
            print(f"\n  Batch failed ({e}), trying without forced lang...")
            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs, language=None, return_time_stamps=False,
                )
            except Exception as e2:
                print(f"  Batch FAILED: {e2}, skipping {len(batch)} samples")
                transcriptions = [type('T', (), {'text': '', 'language': ''})() for _ in batch]

        for j, t in enumerate(transcriptions):
            result = {
                **metas[j],
                "hypothesis": t.text,
                "detected_language": t.language,
            }
            outfile.write(json.dumps(result, ensure_ascii=False) + "\n")
            total_audio_sec += metas[j]["duration"]

        total_done += len(batch)
        batch_count += 1
        elapsed = time.time() - t_start
        rate = total_done / elapsed if elapsed > 0 else 0
        eta_sec = (300000 - total_done) / rate if rate > 0 else 0
        eta_min = eta_sec / 60

        if batch_count % 5 == 0:
            outfile.flush()
            rtf = elapsed / total_audio_sec if total_audio_sec > 0 else 0
            print(f"  [{total_done:>7}/~300k] {rate:.0f} samples/s | "
                  f"RTF={rtf:.4f} | ETA={eta_min:.0f}min", flush=True)

    return total_done, total_audio_sec, batch_count

if __name__ == "__main__":
    main()


Bash

> /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl && python3 /home/ubuntu/training/transcribe_300k.py \
    --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-120000 \
    --output /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl \
    --batch-size 32 \
    --gpu-memory-utilization 0.85 2>&1

Run 300k transcription (parquet shard approach)


Command running in background with ID: bq4txtl6u. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bq4txtl6u.output


sleep 90 && wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1 && grep "samp/s\|ETA\|Shard done\|Model loaded" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bq4txtl6u.output 2>&1 | tail -5

Wait 90s then check


890 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl
Model loaded in 41.2s
  [     32/~300k] 5 samp/s | ETA=958min
  [     64/~300k] 9 samp/s | ETA=580min
  [     96/~300k] 11 samp/s | ETA=472min
  [    128/~300k] 12 samp/s | ETA=419min
  [    160/~300k] 12 samp/s | ETA=400min
  [    192/~300k] 13 samp/s | ETA=387min
  [    224/~300k] 14 samp/s | ETA=368min
  [    256/~300k] 14 samp/s | ETA=348min
  [    288/~300k] 15 samp/s | ETA=341min
  [    320/~300k] 15 samp/s | ETA=333min
  [    352/~300k] 15 samp/s | ETA=323min
  [    384/~300k] 16 samp/s | ETA=317min
  [    416/~300k] 16 samp/s | ETA=311min
  [    448/~300k] 16 samp/s | ETA=307min
  [    480/~300k] 16 samp/s | ETA=304min
  [    512/~300k] 16 samp/s | ETA=303min
  [    544/~300k] 16 samp/s | ETA=303min
  [    576/~300k] 17 samp/s | ETA=300min
  [    608/~300k] 17 samp/s | ETA=298min
  [    640/~300k] 17 samp/s | ETA=295min
  [    672/~300k] 17 samp/s | ETA=295min
  [    704/~300k] 17 samp/s | ETA=293min
  [    736/~300k] 17 samp/s | ETA=288min
  [    768/~300k] 17 samp/s | ETA=289min
  [    800/~300k] 17 samp/s | ETA=289min
  [    832/~300k] 17 samp/s | ETA=288min
  [    864/~300k] 17 samp/s | ETA=287min
  [    896/~300k] 17 samp/s | ETA=286min


It's running. 890 samples done, ~17 samples/sec, **ETA ~285 minutes (~4.7 hours)**. The bottleneck is audio decoding from parquet (not GPU inference).

The run is stable with `--resume` support, so if it gets interrupted it can pick up where it left off. I'll keep monitoring.


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1 && grep "Shard done" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bq4txtl6u.output 2>&1 | tail -3

Check progress


1114 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check progress


1337 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check progress


1501 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

Check progress


1656 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl


Steady at ~17 samples/sec. 1656/~300k done. Running in background — **ETA ~4.5 hours**. The script has `--resume` support so it's safe if interrupted. I'll be notified when it completes.


Those are the earlier failed attempts — expected. The current working run (`bq4txtl6u`) is still going strong at ~17 samples/sec. It'll run for ~4.5 hours.


status


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1 && grep "Shard done" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bq4txtl6u.output 2>&1 | tail -3

Check current progress


42375 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl
  Shard done. Total=36431 | 18 samp/s | RTF=0.0085 | ETA=245min
  Shard done. Total=38574 | 18 samp/s | RTF=0.0085 | ETA=242min
  Shard done. Total=40717 | 18 samp/s | RTF=0.0085 | ETA=240min


**42,375 / ~300k** done (14%). Running at **18 samples/sec**, RTF 0.0085. **ETA ~240 minutes (~4 hours)**.


can i check live status of the scores.


python3 << 'PYEOF'
import json, re, unicodedata
from collections import defaultdict
from jiwer import wer, cer

LANG_TAG_RE = re.compile(r"^<\|[a-z]{2}\|>\s*")

def norm(t):
    t = unicodedata.normalize("NFC", t).lower()
    t = re.sub(r"[^\w\s]", "", t, flags=re.UNICODE)
    return re.sub(r"\s+", " ", t).strip()

by_lang = defaultdict(lambda: {"rr": [], "hr": [], "rn": [], "hn": [], "n": 0})
total = 0

with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    for line in f:
        r = json.loads(line)
        lang = r["lang"]
        ref = r["reference"]
        hyp = r.get("hypothesis", "")
        by_lang[lang]["rr"].append(ref)
        by_lang[lang]["hr"].append(hyp)
        by_lang[lang]["rn"].append(norm(ref))
        by_lang[lang]["hn"].append(norm(hyp))
        by_lang[lang]["n"] += 1
        total += 1

print(f"Samples processed so far: {total}")
print(f"\n{'Lang':<12} {'N':>7} {'WER_raw':>8} {'WER_nrm':>8} {'CER_nrm':>8}")
print(f"{'-'*12} {'-'*7} {'-'*8} {'-'*8} {'-'*8}")

all_rr, all_hr, all_rn, all_hn = [], [], [], []
for lang in sorted(by_lang.keys()):
    d = by_lang[lang]
    if not d["rn"]: continue
    wr = round(wer(d["rr"], d["hr"]) * 100, 2)
    wn = round(wer(d["rn"], d["hn"]) * 100, 2)
    cn = round(cer(d["rn"], d["hn"]) * 100, 2)
    print(f"{lang:<12} {d['n']:>7} {wr:>8} {wn:>8} {cn:>8}")
    all_rr.extend(d["rr"]); all_hr.extend(d["hr"])
    all_rn.extend(d["rn"]); all_hn.extend(d["hn"])

print(f"{'-'*12} {'-'*7} {'-'*8} {'-'*8} {'-'*8}")
wr = round(wer(all_rr, all_hr) * 100, 2)
wn = round(wer(all_rn, all_hn) * 100, 2)
cn = round(cer(all_rn, all_hn) * 100, 2)
print(f"{'OVERALL':<12} {total:>7} {wr:>8} {wn:>8} {cn:>8}")
PYEOF

Live scores from transcriptions so far


Samples processed so far: 71673

Lang               N  WER_raw  WER_nrm  CER_nrm
------------ ------- -------- -------- --------
as              3159    35.65    27.17    17.92
bn              6807    27.88    17.63    13.13
en              6047    24.75    10.07     6.03
gu              5620     20.8    16.01     12.8
hi              6451    27.27    21.37    27.32
kn              5399    37.18    29.13     22.2
ml              6488    37.89     30.2    17.35
mr              5654     25.7    19.41    17.41
or              7658    32.31    24.85    17.51
pa              5866    21.01    15.88    12.71
ta              6295    37.13    27.58    16.77
te              6229    41.87    34.37     28.5
------------ ------- -------- -------- --------
OVERALL        71673     30.0    21.97    16.85


**Live scores at 71,673 / ~300k (24%):**

| Lang | N | WER_raw | WER_norm | CER_norm |
|------|----:|------:|-------:|-------:|
| English | 6,047 | 24.75 | **10.07** | 6.03 |
| Punjabi | 5,866 | 21.01 | **15.88** | 12.71 |
| Gujarati | 5,620 | 20.80 | **16.01** | 12.80 |
| Bengali | 6,807 | 27.88 | **17.63** | 13.13 |
| Marathi | 5,654 | 25.70 | **19.41** | 17.41 |
| Hindi | 6,451 | 27.27 | **21.37** | 27.32 |
| Odia | 7,658 | 32.31 | **24.85** | 17.51 |
| Assamese | 3,159 | 35.65 | **27.17** | 17.92 |
| Tamil | 6,295 | 37.13 | **27.58** | 16.77 |
| Kannada | 5,399 | 37.18 | **29.13** | 22.20 |
| Malayalam | 6,488 | 37.89 | **30.20** | 17.35 |
| Telugu | 6,229 | 41.87 | **34.37** | 28.50 |
| **OVERALL** | **71,673** | **30.00** | **21.97** | **16.85** |

These are running numbers — will stabilize as more samples come in. Hindi CER is unusually high (27.32%) compared to WER (21.37%), worth investigating later.


do you wanna check in, maybe debug what may be happening, for such high numbers compared to kathbath dataset.


python3 << 'PYEOF'
import json, re

# Sample some predictions to see what's happening
langs_to_check = ["hi", "te", "kn", "as"]
samples_per_lang = 3

with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    by_lang = {}
    for line in f:
        r = json.loads(line)
        lang = r["lang"]
        if lang in langs_to_check and lang not in by_lang:
            by_lang[lang] = []
        if lang in langs_to_check and len(by_lang[lang]) < samples_per_lang:
            by_lang[lang].append(r)

for lang in langs_to_check:
    print(f"\n{'='*60}")
    print(f"  {lang} — sample predictions")
    print(f"{'='*60}")
    for s in by_lang.get(lang, []):
        print(f"  REF: {s['reference'][:100]}")
        print(f"  HYP: {s['hypothesis'][:100]}")
        print(f"  dur={s['duration']:.1f}s  src={s['source']}")
        print()
PYEOF

Sample predictions for high-error languages


============================================================
  hi — sample predictions
============================================================
  REF: वो अच्छा रहा है और मुझे ये है अभी जैसे यश है ठीक है छोटा भाई जैसा है हमारे ठीक है
  HYP: वो अच्छा रहा है और मुझे ये है अभी जैसे यश है, ठीक है? छोटा भाई जैसा है हमारे, ठीक है?
  dur=7.8s  src=final-export

  REF: कर सकते हैं आपने इतने लोगों को पहुँचा दिया मेरे एज के जो चालीस प्लस के आदमी हैं तो मुझे भी पहुँचा ही
  HYP: कर सकते हैं आपने इतने लोगों को पहुँचा दिया मेरे age के जो 40 plus के आदमी हैं तो मुझे भी पहुँचा ही द
  dur=6.5s  src=final-export

  REF: वेरी नाइस पॉइंट अमोल आदमी आज तक वही है और हम लोगों को एक्सपीरियंस लेकर आगे बढ़ना है
  HYP: Very nice point Amol. आदमी आशक वही है और हम लोगों को experience लेके आगे बढ़ना है.
  dur=7.5s  src=final-export

============================================================
  te — sample predictions
============================================================
  REF: అవుట్ డోర్ లో చేయడం అంటే మాటలు కాదు కదా 800 మెంబర్స్ ని నువ్వు లీడ్ చేయాలంటే అది మామూలు విషయం కాదు న
  HYP: outdoor లో చేయడం అని మాటలు కాదు కదా, 800 members ని నువ్వు lead చేయాలంటే అది మామూలు విషయం కాదు. నిజం
  dur=8.0s  src=final-export

  REF: kitchen లో north west లో ఇడవచ్చు ఇడవచ్చు అయితే అప్పుడు south east లో ఏముంది చూడండి ఇప్పుడు నేను చెప్
  HYP: kitchen-ఉ north-west లో పెట్టొచ్చు. పెట్టొచ్చు. అయితే అప్పుడు south-east లో అయింది చూడు, ఇప్పుడు నేన
  dur=7.5s  src=final-export

  REF: రెండు వేల పద్దెనిమిది పంతొమ్మిది విద్యా సంవత్సరం ఏఐసిటిఈ చేత ఆమోదించబడిన ప్రైవేట్ సహాయం పొందిన విద్య
  HYP: రెండు వేల పద్దెనిమిది పంతొమ్మిది విద్యా సంవత్సరం ఏఐసీటీఈ చేత ఆమోదించబడిన ప్రైవేట్ సహాయం పొందిన విద్య
  dur=14.9s  src=indicvoices

============================================================
  kn — sample predictions
============================================================
  REF: ಪುಣ್ಯವನ್ನ ಮಾತ್ರ ಉಳಿಸಿಕೊಂಡು ಉಳಿದ ಪುಣ್ಯ ಪಾಪಗಳನ್ನೆಲ್ಲ ತೊರೆದು ಪಾಪದಲ್ಲಿ ಎರಡು ವಿಧ ಇಲ್ಲ ಬೇಕಾದ ಪುಣ್ಯ
  HYP: ಪುಣ್ಯವನ್ನ ಮಾತ್ರ ಉಳಿಸಿಕೊಂಡು ಉಳಿದ ಪುಣ್ಯಪಾಪಗಳನ್ನೆಲ್ಲ ತೊರೆದು ಪಾಪದಲ್ಲಿ ಎರಡು ವಿಧ ಇಲ್ಲ ಬೇಕಾದ ಪುಣ್ಯ
  dur=5.9s  src=final-export

  REF: ಮುರುಮುರ ಅಂತ ಲೈನ್ ಮಾಡ್ಕೊಂಡು ಇಡೀ ಏರಿಯಾ ಸುತ್ತಾಕೊಂಡು ಬರ್ಲಿ
  HYP: ಮುರ್ಮುರಂತ ಲೈನ್ ಮಾಡ್ಕೊಂಡು ಇಡೀ ಏರಿಯಾ ಸುತ್ತಾಕೊಂಡು ಬರ್ಲಿ
  dur=2.7s  src=final-export

  REF: ಇಲ್ಲಿ ಎಮೋಷನಲ್ ವೀಡಿಯೋಸ್ ಗಳು ಸಿಗುತ್ತೆ ಫ್ಯಾಮಿಲಿ ವೀಡಿಯೋಸ್ ಗಳು ಸಿಗುತ್ತೆ ಕಪಲ್
  HYP: ಮತ್ತೆ emotional videos ಗಳು ಸಿಗುತ್ತೆ family videos ಗಳು ಸಿಗುತ್ತೆ couple
  dur=3.8s  src=final-export

============================================================
  as — sample predictions
============================================================
  REF: গোঁহাই ঘৰৰ কমালে আকৌ মহ এটা পহিলে কিমান ধৈৰ্য আছে আমাৰ ওলাই পৰে নহয়
  HYP: বখাই হ’ব মানে আকৌ মহেতা পৰিয়ালৰ যিমান ধৈৰ্য আছে আমাৰ ওচৰত পৰিবলৈ
  dur=3.6s  src=final-export

  REF: ভক্ত প্রহ্লাদ ভক্ত ধ্রুব রাজা অম্বরীষ রাজা হরিশ্চন্দ্র এই সকল কিমান কঠিন পরিস্থিতি
  HYP: ভক্ত প্রহ্লাদ ভক্ত ধ্রুব রজা অম্বৰী রজা হৰিশ্চন্দ্ৰ এইসকল কিমান কঠিন পৰিস্থিতি
  dur=6.4s  src=final-export

  REF: তলতে আমি ইমান গৰম অনুভৱ কৰোঁ আৰু পাহাৰলৈ গলে চাগৈ সূৰ্যটো আৰু ওচৰ পামগৈ গতিকে তাঁত কিজানি বেছি গৰম উ
  HYP: তলতে আমি ইমান গৰম অনুভৱ কৰোঁ আৰু পাহাৰলৈ গ লে চাগে সূৰ্যটো আৰু ওচৰ পামগৈ গতিকে তাত কিজানী বেছি গৰম উ
  dur=10.3s  src=indicvoices


python3 << 'PYEOF'
import json, re, unicodedata
from collections import Counter, defaultdict

def norm(t):
    t = unicodedata.normalize("NFC", t).lower()
    t = re.sub(r"[^\w\s]", "", t, flags=re.UNICODE)
    return re.sub(r"\s+", " ", t).strip()

# Check key error patterns
issues = defaultdict(lambda: {"code_switch": 0, "punct_only": 0, "numeral": 0, "total": 0, "empty_hyp": 0})
code_switch_re = re.compile(r'[a-zA-Z]{3,}')  # 3+ Latin chars in Indic text
digit_re = re.compile(r'\d')

with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    for line in f:
        r = json.loads(line)
        lang = r["lang"]
        ref = r["reference"]
        hyp = r.get("hypothesis", "")
        ref_n = norm(ref)
        hyp_n = norm(hyp)

        issues[lang]["total"] += 1

        # Empty hypothesis
        if not hyp.strip():
            issues[lang]["empty_hyp"] += 1

        # Code-switching: Latin words in Indic ref or hyp
        if lang != "en":
            ref_latin = len(code_switch_re.findall(ref))
            hyp_latin = len(code_switch_re.findall(hyp))
            if ref_latin > 0 or hyp_latin > 0:
                issues[lang]["code_switch"] += 1

        # Punct-only diff
        if ref != hyp and ref_n == hyp_n:
            issues[lang]["punct_only"] += 1

        # Numerals in ref or hyp
        if digit_re.search(ref) or digit_re.search(hyp):
            issues[lang]["numeral"] += 1

print(f"{'Lang':<6} {'Total':>7} {'CodeSw':>7} {'%':>5} {'PunctOnly':>10} {'%':>5} {'Numerals':>9} {'%':>5} {'Empty':>6}")
print(f"{'-'*6} {'-'*7} {'-'*7} {'-'*5} {'-'*10} {'-'*5} {'-'*9} {'-'*5} {'-'*6}")
for lang in sorted(issues.keys()):
    d = issues[lang]
    t = d["total"]
    print(f"{lang:<6} {t:>7} {d['code_switch']:>7} {d['code_switch']/t*100:>4.0f}% {d['punct_only']:>10} {d['punct_only']/t*100:>4.0f}% {d['numeral']:>9} {d['numeral']/t*100:>4.0f}% {d['empty_hyp']:>6}")

# Also check: how does source distribution differ?
print(f"\nSource distribution:")
sources = defaultdict(lambda: defaultdict(int))
with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    for line in f:
        r = json.loads(line)
        sources[r["lang"]][r["source"]] += 1
for lang in sorted(sources.keys()):
    parts = ", ".join(f"{s}={c}" for s,c in sorted(sources[lang].items(), key=lambda x:-x[1]))
    print(f"  {lang}: {parts}")
PYEOF

Diagnose error patterns across languages


Lang     Total  CodeSw     %  PunctOnly     %  Numerals     %  Empty
------ ------- ------- ----- ---------- ----- --------- ----- ------
as        3171     835   26%        155    5%       125    4%      0
bn        7722    1517   20%       1145   15%       448    6%      0
en        6700       0    0%       1876   28%       501    7%      0
gu        6302    1346   21%        641   10%       611   10%      0
hi        7235    3949   55%        535    7%       553    8%      0
kn        6010    2090   35%        396    7%       472    8%      0
ml        7252    1883   26%        382    5%       343    5%      0
mr        6301    2504   40%        473    8%       447    7%      0
or        8576    2137   25%        743    9%       573    7%      0
pa        6531    1465   22%        471    7%       403    6%      0
ta        7010    2259   32%        409    6%       601    9%      0
te        6985    3063   44%        388    6%       586    8%      0

Source distribution:
  as: final-export=1978, indicvoices=1193
  bn: final-export=6645, indicvoices=1077
  en: final-export=6700
  gu: final-export=6302
  hi: final-export=6864, indicvoices=371
  kn: final-export=6010
  ml: final-export=6535, indicvoices=717
  mr: final-export=6298, indicvoices=3
  or: final-export=5691, indicvoices=2885
  pa: final-export=6529, indicvoices=2
  ta: final-export=6273, indicvoices=737
  te: final-export=6827, indicvoices=158


Now I can see the pattern clearly. Here's what's driving the higher WER on the 300k dataset vs the 6k Kathbath benchmark:

**1. Massive code-switching** — the #1 factor:
- Hindi: **55%** of samples have English words mixed in
- Telugu: **44%**, Marathi: **40%**, Kannada: **35%**, Tamil: **32%**
- The model often outputs English words in Latin script while the reference has them transliterated into native script (or vice versa): e.g. REF `"एज"` vs HYP `"age"`, REF `"ऑल TP"` vs HYP `"all TP"`
- This inflates WER because `"एज" != "age"` even though they're the same word

**2. Punctuation inflation** — especially English:
- English: **28%** of samples differ only in punctuation (model adds commas, question marks)
- Bengali: **15%** punct-only diffs
- This is the raw→norm delta we saw earlier

**3. Numeral format mismatch**:
- ~7-10% of samples across all languages have digits
- REF: `"चालीस प्लस"` (spelled out) vs HYP: `"40 plus"` (digits)

**4. Data source difference**:
- The 6k benchmark was from Kathbath (read speech, clean, single-language)
- The 300k set is `final-export` + `indicvoices` — conversational, code-mixed, noisy
- This is a fundamentally harder eval set

**Bottom line:** The scores aren't worse because the model is bad — the 300k dataset is harder (code-mixed conversational speech). The 6k Kathbath benchmark was testing clean single-language read speech. Both are valid but measure different things.


i think we are cehcking the dataset on differnt format. we should be checking it on transcript_mixed format. did yoiu check if there's any other format of transcript.


python3 << 'PYEOF'
import json

# Check a range of samples to see all text patterns
seen = set()
with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    for i, line in enumerate(f):
        r = json.loads(line)
        ref = r["reference"]
        # Show the raw text field before our strip_lang_tag
        if i < 3:
            print(f"Our stored ref: {repr(ref[:100])}")
        if i > 20: break

# Now check the ACTUAL parquet to see the raw text field
import pandas as pd
df = pd.read_parquet("/home/ubuntu/training/datasets/indic-asr-validation-300k/data/train-00000-of-00140.parquet")
print(f"\nParquet columns: {list(df.columns)}")
print(f"\nFirst 10 raw 'text' values:")
for i, row in df.head(10).iterrows():
    print(f"  [{row['lang']}] {repr(row['text'][:120])}")

# Check if there are other text columns
for col in df.columns:
    if col not in ['audio', 'segment_id', 'duration']:
        print(f"\nColumn '{col}' unique samples:")
        for v in df[col].head(3):
            print(f"  {repr(str(v)[:120])}")
PYEOF

Check raw parquet text format and other columns


Our stored ref: '24000 પંક્તિઓનું આ મહાકાવ્ય છે'
Our stored ref: 'કેહતા હતા કે પ્રતીક તમે યાર આટલું બધું ડિજિટલ કરો છો તો તમારો યાર તમે પ્રતીકનું મિનિંગ શું છે સિમ્બો'
Our stored ref: 'वो अच्छा रहा है और मुझे ये है अभी जैसे यश है ठीक है छोटा भाई जैसा है हमारे ठीक है'

Parquet columns: ['audio', 'segment_id', 'text', 'duration', 'lang', 'source']

First 10 raw 'text' values:
  [gu] '<|gu|> 24000 પંક્તિઓનું આ મહાકાવ્ય છે'
  [gu] '<|gu|> કેહતા હતા કે પ્રતીક તમે યાર આટલું બધું ડિજિટલ કરો છો તો તમારો યાર તમે પ્રતીકનું મિનિંગ શું છે સિમ્બોલ સિમ્બોલ'
  [hi] '<|hi|> वो अच्छा रहा है और मुझे ये है अभी जैसे यश है ठीक है छोटा भाई जैसा है हमारे ठीक है'
  [bn] '<|bn|> পরবর্তীতে জানো এই মানুষগুলোকে কি করে মারা হয়েছিল তাদেরকে যখন পুলিশটুলিশ জানতে পারে অকথ্য অত্যাচার করা হয়েছিল তা'
  [kn] '<|kn|> ಪುಣ್ಯವನ್ನ ಮಾತ್ರ ಉಳಿಸಿಕೊಂಡು ಉಳಿದ ಪುಣ್ಯ ಪಾಪಗಳನ್ನೆಲ್ಲ ತೊರೆದು ಪಾಪದಲ್ಲಿ ಎರಡು ವಿಧ ಇಲ್ಲ ಬೇಕಾದ ಪುಣ್ಯ'
  [gu] '<|gu|> જ્યારે ભૂખ લાગે ત્યારે માત્ર શાકભાજી ખાતા હતા'
  [as] '<|as|> গোঁহাই ঘৰৰ কমালে আকৌ মহ এটা পহিলে কিমান ধৈৰ্য আছে আমাৰ ওলাই পৰে নহয়'
  [pa] '<|pa|> ਸਾਰੇ ਫੀਚਰਸ ਹੈਗੇ ਨੇ ਜਿਹੜੀ ਸਭ ਤੋਂ ਵੱਡੀ ਸੇਫਟੀ ਫੀਚਰ ਆ ਜਿਹੜੀ ਕਾਫ਼ੀ ਬੰਦਿਆਂ ਨੇ ਮੈਨੂੰ ਕਾਲ ਕੀਤੀ ਕਿ ਉਹਦੇ ਵਿੱਚ ਜਿਹੜਾ ਕਈ ਜਣੇ '
  [te] '<|te|> అవుట్ డోర్ లో చేయడం అంటే మాటలు కాదు కదా 800 మెంబర్స్ ని నువ్వు లీడ్ చేయాలంటే అది మామూలు విషయం కాదు నిజంగా క్రిసోస'
  [ta] '<|ta|> அப்ப இந்த இந்த வார்த்தைக்கு அர்த்தம் வந்து இரவில் குறைகிற நேரத்தை பகல்ல போட்டு போடுறான் பகல்ல குறையிற நேரத்தை இரவ'

Column 'text' unique samples:
  '<|gu|> 24000 પંક્તિઓનું આ મહાકાવ્ય છે'
  '<|gu|> કેહતા હતા કે પ્રતીક તમે યાર આટલું બધું ડિજિટલ કરો છો તો તમારો યાર તમે પ્રતીકનું મિનિંગ શું છે સિમ્બોલ સિમ્બોલ'
  '<|hi|> वो अच्छा रहा है और मुझे ये है अभी जैसे यश है ठीक है छोटा भाई जैसा है हमारे ठीक है'

Column 'lang' unique samples:
  'gu'
  'gu'
  'hi'

Column 'source' unique samples:
  'final-export'
  'final-export'
  'final-export'


Only one `text` column — no alternate "mixed" format. But you said the model was trained on `transcript_mixed`. Let me check the training data format.


# Check the training repo for data format clues
ls /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/ 2>&1 | head -20
# Check if there's a data prep script or config that shows the mixed transcript format
grep -r "transcript_mixed\|mixed_transcript\|code.switch\|transliterat" /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/ --include="*.py" --include="*.yaml" --include="*.json" -l 2>&1 | head -10

Check training repo for mixed transcript format


assets
docker
examples
finetuning
gpu_idle_watchdog.sh
launch_production.sh
launch_resilient.sh
LICENSE
MANIFEST.in
monitor_training.sh
OVERNIGHT_TODO.md
push_checkpoint_r2.sh
pyproject.toml
qwen_asr
qwen_asr.egg-info
README.md
run_bench_final.sh
run_benchmarks.sh
run_benchmarks_v2.sh
run_bucketed_smoke.sh


grep -r "transcript_mixed\|mixed" /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/finetuning/ --include="*.py" --include="*.yaml" --include="*.json" -l 2>&1 | head -10

Search finetuning dir for mixed format


(Bash completed with no output)


# The model was qwen3-asr-1.7B-mixed — check the training data format
# The ref text has <|gu|> tags — maybe the model was trained to OUTPUT with these tags?
# Let's check what the model actually outputs vs what training expects

# Check a Hindi code-switched sample more carefully
python3 << 'PYEOF'
import json

# The key question: was the training data in "mixed" format where
# code-switched words stay in Latin script?
# e.g. REF should be: "वो अच्छा रहा है और age के जो 40 plus के"
# instead of: "वो अच्छा रहा है और एज के जो चालीस प्लस के"

# Let's look at the actual ref vs hyp for code-switched Hindi samples
count = 0
with open("/home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl") as f:
    for line in f:
        r = json.loads(line)
        if r["lang"] == "hi" and count < 10:
            ref = r["reference"]
            hyp = r["hypothesis"]
            # Check if ref has Latin chars (code-mixing in ref)
            import re
            latin_in_ref = re.findall(r'[a-zA-Z]+', ref)
            latin_in_hyp = re.findall(r'[a-zA-Z]+', hyp)
            if latin_in_ref or latin_in_hyp:
                print(f"REF: {ref[:120]}")
                print(f"HYP: {hyp[:120]}")
                print(f"  Latin in REF: {latin_in_ref[:5]}")
                print(f"  Latin in HYP: {latin_in_hyp[:5]}")
                print()
                count += 1
PYEOF

Check code-switching pattern in ref vs hyp


REF: कर सकते हैं आपने इतने लोगों को पहुँचा दिया मेरे एज के जो चालीस प्लस के आदमी हैं तो मुझे भी पहुँचा ही दोगे
HYP: कर सकते हैं आपने इतने लोगों को पहुँचा दिया मेरे age के जो 40 plus के आदमी हैं तो मुझे भी पहुँचा ही दोगे
  Latin in REF: []
  Latin in HYP: ['age', 'plus']

REF: वेरी नाइस पॉइंट अमोल आदमी आज तक वही है और हम लोगों को एक्सपीरियंस लेकर आगे बढ़ना है
HYP: Very nice point Amol. आदमी आशक वही है और हम लोगों को experience लेके आगे बढ़ना है.
  Latin in REF: []
  Latin in HYP: ['Very', 'nice', 'point', 'Amol', 'experience']

REF: इनमें से कोई भी सीईओ नहीं है सीईओ इन्होंने हायर किया है बाहर से प्रोफेशनल बंदा हायर किया है जो इन कंपनीज़ को रन करता है
HYP: इनमें से कोई भी CEO नहीं है. CEO इन्होंने hire किया है बाहर से, professional बंदा hire किया है जो इन companies को run कर
  Latin in REF: []
  Latin in HYP: ['CEO', 'CEO', 'hire', 'professional', 'hire']

REF: अदर्स में क्वायर में बजाता था
HYP: gears में cawr में बजाता था
  Latin in REF: []
  Latin in HYP: ['gears', 'cawr']

REF: अच्छा लास्ट वीडियो हमारे फुल स्टैक डेवलपमेंट सीरीज की नए टॉपिक जो आपको पढ़ने चाहिए अपने फुल स्टैक डेवलपमेंट के आगे के लिए
HYP: अच्छा last video हमारे full stack development series की नया topic जो आपको पढ़नी चाहिए अपने full stack development के आगे
  Latin in REF: ['AWS']
  Latin in HYP: ['last', 'video', 'full', 'stack', 'development']

REF: तो पहले सबसे पहले एक ऑटोमेटेड कॉल वाला फनल है या वॉट्सऐप कॉल वाला फनल
HYP: सो पहले सबसे पहले एक automated call वाला funnel है या WhatsApp call वाला फनल है
  Latin in REF: []
  Latin in HYP: ['automated', 'call', 'funnel', 'WhatsApp', 'call']

REF: दूसरी कैटेगरी जो पैसिव की है वहां पे ये चीज़ है कि वो सॉफ्ट है वो इमोशंस को समझता है वहां पे स्ट्रेंथ नहीं है
HYP: दूसरी category जो passive की है, वहां पे ये चीज़ है कि वो soft है, वो emotions को समझता है, वहां पे strength नहीं है।
  Latin in REF: []
  Latin in HYP: ['category', 'passive', 'soft', 'emotions', 'strength']

REF: अब जैसे ऑफ कोर्स अगर हम एक प्लेस पे ऑपरेट कर रहे हैं हमें वैलिडेशन चाहिए होती है
HYP: अब जैसे of course अगर हम एक place पे operate कर रहे हैं हमें validation चाहिए होती है
  Latin in REF: []
  Latin in HYP: ['of', 'course', 'place', 'operate', 'validation']

REF: टाइगर्स आपका किसी के साथ कोई कार्मिक बॉन्ड है
HYP: Time cuts. आपका किसी के साथ कोई karmic bond है?
  Latin in REF: []
  Latin in HYP: ['Time', 'cuts', 'karmic', 'bond']

REF: वो पिछले वाले एपिसोड में वो शेर नहीं था वो मुझे लगता है वो रेज़ोल्यूशंस पे ज़्यादा फिट है कि यकुम जनवरी है नया साल है दि
HYP: वो पिछले वाले episode में वो share नहीं था, वो मुझे लगता है वो resolutions पे ज़्यादा fit है कि यकुम जनवरी है, नया साल ह
  Latin in REF: []
  Latin in HYP: ['episode', 'share', 'resolutions', 'fit']


There it is. The problem is crystal clear:

**The reference text has ALL code-switched English words transliterated into Indic script** (e.g. `"एक्सपीरियंस"`, `"सीईओ"`, `"ऑटोमेटेड"`), but **the model (trained on "mixed" data) outputs them in Latin script** (e.g. `"experience"`, `"CEO"`, `"automated"`).

This is a **format mismatch**, not a recognition error. The model is actually getting the words RIGHT — it's just outputting them in the script it was trained on (mixed/code-switched), while the reference has them force-transliterated into native script.

This means we need to either:
1. Transliterate the Latin words in the hypothesis to native script before comparing, OR
2. Transliterate the native-script English words in the reference to Latin before comparing, OR
3. Check if the dataset has an alternate `transcript_mixed` column that keeps English words in Latin

Since there's no alternate column in the parquet, option 2 is the fair comparison for a model trained on mixed data. Want me to build a transliteration-aware normalizer that converts code-switched words to a common form before computing WER?


is it finished ?


wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1 && grep "Shard done" /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/bq4txtl6u.output 2>&1 | tail -3

Check progress


269499 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl
  Shard done. Total=263586 | 18 samp/s | RTF=0.0084 | ETA=34min
  Shard done. Total=265728 | 18 samp/s | RTF=0.0084 | ETA=32min
  Shard done. Total=267870 | 18 samp/s | RTF=0.0084 | ETA=30min


269k done, **ETA ~30 minutes**. Almost there.


that process died. nothing running in GPUs. anyways, let's stop it. there is no use on doing that. so there's 150k ckpt in R2. mind running our 6000 and 5.5k ish clean samples  benchmark on this ckpt. and compare with 120k. and add to benchmarks.


# Check how many transcriptions we got before it died
wc -l /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl 2>&1

# Find 150k checkpoint in R2
source /home/ubuntu/training/.env && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 ls "s3://ptcheckpoints/qwen3-asr-1.7B-mixed/03-31-2026/" --endpoint-url "$R2_ENDPOINT_URL" --region auto 2>&1

Check state and find 150k ckpt


269499 /home/ubuntu/training/transcriptions_300k/ckpt-120000-mixed.jsonl
                           PRE ckpt-100000/
                           PRE ckpt-120000/
                           PRE ckpt-150000/
                           PRE ckpt-74000/
                           PRE ckpt-76000/


269k transcriptions saved (can process metrics later). ckpt-150000 is there. Let me pull it and run both benchmarks.


source /home/ubuntu/training/.env && mkdir -p /home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-150000 && AWS_ACCESS_KEY_ID=$R2_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$R2_SECRET_ACCESS_KEY aws s3 cp "s3://ptcheckpoints/qwen3-asr-1.7B-mixed/03-31-2026/ckpt-150000/" "/home/ubuntu/training/checkpoints/qwen3-asr-mixed-ckpt-150000/" --endpoint-url "$R2_ENDPOINT_URL" --region auto --recursive --exclude "optimizer*" --exclude "rng_state*" --exclude "scheduler*" --exclude "trainer_state*" --exclude "training_args*" 2>&1 | tail -3

Download ckpt-150000