---
name: ASR benchmarking setup
description: Multiple ASR models being fine-tuned; benchmark infrastructure for qwen3-asr with indic-asr-benchmark-6k dataset
type: project
---

Multiple ASR models being fine-tuned in parallel by different agents:
- llama3.2-1b (TTS, another agent)
- qwen3-asr-1.7b (this agent handling benchmarks)
- gemma3n-E2B-asr (assigned to another agent per feedback_agent_assignment.md)

**Why:** Need to evaluate checkpoint quality during training to confirm models are learning meaningful representations for 12 Indic languages.

**How to apply:** Benchmark script at `/home/ubuntu/training/benchmark_qwen3_asr.py` uses HF dataset `BayAreaBoys/indic-asr-benchmark-6k` (500 samples x 12 languages). Supports `--backend vllm` (default, ~148s/run) and `--backend transformers`. Checkpoints stored locally at `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{step}`. Results go to `/home/ubuntu/training/benchmark_results/`.

**Qwen3-ASR training details (as of 2026-03-26):**
- Cosine LR schedule over 612k steps, peak LR=2e-5. At 100k steps = 16% through training, LR still near peak (1.886e-5).
- Data mix: 75M samples, 148k hours. English 40%, Hindi 12.5%, Telugu 10.5%, then long tail of Indic languages (Assamese lowest at 1.2%).
- NOT temperature-sampled by language — duration-bucketed only.
- max_batch_seq_len was raised from 500→700 at ~72k steps (was skipping 20.9% of batches).
- R2 checkpoint paths: `s3://ptcheckpoints/qwen3-asr-1.7B/{date}/ckpt-{step}/`

**Benchmark progression (vLLM, macro-avg WER):**
- ckpt-24k: 40.60%
- ckpt-72k: 38.01% (Δ=-2.59, 0.054 WER/1k steps)
- ckpt-100k: 37.44% (Δ=-0.57, 0.020 WER/1k steps — diminishing returns)
- English/Hindi/Marathi showing slight regression at 100k; expected to stabilize as cosine LR decays.
- Next eval planned at ~150k to check if regressions stabilize.

**vLLM 0.16 fix:** `qwen_asr/core/vllm_backend/qwen3_asr.py` — moved `_get_data_parser` to `Qwen3ASRProcessingInfo.build_data_parser()`. This is an uncommitted local change needed for vLLM 0.16 compat.