File created successfully at: /home/ubuntu/training/benchmark_results/analysis_report.md

# Qwen3-ASR Indic Benchmark Analysis Report
## ckpt-24000 vs ckpt-72000 on indic-asr-benchmark-6k

### Metrics Explanation

| Metric | Description | Industry Standard? |
|--------|-------------|--------------------|
| **WER** (Word Error Rate) | Primary ASR metric. Measures (insertions + deletions + substitutions) / reference words. | Yes - universal standard |
| **CER** (Character Error Rate) | Same as WER but at character level. Critical for morphologically rich / agglutinative languages (Tamil, Malayalam, Kannada). | Yes - especially for CJK, Indic |
| **WER-normalized** | WER after text normalization: NFC unicode, lowercase, punctuation removal, whitespace collapse. Removes surface-level mismatches. | Yes - all major benchmarks (OpenASR, CommonVoice) normalize |
| **CER-normalized** | Same normalization applied to CER. | Yes |

*Note: "Space-normalized WER" isn't a formal separate metric - it's part of standard normalization. Our WER-normalized covers this.*

---

### Results Summary

| Language    | WER 24k | WER 72k | Delta  | Rel. Improv. | CER 24k | CER 72k | Delta   |
|-------------|---------|---------|--------|--------------|---------|---------|---------|
| assamese    | 58.52   | 57.44   | -1.08  | +1.8%        | 29.59   | 29.44   | -0.15   |
| bengali     | 43.59   | 35.02   | -8.57  | **+19.7%**   | 21.97   | 13.77   | -8.20   |
| english     | 27.66   | 26.10   | -1.56  | +5.6%        | 11.77   | 11.31   | -0.46   |
| gujarati    | 31.12   | 28.18   | -2.94  | +9.4%        | 12.15   | 10.24   | -1.91   |
| hindi       | 14.07   | 14.04   | -0.03  | +0.2%        | 5.17    | 5.03    | -0.14   |
| kannada     | 55.09   | 48.83   | -6.26  | +11.4%       | 20.77   | 15.71   | -5.06   |
| malayalam   | 60.10   | 55.72   | -4.38  | +7.3%        | 17.31   | 15.46   | -1.85   |
| **marathi** | 62.82   | 43.35   | **-19.47** | **+31.0%** | 35.92 | 17.33   | **-18.59** |
| odia        | 48.37   | 44.75   | -3.62  | +7.5%        | 19.28   | 17.23   | -2.05   |
| **punjabi** | 32.41   | 35.01   | **+2.60** | **-8.0%** | 14.89   | 17.57   | +2.68   |
| tamil       | 62.92   | 53.33   | -9.59  | +15.2%       | 28.07   | 16.94   | -11.13  |
| telugu      | 46.11   | 41.89   | -4.22  | +9.2%        | 12.38   | 10.62   | -1.76   |
| **OVERALL** | 45.38   | 40.58   | -4.80  | +10.6%       | 19.83   | 15.45   | -4.38   |
| **MACRO**   | 45.23   | 40.30   | -4.93  | +10.9%       | 19.11   | 15.05   | -4.06   |

---

### Key Findings

#### 1. Strong overall progress: 48k more steps = ~11% relative WER improvement
- **Overall WER**: 45.38% -> 40.58% (-4.80 absolute, +10.6% relative)
- **Overall CER**: 19.83% -> 15.45% (-4.38 absolute, +22.1% relative)
- The model is clearly learning and improving across the board.

#### 2. Marathi: The biggest winner (+31% relative improvement)
- WER dropped from 62.82% to 43.35% (-19.47 absolute)
- CER dropped from 35.92% to 17.33% (-18.59 absolute)
- **Root cause**: At ckpt-24000, the model frequently **misidentified Marathi as Gujarati** (both use similar Devanagari/Gujarati scripts). By ckpt-72000, the language detection is fixed - the model correctly identifies and transcribes in Marathi script.
- This is a language confusion fix, not just transcription accuracy improvement.

#### 3. Punjabi: The only regression (-8% relative)
- WER went UP from 32.41% to 35.01% (+2.60 absolute)
- CER went UP from 14.89% to 17.57% (+2.68 absolute)
- Sample analysis shows the regression is subtle - both checkpoints handle common words well, but ckpt-72000 makes slightly more errors on proper nouns and place names. This could be catastrophic forgetting or training data distribution shift at 72k steps.
- **Action needed**: Check if Punjabi training data proportion dropped, or if the learning rate schedule is causing instability for this language.

#### 4. Language tiers by difficulty

**Tier 1 - Strong (<20% WER-norm):**
- Hindi: 10.04% (already near-converged at 24k, barely improved)
- English: 14.66% (good baseline, slight regression on normalized WER)

**Tier 2 - Good (20-30% WER-norm):**
- Gujarati: 22.86%
- Punjabi: 28.46%

**Tier 3 - Moderate (30-40% WER-norm):**
- Bengali: 30.42% (big improvement from 39.21%)
- Telugu: 34.14%
- Marathi: 37.86% (massive jump from 58.72%)
- Odia: 38.69%

**Tier 4 - Needs work (>40% WER-norm):**
- Kannada: 42.13%
- Tamil: 47.82%
- Malayalam: 49.65%
- Assamese: 53.38%

#### 5. Tamil: interesting progress pattern
- Large WER improvement (-9.59) but the model still hallucinates in wrong scripts sometimes.
- At 24k, one Tamil sample was transcribed in **Malayalam script** entirely. By 72k, this is fixed to Tamil.
- Shows the model is still resolving script/language boundaries for Dravidian languages.

#### 6. Hindi & English: already plateaued
- Hindi barely moved (14.07 -> 14.04 WER), suggesting the base Qwen3 model already had strong Hindi capability from pretraining.
- English similarly stable. Further gains for these languages would likely need domain-specific data, not more steps.

---

### Recommendations

1. **Continue training** - 11% relative improvement in 48k steps is solid progress. Most languages are still improving.
2. **Investigate Punjabi regression** - Could be data distribution, LR schedule, or catastrophic forgetting. Consider adding more Punjabi data.
3. **Dravidian languages need more data** - Tamil/Malayalam/Kannada are the weakest. Their agglutinative morphology makes them harder; more training data + CER-focused loss weighting could help.
4. **Assamese plateau** - Only 1.8% relative improvement suggests the model is struggling. Assamese has the least training data (from IndicVoices vs Kathbath for others); data quantity may be the bottleneck.
5. **Next checkpoint eval** - Run again at ~120k-150k steps to see if Punjabi regression persists and whether Tier 3/4 languages continue improving.

---

### RTF (Real-Time Factor)

| Checkpoint | Inference Time | Audio Duration | RTF |
|------------|---------------|----------------|-----|
| ckpt-24000 | 2472.6s | 40354s | 0.0613 |
| ckpt-72000 | ~2500s | 40354s | ~0.062 |

Both checkpoints run at ~16x real-time on single GPU with batch_size=16, which is good for evaluation throughput.


Here's the full picture:

---

## Benchmark Results: ckpt-24000 vs ckpt-72000

### Overall Numbers
| | WER | CER | WER-norm | CER-norm |
|---|---|---|---|---|
| **ckpt-24000** | 45.38% | 19.83% | 39.23% | 17.60% |
| **ckpt-72000** | 40.58% | 15.45% | 34.53% | 13.70% |
| **Improvement** | **-4.80** | **-4.38** | **-4.70** | **-3.90** |
| **Relative** | **+10.6%** | **+22.1%** | | |

### Metrics Used (industry standard)
- **WER** - universal primary ASR metric
- **CER** - critical for agglutinative Indic languages (Tamil, Malayalam, Kannada)
- **WER/CER-normalized** - after unicode NFC, lowercase, punctuation removal, whitespace normalization. Every major benchmark (OpenASR, CommonVoice, LibriSpeech) uses normalization. "Space-normalized WER" isn't a separate thing - it's just part of standard text normalization.

### Key Findings

1. **Meaningful progress**: 48k more steps = ~11% relative WER improvement across 12 languages. The model is learning.

2. **Marathi was the biggest winner** (-19.5 WER, +31% relative). Root cause: at 24k steps, the model was **confusing Marathi with Gujarati** and transcribing in Gujarati script. By 72k, language detection is fixed. This is a critical qualitative improvement.

3. **Punjabi is the only regression** (+2.6 WER, -8% relative). Both checkpoints handle common words well, but 72k makes more errors on proper nouns. Worth investigating - could be catastrophic forgetting or data imbalance.

4. **Hindi & English already plateaued** (~14% and ~26% WER). The base Qwen3 model already had strong coverage. Further Indic fine-tuning barely moves these.

5. **Dravidian languages (Tamil/Malayalam/Kannada)** are the weakest (42-50% WER-norm). Agglutinative morphology makes them inherently harder. More data + possibly CER-focused training would help.

6. **Tamil had script confusion** - at 24k, some samples were transcribed in Malayalam script. Fixed by 72k.

Full analysis saved to `benchmark_results/analysis_report.md`, predictions in `benchmark_results/ckpt-{24000,72000}/predictions.json`, metrics in `metrics.json`.


<task-notification>
<task-id>bmmm4127b</task-id>
<tool-use-id>toolu_0179bWRR7cYiE86LLywGrUAo</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bmmm4127b.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-72000 (inference files only, skip optimizer/rng)" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bmmm4127b.output


<task-notification>
<task-id>bwvqr4ho0</task-id>
<tool-use-id>toolu_01SbfKzqXAHZZwregNtzR3HF</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bwvqr4ho0.output</output-file>
<status>completed</status>
<summary>Background command "Download ckpt-24000 (inference files only, skip optimizer/rng)" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bwvqr4ho0.output


<task-notification>
<task-id>bvw7d6zll</task-id>
<tool-use-id>toolu_01GxMq7ieMiG6cAYTSAME8AB</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bvw7d6zll.output</output-file>
<status>completed</status>
<summary>Background command "Run benchmark on ckpt-24000" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/bvw7d6zll.output


<task-notification>
<task-id>b48bg2kgt</task-id>
<tool-use-id>toolu_016nHbaju3zyXznBEfbS7FBa</tool-use-id>
<output-file>/tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/b48bg2kgt.output</output-file>
<status>completed</status>
<summary>Background command "Run benchmark on ckpt-72000" completed (exit code 0)</summary>
</task-notification>
Read the output file to retrieve the result: /tmp/claude-1001/-home-ubuntu-training/48d580e9-023c-4ccb-9c00-3c67f5ca3e60/tasks/b48bg2kgt.output


All background tasks completed successfully - both checkpoint downloads and both benchmark runs finished. The results and analysis I shared above are the final output. Everything is saved under `benchmark_results/`.


Can we now setup vllm on these models and retest so analysing further checkpoints gets easier and quicker.


Bash

python3 -c "import vllm; print(vllm.__version__)" 2>&1

Check vLLM version


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→"""
    17→Examples for Qwen3ASRModel (vLLM backend).
    18→
    19→Covers:
    20→  - single-sample inference (URL audio)
    21→  - batch inference (mixed URL / base64 / (np.ndarray, sr))
    22→  - forcing language (text-only output)
    23→  - returning timestamps (single + batch) via Qwen3ForcedAligner
    24→
    25→Note:
    26→  Requires vLLM extra:
    27→    pip install qwen-asr[vllm]
    28→"""
    29→
    30→import base64
    31→import io
    32→import urllib.request
    33→from typing import Tuple
    34→
    35→import numpy as np
    36→import soundfile as sf
    37→import torch
    38→
    39→from qwen_asr import Qwen3ASRModel
    40→
    41→
    42→ASR_MODEL_PATH = "Qwen/Qwen3-ASR-1.7B"
    43→FORCED_ALIGNER_PATH = "Qwen/Qwen3-ForcedAligner-0.6B"
    44→
    45→URL_ZH = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav"
    46→URL_EN = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
    47→
    48→
    49→def _download_audio_bytes(url: str, timeout: int = 30) -> bytes:
    50→    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    51→    with urllib.request.urlopen(req, timeout=timeout) as resp:
    52→        return resp.read()
    53→
    54→
    55→def _read_wav_from_bytes(audio_bytes: bytes) -> Tuple[np.ndarray, int]:
    56→    with io.BytesIO(audio_bytes) as f:
    57→        wav, sr = sf.read(f, dtype="float32", always_2d=False)
    58→    return np.asarray(wav, dtype=np.float32), int(sr)
    59→
    60→
    61→def _to_data_url_base64(audio_bytes: bytes, mime: str = "audio/wav") -> str:
    62→    b64 = base64.b64encode(audio_bytes).decode("utf-8")
    63→    return f"data:{mime};base64,{b64}"
    64→
    65→
    66→def _print_result(title: str, results) -> None:
    67→    print(f"\n===== {title} =====")
    68→    for i, r in enumerate(results):
    69→        print(f"[sample {i}] language={r.language!r}")
    70→        print(f"[sample {i}] text={r.text!r}")
    71→        if r.time_stamps is not None and len(r.time_stamps) > 0:
    72→            head = r.time_stamps[0]
    73→            tail = r.time_stamps[-1]
    74→            print(f"[sample {i}] ts_first: {head.text!r} {head.start_time}->{head.end_time} s")
    75→            print(f"[sample {i}] ts_last : {tail.text!r} {tail.start_time}->{tail.end_time} s")
    76→
    77→
    78→def test_single_url(asr: Qwen3ASRModel) -> None:
    79→    results = asr.transcribe(
    80→        audio=URL_ZH,
    81→        language=None,
    82→        return_time_stamps=False,
    83→    )
    84→    assert isinstance(results, list) and len(results) == 1
    85→    _print_result("single-url (no forced language, no timestamps)", results)
    86→
    87→
    88→def test_batch_mixed(asr: Qwen3ASRModel) -> None:
    89→    zh_bytes = _download_audio_bytes(URL_ZH)
    90→    en_bytes = _download_audio_bytes(URL_EN)
    91→
    92→    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
    93→    en_wav, en_sr = _read_wav_from_bytes(en_bytes)
    94→
    95→    results = asr.transcribe(
    96→        audio=[URL_ZH, zh_b64, (en_wav, en_sr)],
    97→        context=["", "交易 停滞", ""],
    98→        language=[None, "Chinese", "English"],
    99→        return_time_stamps=False,
   100→    )
   101→    assert len(results) == 3
   102→    _print_result("batch-mixed (forced language for some)", results)
   103→
   104→
   105→def test_single_with_timestamps(asr: Qwen3ASRModel) -> None:
   106→    results = asr.transcribe(
   107→        audio=URL_EN,
   108→        language="English",
   109→        return_time_stamps=True,
   110→    )
   111→    assert len(results) == 1
   112→    assert results[0].time_stamps is not None
   113→    _print_result("single-url (forced language + timestamps)", results)
   114→
   115→
   116→def test_batch_with_timestamps(asr: Qwen3ASRModel) -> None:
   117→    zh_bytes = _download_audio_bytes(URL_ZH)
   118→    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
   119→
   120→    results = asr.transcribe(
   121→        audio=[URL_ZH, zh_b64, URL_EN],
   122→        context=["", "交易 停滞", ""],
   123→        language=["Chinese", "Chinese", "English"],
   124→        return_time_stamps=True,
   125→    )
   126→    assert len(results) == 3
   127→    assert all(r.time_stamps is not None for r in results)
   128→    _print_result("batch (forced language + timestamps)", results)
   129→
   130→
   131→def main() -> None:
   132→    asr = Qwen3ASRModel.LLM(
   133→        model=ASR_MODEL_PATH,
   134→        gpu_memory_utilization=0.8,
   135→        forced_aligner=FORCED_ALIGNER_PATH,
   136→        forced_aligner_kwargs=dict(
   137→            dtype=torch.bfloat16,
   138→            device_map="cuda:0",
   139→            # attn_implementation="flash_attention_2",
   140→        ),
   141→        max_inference_batch_size=32,
   142→        max_new_tokens=1024,
   143→    )
   144→
   145→    test_single_url(asr)
   146→    test_batch_mixed(asr)
   147→    test_single_with_timestamps(asr)
   148→    test_batch_with_timestamps(asr)
   149→
   150→
   151→if __name__ == "__main__":
   152→    main()
   153→

# coding=utf-8
# Copyright 2026 The Alibaba Qwen team.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Examples for Qwen3ASRModel (vLLM backend).

Covers:
  - single-sample inference (URL audio)
  - batch inference (mixed URL / base64 / (np.ndarray, sr))
  - forcing language (text-only output)
  - returning timestamps (single + batch) via Qwen3ForcedAligner

Note:
  Requires vLLM extra:
    pip install qwen-asr[vllm]
"""

import base64
import io
import urllib.request
from typing import Tuple

import numpy as np
import soundfile as sf
import torch

from qwen_asr import Qwen3ASRModel

ASR_MODEL_PATH = "Qwen/Qwen3-ASR-1.7B"
FORCED_ALIGNER_PATH = "Qwen/Qwen3-ForcedAligner-0.6B"

URL_ZH = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav"
URL_EN = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"

def _download_audio_bytes(url: str, timeout: int = 30) -> bytes:
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return resp.read()

def _read_wav_from_bytes(audio_bytes: bytes) -> Tuple[np.ndarray, int]:
    with io.BytesIO(audio_bytes) as f:
        wav, sr = sf.read(f, dtype="float32", always_2d=False)
    return np.asarray(wav, dtype=np.float32), int(sr)

def _to_data_url_base64(audio_bytes: bytes, mime: str = "audio/wav") -> str:
    b64 = base64.b64encode(audio_bytes).decode("utf-8")
    return f"data:{mime};base64,{b64}"

def _print_result(title: str, results) -> None:
    print(f"\n===== {title} =====")
    for i, r in enumerate(results):
        print(f"[sample {i}] language={r.language!r}")
        print(f"[sample {i}] text={r.text!r}")
        if r.time_stamps is not None and len(r.time_stamps) > 0:
            head = r.time_stamps[0]
            tail = r.time_stamps[-1]
            print(f"[sample {i}] ts_first: {head.text!r} {head.start_time}->{head.end_time} s")
            print(f"[sample {i}] ts_last : {tail.text!r} {tail.start_time}->{tail.end_time} s")

def test_single_url(asr: Qwen3ASRModel) -> None:
    results = asr.transcribe(
        audio=URL_ZH,
        language=None,
        return_time_stamps=False,
    )
    assert isinstance(results, list) and len(results) == 1
    _print_result("single-url (no forced language, no timestamps)", results)

def test_batch_mixed(asr: Qwen3ASRModel) -> None:
    zh_bytes = _download_audio_bytes(URL_ZH)
    en_bytes = _download_audio_bytes(URL_EN)

    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
    en_wav, en_sr = _read_wav_from_bytes(en_bytes)

    results = asr.transcribe(
        audio=[URL_ZH, zh_b64, (en_wav, en_sr)],
        context=["", "交易 停滞", ""],
        language=[None, "Chinese", "English"],
        return_time_stamps=False,
    )
    assert len(results) == 3
    _print_result("batch-mixed (forced language for some)", results)

def test_single_with_timestamps(asr: Qwen3ASRModel) -> None:
    results = asr.transcribe(
        audio=URL_EN,
        language="English",
        return_time_stamps=True,
    )
    assert len(results) == 1
    assert results[0].time_stamps is not None
    _print_result("single-url (forced language + timestamps)", results)

def test_batch_with_timestamps(asr: Qwen3ASRModel) -> None:
    zh_bytes = _download_audio_bytes(URL_ZH)
    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")

    results = asr.transcribe(
        audio=[URL_ZH, zh_b64, URL_EN],
        context=["", "交易 停滞", ""],
        language=["Chinese", "Chinese", "English"],
        return_time_stamps=True,
    )
    assert len(results) == 3
    assert all(r.time_stamps is not None for r in results)
    _print_result("batch (forced language + timestamps)", results)

def main() -> None:
    asr = Qwen3ASRModel.LLM(
        model=ASR_MODEL_PATH,
        gpu_memory_utilization=0.8,
        forced_aligner=FORCED_ALIGNER_PATH,
        forced_aligner_kwargs=dict(
            dtype=torch.bfloat16,
            device_map="cuda:0",
            # attn_implementation="flash_attention_2",
        ),
        max_inference_batch_size=32,
        max_new_tokens=1024,
    )

    test_single_url(asr)
    test_batch_mixed(asr)
    test_single_with_timestamps(asr)
    test_batch_with_timestamps(asr)

if __name__ == "__main__":
    main()


Good, vLLM 0.16.0 is already installed. Let me check the vLLM backend code to understand the `Qwen3ASRModel.LLM` API:


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→from dataclasses import dataclass
    17→from typing import Any, Dict, List, Optional, Union
    18→
    19→import numpy as np
    20→import torch
    21→from qwen_asr.core.transformers_backend import (
    22→    Qwen3ASRConfig,
    23→    Qwen3ASRForConditionalGeneration,
    24→    Qwen3ASRProcessor,
    25→)
    26→from transformers import AutoConfig, AutoModel, AutoProcessor
    27→
    28→AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
    29→AutoModel.register(Qwen3ASRConfig, Qwen3ASRForConditionalGeneration)
    30→AutoProcessor.register(Qwen3ASRConfig, Qwen3ASRProcessor)
    31→
    32→from .qwen3_forced_aligner import Qwen3ForcedAligner
    33→from .utils import (
    34→    MAX_ASR_INPUT_SECONDS,
    35→    MAX_FORCE_ALIGN_INPUT_SECONDS,
    36→    SAMPLE_RATE,
    37→    SUPPORTED_LANGUAGES,
    38→    AudioChunk,
    39→    AudioLike,
    40→    chunk_list,
    41→    merge_languages,
    42→    normalize_audios,
    43→    normalize_language_name,
    44→    parse_asr_output,
    45→    split_audio_into_chunks,
    46→    validate_language,
    47→)
    48→
    49→try:
    50→    from qwen_asr.core.vllm_backend import Qwen3ASRForConditionalGeneration
    51→    from vllm import ModelRegistry
    52→    ModelRegistry.register_model("Qwen3ASRForConditionalGeneration", Qwen3ASRForConditionalGeneration)
    53→except:
    54→    pass
    55→
    56→
    57→@dataclass
    58→class ASRTranscription:
    59→    """
    60→    One transcription result.
    61→
    62→    Attributes:
    63→        language (str):
    64→            Merged language string for the sample, e.g. "Chinese" or "Chinese,English".
    65→            Empty string if unknown or silent audio.
    66→        text (str):
    67→            Transcribed text.
    68→        time_stamps (Optional[Any]):
    69→            Forced aligner output (ForcedAlignResult).
    70→            Present only when return_time_stamps=True.
    71→    """
    72→    language: str
    73→    text: str
    74→    time_stamps: Optional[Any] = None
    75→
    76→
    77→@dataclass
    78→class ASRStreamingState:
    79→    """
    80→    Streaming ASR state for one audio stream (single utterance).

# coding=utf-8
# Copyright 2026 The Alibaba Qwen team.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from dataclasses import dataclass
from typing import Any, Dict, List, Optional, Union

import numpy as np
import torch
from qwen_asr.core.transformers_backend import (
    Qwen3ASRConfig,
    Qwen3ASRForConditionalGeneration,
    Qwen3ASRProcessor,
)
from transformers import AutoConfig, AutoModel, AutoProcessor

AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
AutoModel.register(Qwen3ASRConfig, Qwen3ASRForConditionalGeneration)
AutoProcessor.register(Qwen3ASRConfig, Qwen3ASRProcessor)

from .qwen3_forced_aligner import Qwen3ForcedAligner
from .utils import (
    MAX_ASR_INPUT_SECONDS,
    MAX_FORCE_ALIGN_INPUT_SECONDS,
    SAMPLE_RATE,
    SUPPORTED_LANGUAGES,
    AudioChunk,
    AudioLike,
    chunk_list,
    merge_languages,
    normalize_audios,
    normalize_language_name,
    parse_asr_output,
    split_audio_into_chunks,
    validate_language,
)

try:
    from qwen_asr.core.vllm_backend import Qwen3ASRForConditionalGeneration
    from vllm import ModelRegistry
    ModelRegistry.register_model("Qwen3ASRForConditionalGeneration", Qwen3ASRForConditionalGeneration)
except:
    pass

@dataclass
class ASRTranscription:
    """
    One transcription result.

    Attributes:
        language (str):
            Merged language string for the sample, e.g. "Chinese" or "Chinese,English".
            Empty string if unknown or silent audio.
        text (str):
            Transcribed text.
        time_stamps (Optional[Any]):
            Forced aligner output (ForcedAlignResult).
            Present only when return_time_stamps=True.
    """
    language: str
    text: str
    time_stamps: Optional[Any] = None

@dataclass
class ASRStreamingState:
    """
    Streaming ASR state for one audio stream (single utterance).


Grep

class Qwen3ASRModel|def LLM|def from_pretrained


131:class Qwen3ASRModel:
132-    """
133-    Unified inference wrapper for Qwen3-ASR with two backends:
134-      - Transformers backend
135-      - vLLM backend
136-
137-    It optionally supports time stamp output via Qwen3-ForcedAligner.
138-
139-    Notes:
140-      - Each request uses a context text and exactly one audio.
141-      - If language is provided, the prompt will force the output to be text-only by appending
142-        "language {Language}<asr_text>" to the assistant prompt.
143-    """
144-
145-    def __init__(
146-        self,
147-        backend: str,
148-        model: Any,
149-        processor: Any,
150-        sampling_params: Optional[Any] = None,
151-        forced_aligner: Optional[Qwen3ForcedAligner] = None,
--
176:    def from_pretrained(
177-        cls,
178-        pretrained_model_name_or_path: str,
179-        forced_aligner: Optional[str] = None,
180-        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
181-        max_inference_batch_size: int = 32,
182-        max_new_tokens: Optional[int] = 512,
183-        **kwargs,
184-    ) -> "Qwen3ASRModel":
185-        """
186-        Initialize using Transformers backend.
187-
188-        Args:
189-            pretrained_model_name_or_path:
190-                HuggingFace repo id or local directory.
191-            forced_aligner:
192-                Optional forced aligner model path/repo id.
193-            forced_aligner_kwargs:
194-                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
195-            max_inference_batch_size:
196-                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
--
227:    def LLM(
228-        cls,
229-        model: str,
230-        forced_aligner: Optional[str] = None,
231-        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
232-        max_inference_batch_size: int = -1,
233-        max_new_tokens: Optional[int] = 4096,
234-        **kwargs,
235-    ) -> "Qwen3ASRModel":
236-        """
237-        Initialize using vLLM backend.
238-
239-        Import is isolated to keep vLLM optional.
240-
241-        Args:
242-            model:
243-                Model path/repo for vLLM.
244-            forced_aligner:
245-                Optional forced aligner model path/repo id.
246-            forced_aligner_kwargs:
247-                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).


   227→    def LLM(
   228→        cls,
   229→        model: str,
   230→        forced_aligner: Optional[str] = None,
   231→        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
   232→        max_inference_batch_size: int = -1,
   233→        max_new_tokens: Optional[int] = 4096,
   234→        **kwargs,
   235→    ) -> "Qwen3ASRModel":
   236→        """
   237→        Initialize using vLLM backend.
   238→
   239→        Import is isolated to keep vLLM optional.
   240→
   241→        Args:
   242→            model:
   243→                Model path/repo for vLLM.
   244→            forced_aligner:
   245→                Optional forced aligner model path/repo id.
   246→            forced_aligner_kwargs:
   247→                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
   248→            max_inference_batch_size:
   249→                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
   250→            max_new_tokens:
   251→                Maximum number of tokens to generate.
   252→            **kwargs:
   253→                Forwarded to vllm.LLM(...).
   254→
   255→        Returns:
   256→            Qwen3ASRModel
   257→
   258→        Raises:
   259→            ImportError: If vLLM is not installed.
   260→        """
   261→        try:
   262→            from vllm import LLM as vLLM
   263→            from vllm import SamplingParams
   264→        except Exception as e:
   265→            raise ImportError(
   266→                "vLLM is not available. Install with: pip install qwen-asr[vllm]"
   267→            ) from e
   268→
   269→        llm = vLLM(model=model, **kwargs)
   270→
   271→        processor = Qwen3ASRProcessor.from_pretrained(model, fix_mistral_regex=True)
   272→        sampling_params = SamplingParams(**({"temperature": 0.0, "max_tokens": max_new_tokens}))
   273→
   274→        forced_aligner_model = None
   275→        if forced_aligner is not None:
   276→            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
   277→                forced_aligner, **(forced_aligner_kwargs or {})
   278→            )
   279→
   280→        return cls(
   281→            backend="vllm",
   282→            model=llm,
   283→            processor=processor,
   284→            sampling_params=sampling_params,
   285→            forced_aligner=forced_aligner_model,
   286→            max_inference_batch_size=max_inference_batch_size,

    def LLM(
        cls,
        model: str,
        forced_aligner: Optional[str] = None,
        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
        max_inference_batch_size: int = -1,
        max_new_tokens: Optional[int] = 4096,
        **kwargs,
    ) -> "Qwen3ASRModel":
        """
        Initialize using vLLM backend.

        Import is isolated to keep vLLM optional.

        Args:
            model:
                Model path/repo for vLLM.
            forced_aligner:
                Optional forced aligner model path/repo id.
            forced_aligner_kwargs:
                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
            max_inference_batch_size:
                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
            max_new_tokens:
                Maximum number of tokens to generate.
            **kwargs:
                Forwarded to vllm.LLM(...).

        Returns:
            Qwen3ASRModel

        Raises:
            ImportError: If vLLM is not installed.
        """
        try:
            from vllm import LLM as vLLM
            from vllm import SamplingParams
        except Exception as e:
            raise ImportError(
                "vLLM is not available. Install with: pip install qwen-asr[vllm]"
            ) from e

        llm = vLLM(model=model, **kwargs)

        processor = Qwen3ASRProcessor.from_pretrained(model, fix_mistral_regex=True)
        sampling_params = SamplingParams(**({"temperature": 0.0, "max_tokens": max_new_tokens}))

        forced_aligner_model = None
        if forced_aligner is not None:
            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
                forced_aligner, **(forced_aligner_kwargs or {})
            )

        return cls(
            backend="vllm",
            model=llm,
            processor=processor,
            sampling_params=sampling_params,
            forced_aligner=forced_aligner_model,
            max_inference_batch_size=max_inference_batch_size,


Good - the API is identical, just use `Qwen3ASRModel.LLM(model=path)` instead of `from_pretrained`. Now let me update the benchmark script to support both backends:


   160→                        "reference": references[i],
   161→                        "hypothesis": "",
   162→                        "error": str(e2),
   163→                    })
   164→                continue
   165→
   166→        for i, t in enumerate(transcriptions):
   167→            results.append({
   168→                "id": batch.iloc[i]["id"],
   169→                "language": batch.iloc[i]["language"],
   170→                "lang_code": lang_codes[i],
   171→                "reference": references[i],
   172→                "hypothesis": t.text,
   173→                "detected_language": t.language,
   174→            })
   175→
   176→        elapsed_pct = end / total * 100
   177→        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)
   178→
   179→    return results
   180→
   181→
   182→def compute_metrics(results: list[dict]) -> dict:
   183→    """Compute WER, CER, normalized-WER per language and overall."""
   184→    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})
   185→
   186→    for r in results:
   187→        lang = r["language"]
   188→        ref = r["reference"]
   189→        hyp = r.get("hypothesis", "")
   190→
   191→        ref_norm = normalize_text(ref)
   192→        hyp_norm = normalize_text(hyp)
   193→
   194→        # Skip empty references
   195→        if not ref_norm:
   196→            continue
   197→
   198→        by_lang[lang]["refs"].append(ref)
   199→        by_lang[lang]["hyps"].append(hyp)

                        "reference": references[i],
                        "hypothesis": "",
                        "error": str(e2),
                    })
                continue

        for i, t in enumerate(transcriptions):
            results.append({
                "id": batch.iloc[i]["id"],
                "language": batch.iloc[i]["language"],
                "lang_code": lang_codes[i],
                "reference": references[i],
                "hypothesis": t.text,
                "detected_language": t.language,
            })

        elapsed_pct = end / total * 100
        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)

    return results

def compute_metrics(results: list[dict]) -> dict:
    """Compute WER, CER, normalized-WER per language and overall."""
    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})

    for r in results:
        lang = r["language"]
        ref = r["reference"]
        hyp = r.get("hypothesis", "")

        ref_norm = normalize_text(ref)
        hyp_norm = normalize_text(hyp)

        # Skip empty references
        if not ref_norm:
            continue

        by_lang[lang]["refs"].append(ref)
        by_lang[lang]["hyps"].append(hyp)


   240→        "n_languages": len(lang_keys),
   241→        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
   242→        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
   243→        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
   244→        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
   245→    }
   246→
   247→    return metrics
   248→
   249→
   250→def print_metrics_table(metrics: dict, checkpoint_name: str):
   251→    """Pretty print metrics as a table."""
   252→    print(f"\n{'='*85}")
   253→    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
   254→    print(f"{'='*85}")
   255→    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
   256→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   257→
   258→    for lang in sorted(k for k in metrics if not k.startswith("__")):
   259→        m = metrics[lang]
   260→        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
   261→              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")
   262→
   263→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   264→
   265→    ov = metrics["__overall__"]
   266→    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
   267→          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")
   268→
   269→    ma = metrics["__macro_avg__"]
   270→    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
   271→          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
   272→    print(f"{'='*85}\n")
   273→
   274→
   275→def main():
   276→    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
   277→    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
   278→    parser.add_argument("--checkpoint-name", default=None, help="Name for this checkpoint in output files")
   279→    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/indic-asr-benchmark-6k")
   280→    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
   281→    parser.add_argument("--batch-size", type=int, default=16)
   282→    parser.add_argument("--languages", nargs="+", default=["all"],
   283→                        help="Language codes to eval (e.g. hi en ta) or 'all'")
   284→    parser.add_argument("--no-force-language", action="store_true",
   285→                        help="Don't force language, let model auto-detect")
   286→    parser.add_argument("--max-new-tokens", type=int, default=256)
   287→    args = parser.parse_args()
   288→
   289→    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
   290→    output_dir = Path(args.output_dir) / ckpt_name
   291→    output_dir.mkdir(parents=True, exist_ok=True)
   292→
   293→    # Add the repo to path for local imports
   294→    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
   295→    from qwen_asr import Qwen3ASRModel
   296→
   297→    print(f"Loading checkpoint: {args.checkpoint}")
   298→    t0 = time.time()
   299→    model = Qwen3ASRModel.from_pretrained(
   300→        args.checkpoint,
   301→        dtype=torch.bfloat16,
   302→        device_map="cuda:0",
   303→        max_inference_batch_size=args.batch_size,
   304→        max_new_tokens=args.max_new_tokens,
   305→    )
   306→    load_time = time.time() - t0
   307→    print(f"Model loaded in {load_time:.1f}s")
   308→
   309→    # Load dataset
   310→    langs = None if "all" in args.languages else args.languages
   311→    df = load_dataset(args.dataset_dir, langs)
   312→
   313→    # Run inference
   314→    print(f"\nRunning inference (batch_size={args.batch_size}, force_language={not args.no_force_language})...")
   315→    t0 = time.time()
   316→    results = run_inference(model, df, args.batch_size, force_language=not args.no_force_language)
   317→    inference_time = time.time() - t0
   318→    total_audio_sec = df["duration"].sum()
   319→    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0
   320→
   321→    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")
   322→
   323→    # Save raw predictions
   324→    preds_file = output_dir / "predictions.json"
   325→    with open(preds_file, "w", encoding="utf-8") as f:
   326→        json.dump(results, f, ensure_ascii=False, indent=2)
   327→    print(f"Predictions saved: {preds_file}")
   328→
   329→    # Compute metrics
   330→    metrics = compute_metrics(results)
   331→    metrics["__meta__"] = {
   332→        "checkpoint": args.checkpoint,
   333→        "checkpoint_name": ckpt_name,
   334→        "dataset": args.dataset_dir,
   335→        "batch_size": args.batch_size,
   336→        "force_language": not args.no_force_language,
   337→        "inference_time_sec": round(inference_time, 2),
   338→        "total_audio_sec": round(total_audio_sec, 2),
   339→        "rtf": round(rtf, 4),
   340→    }
   341→
   342→    metrics_file = output_dir / "metrics.json"
   343→    with open(metrics_file, "w") as f:
   344→        json.dump(metrics, f, indent=2)
   345→    print(f"Metrics saved: {metrics_file}")
   346→
   347→    # Print table
   348→    print_metrics_table(metrics, ckpt_name)
   349→
   350→
   351→if __name__ == "__main__":
   352→    main()
   353→

        "n_languages": len(lang_keys),
        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
    }

    return metrics

def print_metrics_table(metrics: dict, checkpoint_name: str):
    """Pretty print metrics as a table."""
    print(f"\n{'='*85}")
    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
    print(f"{'='*85}")
    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    for lang in sorted(k for k in metrics if not k.startswith("__")):
        m = metrics[lang]
        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")

    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    ov = metrics["__overall__"]
    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")

    ma = metrics["__macro_avg__"]
    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
    print(f"{'='*85}\n")

def main():
    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
    parser.add_argument("--checkpoint-name", default=None, help="Name for this checkpoint in output files")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/indic-asr-benchmark-6k")
    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
    parser.add_argument("--batch-size", type=int, default=16)
    parser.add_argument("--languages", nargs="+", default=["all"],
                        help="Language codes to eval (e.g. hi en ta) or 'all'")
    parser.add_argument("--no-force-language", action="store_true",
                        help="Don't force language, let model auto-detect")
    parser.add_argument("--max-new-tokens", type=int, default=256)
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.from_pretrained(
        args.checkpoint,
        dtype=torch.bfloat16,
        device_map="cuda:0",
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")

    # Load dataset
    langs = None if "all" in args.languages else args.languages
    df = load_dataset(args.dataset_dir, langs)

    # Run inference
    print(f"\nRunning inference (batch_size={args.batch_size}, force_language={not args.no_force_language})...")
    t0 = time.time()
    results = run_inference(model, df, args.batch_size, force_language=not args.no_force_language)
    inference_time = time.time() - t0
    total_audio_sec = df["duration"].sum()
    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0

    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")

    # Save raw predictions
    preds_file = output_dir / "predictions.json"
    with open(preds_file, "w", encoding="utf-8") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    print(f"Predictions saved: {preds_file}")

    # Compute metrics
    metrics = compute_metrics(results)
    metrics["__meta__"] = {
        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,
        "inference_time_sec": round(inference_time, 2),
        "total_audio_sec": round(total_audio_sec, 2),
        "rtf": round(rtf, 4),
    }

    metrics_file = output_dir / "metrics.json"
    with open(metrics_file, "w") as f:
        json.dump(metrics, f, indent=2)
    print(f"Metrics saved: {metrics_file}")

    # Print table
    print_metrics_table(metrics, ckpt_name)

if __name__ == "__main__":
    main()


Now let me update the script to support vLLM backend:


Edit

    parser.add_argument("--max-new-tokens", type=int, default=256)
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.from_pretrained(
        args.checkpoint,
        dtype=torch.bfloat16,
        device_map="cuda:0",
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")

    parser.add_argument("--max-new-tokens", type=int, default=256)
    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",
                        help="Inference backend (default: vllm)")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
                        help="vLLM GPU memory utilization (default: 0.85)")
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")
    t0 = time.time()
    if args.backend == "vllm":
        model = Qwen3ASRModel.LLM(
            model=args.checkpoint,
            gpu_memory_utilization=args.gpu_memory_utilization,
            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,
            max_new_tokens=args.max_new_tokens,
        )
    else:
        model = Qwen3ASRModel.from_pretrained(
            args.checkpoint,
            dtype=torch.bfloat16,
            device_map="cuda:0",
            max_inference_batch_size=args.batch_size,
            max_new_tokens=args.max_new_tokens,
        )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")


The file /home/ubuntu/training/benchmark_qwen3_asr.py has been updated successfully.

    parser.add_argument("--max-new-tokens", type=int, default=256)
    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",
                        help="Inference backend (default: vllm)")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
                        help="vLLM GPU memory utilization (default: 0.85)")
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")
    t0 = time.time()
    if args.backend == "vllm":
        model = Qwen3ASRModel.LLM(
            model=args.checkpoint,
            gpu_memory_utilization=args.gpu_memory_utilization,
            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,
            max_new_tokens=args.max_new_tokens,
        )
    else:
        model = Qwen3ASRModel.from_pretrained(
            args.checkpoint,
            dtype=torch.bfloat16,
            device_map="cuda:0",
            max_inference_batch_size=args.batch_size,
            max_new_tokens=args.max_new_tokens,
        )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")

#!/usr/bin/env python3
"""
Benchmark Qwen3-ASR checkpoints on indic-asr-benchmark-6k dataset.

Usage:
    python benchmark_qwen3_asr.py \
        --checkpoint /path/to/checkpoint \
        --checkpoint-name ckpt-24000 \
        --dataset-dir /home/ubuntu/training/indic-asr-benchmark-6k \
        --output-dir /home/ubuntu/training/benchmark_results \
        --batch-size 16 \
        --languages all

Metrics computed:
    - WER  (Word Error Rate)
    - CER  (Character Error Rate)
    - WER-norm (WER after unicode/punctuation/space normalization)
"""

import argparse
import json
import os
import re
import sys
import time
import unicodedata
from collections import defaultdict
from pathlib import Path

import numpy as np
import pandas as pd
import soundfile as sf
import torch
from jiwer import cer, wer

# ── Language code → Qwen3-ASR language name mapping ──────────────────────
# The model may auto-detect, but forcing helps for Indic languages added during fine-tuning.
# Languages the model supports (base + Indic from Phase 2 fine-tuning)
_MODEL_SUPPORTED_LANGS = {
    "Chinese", "English", "Cantonese", "Arabic", "German", "French", "Spanish",
    "Portuguese", "Indonesian", "Italian", "Korean", "Russian", "Thai", "Vietnamese",
    "Japanese", "Turkish", "Hindi", "Malay", "Dutch", "Swedish", "Danish", "Finnish",
    "Polish", "Czech", "Filipino", "Persian", "Greek", "Romanian", "Hungarian", "Macedonian",
    # Indic languages (Phase 2 fine-tuning)
    "Assamese", "Bengali", "Gujarati", "Kannada", "Malayalam",
    "Marathi", "Odia", "Punjabi", "Tamil", "Telugu",
}

LANG_CODE_TO_NAME = {
    "as": "Assamese",
    "bn": "Bengali",
    "en": "English",
    "gu": "Gujarati",
    "hi": "Hindi",
    "kn": "Kannada",
    "ml": "Malayalam",
    "mr": "Marathi",
    "or": "Odia",
    "pa": "Punjabi",
    "ta": "Tamil",
    "te": "Telugu",
}

# ── Text normalization (standard ASR eval practice) ──────────────────────
def normalize_text(text: str) -> str:
    """Normalize text for fair WER/CER comparison.

    Steps (order matters):
      1. Unicode NFC normalization
      2. Lowercase
      3. Remove punctuation (keep letters, numbers, spaces, Indic scripts)
      4. Collapse whitespace
      5. Strip
    """
    text = unicodedata.normalize("NFC", text)
    text = text.lower()
    # Remove punctuation but keep Unicode letters/numbers/spaces
    text = re.sub(r"[^\w\s]", "", text, flags=re.UNICODE)
    # Collapse whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text

def load_dataset(dataset_dir: str, languages: list[str] | None = None):
    """Load parquet files and return list of dicts with audio bytes + metadata."""
    data_dir = Path(dataset_dir) / "data"
    dfs = []
    for f in sorted(data_dir.glob("*.parquet")):
        dfs.append(pd.read_parquet(f))
    df = pd.concat(dfs, ignore_index=True)

    if languages and "all" not in languages:
        df = df[df["lang_code"].isin(languages) | df["language"].isin(languages)]
        df = df.reset_index(drop=True)

    print(f"Loaded {len(df)} samples across {df['language'].nunique()} languages")
    return df

def extract_audio_array(audio_cell) -> tuple[np.ndarray, int]:
    """Extract numpy audio array from HF dataset audio column."""
    import io
    audio_bytes = audio_cell["bytes"]
    wav, sr = sf.read(io.BytesIO(audio_bytes), dtype="float32")
    return wav, sr

def run_inference(model, df: pd.DataFrame, batch_size: int, force_language: bool = True):
    """Run batched inference and return list of (hypothesis, reference, language) tuples."""
    results = []
    total = len(df)

    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        batch = df.iloc[start:end]

        audio_inputs = []
        references = []
        languages = []
        lang_codes = []

        for _, row in batch.iterrows():
            wav, sr = extract_audio_array(row["audio"])
            audio_inputs.append((wav, sr))
            references.append(row["reference"])
            languages.append(LANG_CODE_TO_NAME.get(row["lang_code"], None))
            lang_codes.append(row["lang_code"])

        # Only force language for languages the model supports; pass None for others
        if force_language:
            forced_langs = [l if l in _MODEL_SUPPORTED_LANGS else None for l in languages]
            # If all None, just pass None
            if all(l is None for l in forced_langs):
                forced_langs = None
        else:
            forced_langs = None

        try:
            transcriptions = model.transcribe(
                audio=audio_inputs,
                language=forced_langs,
                return_time_stamps=False,
            )
        except Exception as e:
            print(f"  Batch {start}-{end} failed ({e}), retrying without forced language...")
            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs,
                    language=None,
                    return_time_stamps=False,
                )
            except Exception as e2:
                print(f"  Batch {start}-{end} FAILED: {e2}")
                for i in range(len(audio_inputs)):
                    results.append({
                        "id": batch.iloc[i]["id"],
                        "language": batch.iloc[i]["language"],
                        "lang_code": lang_codes[i],
                        "reference": references[i],
                        "hypothesis": "",
                        "error": str(e2),
                    })
                continue

        for i, t in enumerate(transcriptions):
            results.append({
                "id": batch.iloc[i]["id"],
                "language": batch.iloc[i]["language"],
                "lang_code": lang_codes[i],
                "reference": references[i],
                "hypothesis": t.text,
                "detected_language": t.language,
            })

        elapsed_pct = end / total * 100
        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)

    return results

def compute_metrics(results: list[dict]) -> dict:
    """Compute WER, CER, normalized-WER per language and overall."""
    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})

    for r in results:
        lang = r["language"]
        ref = r["reference"]
        hyp = r.get("hypothesis", "")

        ref_norm = normalize_text(ref)
        hyp_norm = normalize_text(hyp)

        # Skip empty references
        if not ref_norm:
            continue

        by_lang[lang]["refs"].append(ref)
        by_lang[lang]["hyps"].append(hyp)
        by_lang[lang]["refs_norm"].append(ref_norm)
        by_lang[lang]["hyps_norm"].append(hyp_norm)

    metrics = {}
    all_refs, all_hyps = [], []
    all_refs_norm, all_hyps_norm = [], []

    for lang in sorted(by_lang.keys()):
        d = by_lang[lang]
        n = len(d["refs"])
        lang_wer = wer(d["refs"], d["hyps"])
        lang_cer = cer(d["refs"], d["hyps"])
        lang_wer_norm = wer(d["refs_norm"], d["hyps_norm"])
        lang_cer_norm = cer(d["refs_norm"], d["hyps_norm"])

        metrics[lang] = {
            "n_samples": n,
            "wer": round(lang_wer * 100, 2),
            "cer": round(lang_cer * 100, 2),
            "wer_normalized": round(lang_wer_norm * 100, 2),
            "cer_normalized": round(lang_cer_norm * 100, 2),
        }

        all_refs.extend(d["refs"])
        all_hyps.extend(d["hyps"])
        all_refs_norm.extend(d["refs_norm"])
        all_hyps_norm.extend(d["hyps_norm"])

    # Overall (corpus-level, not macro-average)
    metrics["__overall__"] = {
        "n_samples": len(all_refs),
        "wer": round(wer(all_refs, all_hyps) * 100, 2),
        "cer": round(cer(all_refs, all_hyps) * 100, 2),
        "wer_normalized": round(wer(all_refs_norm, all_hyps_norm) * 100, 2),
        "cer_normalized": round(cer(all_refs_norm, all_hyps_norm) * 100, 2),
    }

    # Macro average across languages
    lang_keys = [k for k in metrics if k != "__overall__"]
    metrics["__macro_avg__"] = {
        "n_languages": len(lang_keys),
        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
    }

    return metrics

def print_metrics_table(metrics: dict, checkpoint_name: str):
    """Pretty print metrics as a table."""
    print(f"\n{'='*85}")
    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
    print(f"{'='*85}")
    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    for lang in sorted(k for k in metrics if not k.startswith("__")):
        m = metrics[lang]
        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")

    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    ov = metrics["__overall__"]
    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")

    ma = metrics["__macro_avg__"]
    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
    print(f"{'='*85}\n")

def main():
    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
    parser.add_argument("--checkpoint-name", default=None, help="Name for this checkpoint in output files")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/indic-asr-benchmark-6k")
    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
    parser.add_argument("--batch-size", type=int, default=16)
    parser.add_argument("--languages", nargs="+", default=["all"],
                        help="Language codes to eval (e.g. hi en ta) or 'all'")
    parser.add_argument("--no-force-language", action="store_true",
                        help="Don't force language, let model auto-detect")
    parser.add_argument("--max-new-tokens", type=int, default=256)
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint}")
    t0 = time.time()
    model = Qwen3ASRModel.from_pretrained(
        args.checkpoint,
        dtype=torch.bfloat16,
        device_map="cuda:0",
        max_inference_batch_size=args.batch_size,
        max_new_tokens=args.max_new_tokens,
    )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")

    # Load dataset
    langs = None if "all" in args.languages else args.languages
    df = load_dataset(args.dataset_dir, langs)

    # Run inference
    print(f"\nRunning inference (batch_size={args.batch_size}, force_language={not args.no_force_language})...")
    t0 = time.time()
    results = run_inference(model, df, args.batch_size, force_language=not args.no_force_language)
    inference_time = time.time() - t0
    total_audio_sec = df["duration"].sum()
    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0

    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")

    # Save raw predictions
    preds_file = output_dir / "predictions.json"
    with open(preds_file, "w", encoding="utf-8") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    print(f"Predictions saved: {preds_file}")

    # Compute metrics
    metrics = compute_metrics(results)
    metrics["__meta__"] = {
        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,
        "inference_time_sec": round(inference_time, 2),
        "total_audio_sec": round(total_audio_sec, 2),
        "rtf": round(rtf, 4),
    }

    metrics_file = output_dir / "metrics.json"
    with open(metrics_file, "w") as f:
        json.dump(metrics, f, indent=2)
    print(f"Metrics saved: {metrics_file}")

    # Print table
    print_metrics_table(metrics, ckpt_name)

if __name__ == "__main__":
    main()

+    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",

+                        help="Inference backend (default: vllm)")

+    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,

+                        help="vLLM GPU memory utilization (default: 0.85)")

-    print(f"Loading checkpoint: {args.checkpoint}")

+    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")

-    model = Qwen3ASRModel.from_pretrained(

-        max_inference_batch_size=args.batch_size,

-        max_new_tokens=args.max_new_tokens,

+            gpu_memory_utilization=args.gpu_memory_utilization,

+            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,

+            max_new_tokens=args.max_new_tokens,

+        model = Qwen3ASRModel.from_pretrained(

+            max_inference_batch_size=args.batch_size,


Also update the metadata to record backend:


        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,

        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "backend": args.backend,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,


        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "backend": args.backend,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,

#!/usr/bin/env python3
"""
Benchmark Qwen3-ASR checkpoints on indic-asr-benchmark-6k dataset.

Usage:
    python benchmark_qwen3_asr.py \
        --checkpoint /path/to/checkpoint \
        --checkpoint-name ckpt-24000 \
        --dataset-dir /home/ubuntu/training/indic-asr-benchmark-6k \
        --output-dir /home/ubuntu/training/benchmark_results \
        --batch-size 16 \
        --languages all

Metrics computed:
    - WER  (Word Error Rate)
    - CER  (Character Error Rate)
    - WER-norm (WER after unicode/punctuation/space normalization)
"""

import argparse
import json
import os
import re
import sys
import time
import unicodedata
from collections import defaultdict
from pathlib import Path

import numpy as np
import pandas as pd
import soundfile as sf
import torch
from jiwer import cer, wer

# ── Language code → Qwen3-ASR language name mapping ──────────────────────
# The model may auto-detect, but forcing helps for Indic languages added during fine-tuning.
# Languages the model supports (base + Indic from Phase 2 fine-tuning)
_MODEL_SUPPORTED_LANGS = {
    "Chinese", "English", "Cantonese", "Arabic", "German", "French", "Spanish",
    "Portuguese", "Indonesian", "Italian", "Korean", "Russian", "Thai", "Vietnamese",
    "Japanese", "Turkish", "Hindi", "Malay", "Dutch", "Swedish", "Danish", "Finnish",
    "Polish", "Czech", "Filipino", "Persian", "Greek", "Romanian", "Hungarian", "Macedonian",
    # Indic languages (Phase 2 fine-tuning)
    "Assamese", "Bengali", "Gujarati", "Kannada", "Malayalam",
    "Marathi", "Odia", "Punjabi", "Tamil", "Telugu",
}

LANG_CODE_TO_NAME = {
    "as": "Assamese",
    "bn": "Bengali",
    "en": "English",
    "gu": "Gujarati",
    "hi": "Hindi",
    "kn": "Kannada",
    "ml": "Malayalam",
    "mr": "Marathi",
    "or": "Odia",
    "pa": "Punjabi",
    "ta": "Tamil",
    "te": "Telugu",
}

# ── Text normalization (standard ASR eval practice) ──────────────────────
def normalize_text(text: str) -> str:
    """Normalize text for fair WER/CER comparison.

    Steps (order matters):
      1. Unicode NFC normalization
      2. Lowercase
      3. Remove punctuation (keep letters, numbers, spaces, Indic scripts)
      4. Collapse whitespace
      5. Strip
    """
    text = unicodedata.normalize("NFC", text)
    text = text.lower()
    # Remove punctuation but keep Unicode letters/numbers/spaces
    text = re.sub(r"[^\w\s]", "", text, flags=re.UNICODE)
    # Collapse whitespace
    text = re.sub(r"\s+", " ", text).strip()
    return text

def load_dataset(dataset_dir: str, languages: list[str] | None = None):
    """Load parquet files and return list of dicts with audio bytes + metadata."""
    data_dir = Path(dataset_dir) / "data"
    dfs = []
    for f in sorted(data_dir.glob("*.parquet")):
        dfs.append(pd.read_parquet(f))
    df = pd.concat(dfs, ignore_index=True)

    if languages and "all" not in languages:
        df = df[df["lang_code"].isin(languages) | df["language"].isin(languages)]
        df = df.reset_index(drop=True)

    print(f"Loaded {len(df)} samples across {df['language'].nunique()} languages")
    return df

def extract_audio_array(audio_cell) -> tuple[np.ndarray, int]:
    """Extract numpy audio array from HF dataset audio column."""
    import io
    audio_bytes = audio_cell["bytes"]
    wav, sr = sf.read(io.BytesIO(audio_bytes), dtype="float32")
    return wav, sr

def run_inference(model, df: pd.DataFrame, batch_size: int, force_language: bool = True):
    """Run batched inference and return list of (hypothesis, reference, language) tuples."""
    results = []
    total = len(df)

    for start in range(0, total, batch_size):
        end = min(start + batch_size, total)
        batch = df.iloc[start:end]

        audio_inputs = []
        references = []
        languages = []
        lang_codes = []

        for _, row in batch.iterrows():
            wav, sr = extract_audio_array(row["audio"])
            audio_inputs.append((wav, sr))
            references.append(row["reference"])
            languages.append(LANG_CODE_TO_NAME.get(row["lang_code"], None))
            lang_codes.append(row["lang_code"])

        # Only force language for languages the model supports; pass None for others
        if force_language:
            forced_langs = [l if l in _MODEL_SUPPORTED_LANGS else None for l in languages]
            # If all None, just pass None
            if all(l is None for l in forced_langs):
                forced_langs = None
        else:
            forced_langs = None

        try:
            transcriptions = model.transcribe(
                audio=audio_inputs,
                language=forced_langs,
                return_time_stamps=False,
            )
        except Exception as e:
            print(f"  Batch {start}-{end} failed ({e}), retrying without forced language...")
            try:
                transcriptions = model.transcribe(
                    audio=audio_inputs,
                    language=None,
                    return_time_stamps=False,
                )
            except Exception as e2:
                print(f"  Batch {start}-{end} FAILED: {e2}")
                for i in range(len(audio_inputs)):
                    results.append({
                        "id": batch.iloc[i]["id"],
                        "language": batch.iloc[i]["language"],
                        "lang_code": lang_codes[i],
                        "reference": references[i],
                        "hypothesis": "",
                        "error": str(e2),
                    })
                continue

        for i, t in enumerate(transcriptions):
            results.append({
                "id": batch.iloc[i]["id"],
                "language": batch.iloc[i]["language"],
                "lang_code": lang_codes[i],
                "reference": references[i],
                "hypothesis": t.text,
                "detected_language": t.language,
            })

        elapsed_pct = end / total * 100
        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)

    return results

def compute_metrics(results: list[dict]) -> dict:
    """Compute WER, CER, normalized-WER per language and overall."""
    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})

    for r in results:
        lang = r["language"]
        ref = r["reference"]
        hyp = r.get("hypothesis", "")

        ref_norm = normalize_text(ref)
        hyp_norm = normalize_text(hyp)

        # Skip empty references
        if not ref_norm:
            continue

        by_lang[lang]["refs"].append(ref)
        by_lang[lang]["hyps"].append(hyp)
        by_lang[lang]["refs_norm"].append(ref_norm)
        by_lang[lang]["hyps_norm"].append(hyp_norm)

    metrics = {}
    all_refs, all_hyps = [], []
    all_refs_norm, all_hyps_norm = [], []

    for lang in sorted(by_lang.keys()):
        d = by_lang[lang]
        n = len(d["refs"])
        lang_wer = wer(d["refs"], d["hyps"])
        lang_cer = cer(d["refs"], d["hyps"])
        lang_wer_norm = wer(d["refs_norm"], d["hyps_norm"])
        lang_cer_norm = cer(d["refs_norm"], d["hyps_norm"])

        metrics[lang] = {
            "n_samples": n,
            "wer": round(lang_wer * 100, 2),
            "cer": round(lang_cer * 100, 2),
            "wer_normalized": round(lang_wer_norm * 100, 2),
            "cer_normalized": round(lang_cer_norm * 100, 2),
        }

        all_refs.extend(d["refs"])
        all_hyps.extend(d["hyps"])
        all_refs_norm.extend(d["refs_norm"])
        all_hyps_norm.extend(d["hyps_norm"])

    # Overall (corpus-level, not macro-average)
    metrics["__overall__"] = {
        "n_samples": len(all_refs),
        "wer": round(wer(all_refs, all_hyps) * 100, 2),
        "cer": round(cer(all_refs, all_hyps) * 100, 2),
        "wer_normalized": round(wer(all_refs_norm, all_hyps_norm) * 100, 2),
        "cer_normalized": round(cer(all_refs_norm, all_hyps_norm) * 100, 2),
    }

    # Macro average across languages
    lang_keys = [k for k in metrics if k != "__overall__"]
    metrics["__macro_avg__"] = {
        "n_languages": len(lang_keys),
        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
    }

    return metrics

def print_metrics_table(metrics: dict, checkpoint_name: str):
    """Pretty print metrics as a table."""
    print(f"\n{'='*85}")
    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
    print(f"{'='*85}")
    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    for lang in sorted(k for k in metrics if not k.startswith("__")):
        m = metrics[lang]
        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")

    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")

    ov = metrics["__overall__"]
    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")

    ma = metrics["__macro_avg__"]
    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
    print(f"{'='*85}\n")

def main():
    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
    parser.add_argument("--checkpoint-name", default=None, help="Name for this checkpoint in output files")
    parser.add_argument("--dataset-dir", default="/home/ubuntu/training/indic-asr-benchmark-6k")
    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
    parser.add_argument("--batch-size", type=int, default=16)
    parser.add_argument("--languages", nargs="+", default=["all"],
                        help="Language codes to eval (e.g. hi en ta) or 'all'")
    parser.add_argument("--no-force-language", action="store_true",
                        help="Don't force language, let model auto-detect")
    parser.add_argument("--max-new-tokens", type=int, default=256)
    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",
                        help="Inference backend (default: vllm)")
    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
                        help="vLLM GPU memory utilization (default: 0.85)")
    args = parser.parse_args()

    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
    output_dir = Path(args.output_dir) / ckpt_name
    output_dir.mkdir(parents=True, exist_ok=True)

    # Add the repo to path for local imports
    sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    from qwen_asr import Qwen3ASRModel

    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")
    t0 = time.time()
    if args.backend == "vllm":
        model = Qwen3ASRModel.LLM(
            model=args.checkpoint,
            gpu_memory_utilization=args.gpu_memory_utilization,
            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,
            max_new_tokens=args.max_new_tokens,
        )
    else:
        model = Qwen3ASRModel.from_pretrained(
            args.checkpoint,
            dtype=torch.bfloat16,
            device_map="cuda:0",
            max_inference_batch_size=args.batch_size,
            max_new_tokens=args.max_new_tokens,
        )
    load_time = time.time() - t0
    print(f"Model loaded in {load_time:.1f}s")

    # Load dataset
    langs = None if "all" in args.languages else args.languages
    df = load_dataset(args.dataset_dir, langs)

    # Run inference
    print(f"\nRunning inference (batch_size={args.batch_size}, force_language={not args.no_force_language})...")
    t0 = time.time()
    results = run_inference(model, df, args.batch_size, force_language=not args.no_force_language)
    inference_time = time.time() - t0
    total_audio_sec = df["duration"].sum()
    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0

    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")

    # Save raw predictions
    preds_file = output_dir / "predictions.json"
    with open(preds_file, "w", encoding="utf-8") as f:
        json.dump(results, f, ensure_ascii=False, indent=2)
    print(f"Predictions saved: {preds_file}")

    # Compute metrics
    metrics = compute_metrics(results)
    metrics["__meta__"] = {
        "checkpoint": args.checkpoint,
        "checkpoint_name": ckpt_name,
        "dataset": args.dataset_dir,
        "batch_size": args.batch_size,
        "force_language": not args.no_force_language,
        "inference_time_sec": round(inference_time, 2),
        "total_audio_sec": round(total_audio_sec, 2),
        "rtf": round(rtf, 4),
    }

    metrics_file = output_dir / "metrics.json"
    with open(metrics_file, "w") as f:
        json.dump(metrics, f, indent=2)
    print(f"Metrics saved: {metrics_file}")

    # Print table
    print_metrics_table(metrics, ckpt_name)

if __name__ == "__main__":
    main()


Now let me test vLLM with ckpt-72000 first to verify it works:


Bash

python3 benchmark_qwen3_asr.py \
    --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000 \
    --checkpoint-name ckpt-72000-vllm \
    --backend vllm \
    --batch-size 64 \
    --max-new-tokens 512 \
    --gpu-memory-utilization 0.85 2>&1 | head -40

Test vLLM backend loading


WARNING 03-25 22:54:13 [registry.py:847] Model architecture Qwen3ASRForConditionalGeneration is already registered, and will be overwritten by the new model class <class 'qwen_asr.core.vllm_backend.qwen3_asr.Qwen3ASRForConditionalGeneration'>.
Loading checkpoint: /home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000 (backend=vllm)
INFO 03-25 22:54:14 [utils.py:223] non-default args: {'gpu_memory_utilization': 0.85, 'disable_log_stats': True, 'model': '/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000'}
Unrecognized keys in `rope_scaling` for 'rope_type'='default': {'mrope_interleaved', 'interleaved', 'mrope_section'}
INFO 03-25 22:54:14 [qwen3_asr.py:414] thinker_config is None. Initializing thinker model with default values
INFO 03-25 22:54:14 [model.py:529] Resolved architecture: Qwen3ASRForConditionalGeneration
ERROR 03-25 22:54:14 [repo_utils.py:47] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000'. Use `repo_type` argument if needed., retrying 1 of 2
ERROR 03-25 22:54:16 [repo_utils.py:45] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000'. Use `repo_type` argument if needed.
INFO 03-25 22:54:16 [model.py:1871] Downcasting torch.float32 to torch.bfloat16.
INFO 03-25 22:54:16 [model.py:1549] Using max model len 65536
INFO 03-25 22:54:16 [scheduler.py:224] Chunked prefill is enabled with max_num_batched_tokens=8192.
INFO 03-25 22:54:16 [vllm.py:689] Asynchronous scheduling is enabled.
The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The tokenizer you are loading from '/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-72000' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
Traceback (most recent call last):
  File "/home/ubuntu/training/benchmark_qwen3_asr.py", line 365, in <module>
    main()
  File "/home/ubuntu/training/benchmark_qwen3_asr.py", line 304, in main
    model = Qwen3ASRModel.LLM(
  File "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/qwen3_asr.py", line 269, in LLM
    llm = vLLM(model=model, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 346, in __init__
    self.llm_engine = LLMEngine.from_engine_args(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 174, in from_engine_args
    return cls(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 90, in __init__
    self.input_processor = InputProcessor(self.vllm_config)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/input_processor.py", line 80, in __init__
    mm_budget = MultiModalBudget(vllm_config, mm_registry)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/encoder_budget.py", line 62, in __init__
    processor = mm_registry.create_processor(model_config, cache=cache)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py", line 240, in create_processor
    return factories.build_processor(ctx, cache=cache)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py", line 94, in build_processor
    return self.processor(info, dummy_inputs_builder, cache=cache)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing/processor.py", line 993, in __init__
    raise ValueError(
ValueError: BaseMultiModalProcessor._get_data_parser has been moved to `BaseProcessingInfo.build_data_parser` in v0.16. You should override `BaseProcessingInfo.build_data_parser` instead.


vLLM 0.16.0 has breaking API changes from the 0.14.0 that qwen_asr was built for. Let me check the vLLM backend code to fix the compatibility issue:


qwen3-asr-1.7b-phase2-sft/qwen_asr/core/vllm_backend/qwen3_asr.py:87:    BaseProcessingInfo,
qwen3-asr-1.7b-phase2-sft/qwen_asr/core/vllm_backend/qwen3_asr.py:533:class Qwen3ASRProcessingInfo(BaseProcessingInfo):
qwen3-asr-1.7b-phase2-sft/qwen_asr/core/vllm_backend/qwen3_asr.py:625:    def _get_data_parser(self) -> MultiModalDataParser:


   530→        return loaded_params
   531→
   532→
   533→class Qwen3ASRProcessingInfo(BaseProcessingInfo):
   534→    def get_hf_config(self):
   535→        return self.ctx.get_hf_config(Qwen3ASRConfig).thinker_config
   536→
   537→    def get_hf_processor(self, **kwargs: object) -> Qwen3ASRProcessor:
   538→        processor = self.ctx.get_hf_processor(
   539→            Qwen3ASRProcessor,
   540→            use_fast=kwargs.pop("use_fast", True),
   541→            **kwargs,
   542→        )
   543→        if not hasattr(processor, "audio_token"):
   544→            processor.audio_token = "<|audio_pad|>"
   545→        return processor
   546→
   547→    def get_feature_extractor(self, **kwargs: object) -> WhisperFeatureExtractor:
   548→        hf_processor = self.get_hf_processor(**kwargs)
   549→        feature_extractor = hf_processor.feature_extractor
   550→        assert isinstance(feature_extractor, WhisperFeatureExtractor)
   551→        return feature_extractor
   552→
   553→    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
   554→        return {"audio": None}
   555→
   556→
   557→class Qwen3ASRDummyInputsBuilder(BaseDummyInputsBuilder[Qwen3ASRProcessingInfo]):
   558→    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
   559→        num_audios = mm_counts.get("audio", 0)
   560→
   561→        hf_processor = self.info.get_hf_processor()
   562→        audio_token = hf_processor.audio_token
   563→
   564→        return audio_token * num_audios
   565→
   566→    def get_dummy_mm_data(
   567→        self,
   568→        seq_len: int,
   569→        mm_counts: Mapping[str, int],
   570→        mm_options: Mapping[str, BaseDummyOptions] | None = None,
   571→    ) -> MultiModalDataDict:
   572→        num_audios = mm_counts.get("audio", 0)
   573→
   574→        feature_extractor = self.info.get_feature_extractor()
   575→
   576→        target_audio_length = (
   577→            min(
   578→                feature_extractor.chunk_length,
   579→                30,
   580→            )
   581→            * feature_extractor.sampling_rate
   582→        )
   583→
   584→        audio_overrides = mm_options.get("audio") if mm_options else None
   585→
   586→        return {
   587→            "audio": self._get_dummy_audios(
   588→                length=target_audio_length,
   589→                num_audios=num_audios,
   590→                overrides=audio_overrides,
   591→            ),
   592→        }
   593→
   594→
   595→def _qwen3asr_field_config(hf_inputs: Mapping[str, torch.Tensor]):
   596→    audio_feature_lengths = hf_inputs.get("audio_feature_lengths", torch.empty((0,)))
   597→    return dict(
   598→        input_audio_features=MultiModalFieldConfig.flat_from_sizes(
   599→            "audio", audio_feature_lengths, dim=1
   600→        ),
   601→        feature_attention_mask=MultiModalFieldConfig.batched("audio"),
   602→        audio_feature_lengths=MultiModalFieldConfig.batched("audio"),
   603→    )
   604→
   605→
   606→class Qwen3ASRMultiModalDataParser(MultiModalDataParser):
   607→    def _parse_audio_data(
   608→        self,
   609→        data: dict[str, torch.Tensor] | ModalityData[AudioItem],
   610→    ) -> ModalityDataItems[Any, Any] | None:
   611→        if isinstance(data, dict):
   612→            return DictEmbeddingItems(
   613→                data,
   614→                modality="audio",
   615→                required_fields={"input_audio_features", "audio_feature_lengths"},
   616→                fields_factory=_qwen3asr_field_config,
   617→            )
   618→
   619→        return super()._parse_audio_data(data)
   620→
   621→
   622→class Qwen3ASRMultiModalProcessor(
   623→    Qwen3OmniMoeThinkerMultiModalProcessor,
   624→):
   625→    def _get_data_parser(self) -> MultiModalDataParser:
   626→        feature_extractor = self.info.get_feature_extractor()
   627→        return Qwen3ASRMultiModalDataParser(
   628→            target_sr=feature_extractor.sampling_rate,
   629→        )
   630→
   631→    def _get_mm_fields_config(
   632→        self,
   633→        hf_inputs: BatchFeature,
   634→        hf_processor_mm_kwargs: Mapping[str, object],
   635→    ) -> Mapping[str, MultiModalFieldConfig]:
   636→        return _qwen3asr_field_config(hf_inputs)
   637→
   638→    def _get_prompt_updates(
   639→        self,
   640→        mm_items: MultiModalDataItems,
   641→        hf_processor_mm_kwargs: Mapping[str, Any],
   642→        out_mm_kwargs: MultiModalKwargsItems,
   643→    ) -> Sequence[PromptUpdate]:
   644→        processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
   645→        tokenizer = self.info.get_tokenizer()
   646→        vocab = tokenizer.get_vocab()
   647→
   648→        audio_token = processor.audio_token
   649→        audio_token_id = vocab[audio_token]

        return loaded_params

class Qwen3ASRProcessingInfo(BaseProcessingInfo):
    def get_hf_config(self):
        return self.ctx.get_hf_config(Qwen3ASRConfig).thinker_config

    def get_hf_processor(self, **kwargs: object) -> Qwen3ASRProcessor:
        processor = self.ctx.get_hf_processor(
            Qwen3ASRProcessor,
            use_fast=kwargs.pop("use_fast", True),
            **kwargs,
        )
        if not hasattr(processor, "audio_token"):
            processor.audio_token = "<|audio_pad|>"
        return processor

    def get_feature_extractor(self, **kwargs: object) -> WhisperFeatureExtractor:
        hf_processor = self.get_hf_processor(**kwargs)
        feature_extractor = hf_processor.feature_extractor
        assert isinstance(feature_extractor, WhisperFeatureExtractor)
        return feature_extractor

    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
        return {"audio": None}

class Qwen3ASRDummyInputsBuilder(BaseDummyInputsBuilder[Qwen3ASRProcessingInfo]):
    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
        num_audios = mm_counts.get("audio", 0)

        hf_processor = self.info.get_hf_processor()
        audio_token = hf_processor.audio_token

        return audio_token * num_audios

    def get_dummy_mm_data(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int],
        mm_options: Mapping[str, BaseDummyOptions] | None = None,
    ) -> MultiModalDataDict:
        num_audios = mm_counts.get("audio", 0)

        feature_extractor = self.info.get_feature_extractor()

        target_audio_length = (
            min(
                feature_extractor.chunk_length,
                30,
            )
            * feature_extractor.sampling_rate
        )

        audio_overrides = mm_options.get("audio") if mm_options else None

        return {
            "audio": self._get_dummy_audios(
                length=target_audio_length,
                num_audios=num_audios,
                overrides=audio_overrides,
            ),
        }

def _qwen3asr_field_config(hf_inputs: Mapping[str, torch.Tensor]):
    audio_feature_lengths = hf_inputs.get("audio_feature_lengths", torch.empty((0,)))
    return dict(
        input_audio_features=MultiModalFieldConfig.flat_from_sizes(
            "audio", audio_feature_lengths, dim=1
        ),
        feature_attention_mask=MultiModalFieldConfig.batched("audio"),
        audio_feature_lengths=MultiModalFieldConfig.batched("audio"),
    )

class Qwen3ASRMultiModalDataParser(MultiModalDataParser):
    def _parse_audio_data(
        self,
        data: dict[str, torch.Tensor] | ModalityData[AudioItem],
    ) -> ModalityDataItems[Any, Any] | None:
        if isinstance(data, dict):
            return DictEmbeddingItems(
                data,
                modality="audio",
                required_fields={"input_audio_features", "audio_feature_lengths"},
                fields_factory=_qwen3asr_field_config,
            )

        return super()._parse_audio_data(data)

class Qwen3ASRMultiModalProcessor(
    Qwen3OmniMoeThinkerMultiModalProcessor,
):
    def _get_data_parser(self) -> MultiModalDataParser:
        feature_extractor = self.info.get_feature_extractor()
        return Qwen3ASRMultiModalDataParser(
            target_sr=feature_extractor.sampling_rate,
        )

    def _get_mm_fields_config(
        self,
        hf_inputs: BatchFeature,
        hf_processor_mm_kwargs: Mapping[str, object],
    ) -> Mapping[str, MultiModalFieldConfig]:
        return _qwen3asr_field_config(hf_inputs)

    def _get_prompt_updates(
        self,
        mm_items: MultiModalDataItems,
        hf_processor_mm_kwargs: Mapping[str, Any],
        out_mm_kwargs: MultiModalKwargsItems,
    ) -> Sequence[PromptUpdate]:
        processor = self.info.get_hf_processor(**hf_processor_mm_kwargs)
        tokenizer = self.info.get_tokenizer()
        vocab = tokenizer.get_vocab()

        audio_token = processor.audio_token
        audio_token_id = vocab[audio_token]


/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing/processor.py:992:        if hasattr(self, "_get_data_parser"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing/processor.py:994:                "BaseMultiModalProcessor._get_data_parser has been "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing/processor.py:995:                "moved to `BaseProcessingInfo.build_data_parser` in v0.16. "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing/processor.py:996:                "You should override `BaseProcessingInfo.build_data_parser` instead."

[Showing results with pagination = limit: 20, offset: 0]


Clear - in vLLM 0.16, `_get_data_parser` moved from the processor class to `BaseProcessingInfo.build_data_parser`. Let me fix it:


    80→    AudioProcessorItems,
    81→    DictEmbeddingItems,
    82→    ModalityDataItems,
    83→    MultiModalDataItems,
    84→    MultiModalDataParser,
    85→)
    86→from vllm.multimodal.processing import (
    87→    BaseProcessingInfo,
    88→    PromptReplacement,
    89→    PromptUpdate,
    90→)
    91→from vllm.sequence import IntermediateTensors
    92→from vllm.v1.attention.backends.registry import AttentionBackendEnum
    93→from vllm.tokenizers import cached_tokenizer_from_config
    94→from vllm.transformers_utils.processor import cached_processor_from_config
    95→from vllm.model_executor.models.vision import (
    96→    get_vit_attn_backend,
    97→)
    98→from ..transformers_backend.configuration_qwen3_asr import (
    99→    Qwen3ASRConfig,
   100→    Qwen3ASRThinkerConfig,
   101→    Qwen3ASRAudioEncoderConfig
   102→)
   103→from ..transformers_backend.processing_qwen3_asr import (
   104→    Qwen3ASRProcessor,

    AudioProcessorItems,
    DictEmbeddingItems,
    ModalityDataItems,
    MultiModalDataItems,
    MultiModalDataParser,
)
from vllm.multimodal.processing import (
    BaseProcessingInfo,
    PromptReplacement,
    PromptUpdate,
)
from vllm.sequence import IntermediateTensors
from vllm.v1.attention.backends.registry import AttentionBackendEnum
from vllm.tokenizers import cached_tokenizer_from_config
from vllm.transformers_utils.processor import cached_processor_from_config
from vllm.model_executor.models.vision import (
    get_vit_attn_backend,
)
from ..transformers_backend.configuration_qwen3_asr import (
    Qwen3ASRConfig,
    Qwen3ASRThinkerConfig,
    Qwen3ASRAudioEncoderConfig
)
from ..transformers_backend.processing_qwen3_asr import (
    Qwen3ASRProcessor,


    60→    Qwen2_5OmniAudioFeatureInputs,
    61→    Qwen3OmniMoeThinkerMultiModalProcessor,
    62→)
    63→from vllm.model_executor.models.utils import (
    64→    AutoWeightsLoader,
    65→    WeightsMapper,
    66→    _merge_multimodal_embeddings,
    67→    maybe_prefix,
    68→)
    69→from vllm.model_executor.models.whisper import ISO639_1_SUPPORTED_LANGS
    70→from vllm.multimodal import MULTIMODAL_REGISTRY
    71→from vllm.multimodal.inputs import (
    72→    AudioItem,
    73→    ModalityData,
    74→    MultiModalDataDict,
    75→    MultiModalFeatureSpec,
    76→    MultiModalFieldConfig,
    77→    MultiModalKwargsItems,
    78→)
    79→from vllm.multimodal.parse import (

    Qwen2_5OmniAudioFeatureInputs,
    Qwen3OmniMoeThinkerMultiModalProcessor,
)
from vllm.model_executor.models.utils import (
    AutoWeightsLoader,
    WeightsMapper,
    _merge_multimodal_embeddings,
    maybe_prefix,
)
from vllm.model_executor.models.whisper import ISO639_1_SUPPORTED_LANGS
from vllm.multimodal import MULTIMODAL_REGISTRY
from vllm.multimodal.inputs import (
    AudioItem,
    ModalityData,
    MultiModalDataDict,
    MultiModalFeatureSpec,
    MultiModalFieldConfig,
    MultiModalKwargsItems,
)
from vllm.multimodal.parse import (


Now I understand the structure. The fix is to:
1. Move `_get_data_parser` logic from `Qwen3ASRMultiModalProcessor` to `Qwen3ASRProcessingInfo.build_data_parser`
2. Remove `_get_data_parser` from the processor class


Edit

class Qwen3ASRProcessingInfo(BaseProcessingInfo):
    def get_hf_config(self):
        return self.ctx.get_hf_config(Qwen3ASRConfig).thinker_config

    def get_hf_processor(self, **kwargs: object) -> Qwen3ASRProcessor:
        processor = self.ctx.get_hf_processor(
            Qwen3ASRProcessor,
            use_fast=kwargs.pop("use_fast", True),
            **kwargs,
        )
        if not hasattr(processor, "audio_token"):
            processor.audio_token = "<|audio_pad|>"
        return processor

    def get_feature_extractor(self, **kwargs: object) -> WhisperFeatureExtractor:
        hf_processor = self.get_hf_processor(**kwargs)
        feature_extractor = hf_processor.feature_extractor
        assert isinstance(feature_extractor, WhisperFeatureExtractor)
        return feature_extractor

    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
        return {"audio": None}

class Qwen3ASRProcessingInfo(BaseProcessingInfo):
    def get_hf_config(self):
        return self.ctx.get_hf_config(Qwen3ASRConfig).thinker_config

    def get_hf_processor(self, **kwargs: object) -> Qwen3ASRProcessor:
        processor = self.ctx.get_hf_processor(
            Qwen3ASRProcessor,
            use_fast=kwargs.pop("use_fast", True),
            **kwargs,
        )
        if not hasattr(processor, "audio_token"):
            processor.audio_token = "<|audio_pad|>"
        return processor

    def get_feature_extractor(self, **kwargs: object) -> WhisperFeatureExtractor:
        hf_processor = self.get_hf_processor(**kwargs)
        feature_extractor = hf_processor.feature_extractor
        assert isinstance(feature_extractor, WhisperFeatureExtractor)
        return feature_extractor

    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
        return {"audio": None}

    def build_data_parser(self) -> "Qwen3ASRMultiModalDataParser":
        feature_extractor = self.get_feature_extractor()
        return Qwen3ASRMultiModalDataParser(
            target_sr=feature_extractor.sampling_rate,
        )