Find the existing qwen3 ASR benchmark/inference script in /home/ubuntu/training/. I need to understand:
1. How qwen3-asr models are loaded and run for inference (it's a HuggingFace transformers model, not NeMo)
2. The existing benchmark script (likely benchmark_qwen3_asr.py or similar)
3. How checkpoints are loaded — are they HF format with tokenizer, config, etc?
4. What model class is used (Qwen2AudioForConditionalGeneration or similar?)

Check: benchmark_qwen3*.py, qwen3-asr-*/scripts/, and any inference scripts. Be quick — just find the key loading/inference pattern.


I'll help you find and analyze the qwen3 ASR benchmark/inference script. Let me search for relevant files.


Bash

find /home/ubuntu/training -type f -name "*.py" | grep -E "(bench|inference|qwen)" | head -20


Good! I found the main files. Let me read the benchmark script and the inference scripts to understand the loading pattern.


1	#!/usr/bin/env python3
2	"""
3	Benchmark Qwen3-ASR checkpoints on indic-asr-benchmark-6k dataset.
4
5	Usage:
6	    python benchmark_qwen3_asr.py \
7	        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-ckpt-24000 \
8	        --checkpoint-name ckpt-24000 \
9	        --output-dir /home/ubuntu/training/benchmark_results \
10	        --batch-size 8
11
12	Metrics computed per language and overall:
13	    - WER  (Word Error Rate)
14	    - CER  (Character Error Rate)
15	    - WER-norm (WER after unicode/punctuation/space normalization)
16	    - CER-norm (CER after normalization)
17	"""
18
19	import argparse
20	import json
21	import os
22	import re
23	import sys
24	import time
25	import unicodedata
26	from collections import defaultdict
27	from pathlib import Path
28
29	import numpy as np
30	import torch
31	from datasets import load_dataset as hf_load_dataset
32	from jiwer import cer, wer
33
34	# Add the repo to path for local imports
35	sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
36
37	# Language code → Qwen3-ASR language name mapping
38	LANG_CODE_TO_NAME = {
39	    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
40	    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
41	    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
42	}
43
44	# Languages the model supports (base + Indic from Phase 2 fine-tuning)
45	_MODEL_SUPPORTED_LANGS = {
46	    "Chinese", "English", "Cantonese", "Arabic", "German", "French", "Spanish",
47	    "Portuguese", "Indonesian", "Italian", "Korean", "Russian", "Thai", "Vietnamese",
48	    "Japanese", "Turkish", "Hindi", "Malay", "Dutch", "Swedish", "Danish", "Finnish",
49	    "Polish", "Czech", "Filipino", "Persian", "Greek", "Romanian", "Hungarian", "Macedonian",
50	    # Indic languages (Phase 2 fine-tuning)
51	    "Assamese", "Bengali", "Gujarati", "Kannada", "Malayalam",
52	    "Marathi", "Odia", "Punjabi", "Tamil", "Telugu",
53	}
54
55
56	# ── Text normalization (standard ASR eval practice) ──────────────────────
57
58	def normalize_text(text: str) -> str:
59	    """Normalize text for fair WER/CER comparison.
60	    Unicode NFC → lowercase → remove punctuation → collapse whitespace → strip.
61	    """
62	    text = unicodedata.normalize("NFC", text)
63	    text = text.lower()
64	    text = re.sub(r"[^\w\s]", "", text, flags=re.UNICODE)
65	    text = re.sub(r"\s+", " ", text).strip()
66	    return text
67
68
69	def run_inference(model, dataset, batch_size: int, force_language: bool = True):
70	    """Run batched inference on HF dataset and return list of result dicts."""
71	    results = []
72	    total = len(dataset)
73
74	    for start in range(0, total, batch_size):
75	        end = min(start + batch_size, total)
76	        batch = dataset.select(range(start, end))
77
78	        audio_inputs = []
79	        for sample in batch:
80	            audio = sample["audio"]
81	            wav = np.array(audio["array"], dtype=np.float32)
82	            sr = audio["sampling_rate"]
83	            audio_inputs.append((wav, sr))
84
85	        # Force language if supported
86	        if force_language:
87	            forced_langs = []
88	            for sample in batch:
89	                lang_name = LANG_CODE_TO_NAME.get(sample["lang_code"])
90	                forced_langs.append(lang_name)
91	        else:
92	            forced_langs = None
93
94	        try:
95	            transcriptions = model.transcribe(
96	                audio=audio_inputs,
97	                language=forced_langs,
98	                return_time_stamps=False,
99	            )
100	        except Exception as e:
101	            print(f"  Batch {start}-{end} failed ({e}), retrying without forced language...")
102	            try:
103	                transcriptions = model.transcribe(
104	                    audio=audio_inputs, language=None, return_time_stamps=False,
105	                )
106	            except Exception as e2:
107	                print(f"  Batch {start}-{end} FAILED: {e2}, trying one-by-one...")
108	                transcriptions = []
109	                for ai in audio_inputs:
110	                    try:
111	                        r = model.transcribe(audio=ai, language=None, return_time_stamps=False)
112	                        transcriptions.extend(r)
113	                    except Exception as e3:
114	                        from qwen_asr.inference.qwen3_asr import ASRTranscription
115	                        transcriptions.append(ASRTranscription(language="", text=""))
116
117	        for i, t in enumerate(transcriptions):
118	            sample = batch[i]
119	            results.append({
120	                "id": sample["id"],
121	                "language": sample["language"],
122	                "lang_code": sample["lang_code"],
123	                "source": sample["source"],
124	                "duration": sample["duration"],
125	                "reference": sample["reference"],
126	                "hypothesis": t.text,
127	                "detected_language": t.language,
128	            })
129
130	        elapsed_pct = end / total * 100
131	        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)
132
133	    return results
134
135
136	def compute_metrics(results: list) -> dict:
137	    """Compute WER, CER, normalized-WER/CER per language and overall."""
138	    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})
139
140	    for r in results:
141	        lang = r["language"]
142	        ref = r["reference"]
143	        hyp = r.get("hypothesis", "")
144
145	        ref_norm = normalize_text(ref)
146	        hyp_norm = normalize_text(hyp)
147
148	        if not ref_norm:
149	            continue
150
151	        by_lang[lang]["refs"].append(ref)
152	        by_lang[lang]["hyps"].append(hyp)
153	        by_lang[lang]["refs_norm"].append(ref_norm)
154	        by_lang[lang]["hyps_norm"].append(hyp_norm)
155
156	    metrics = {}
157	    all_refs, all_hyps = [], []
158	    all_refs_norm, all_hyps_norm = [], []
159
160	    for lang in sorted(by_lang.keys()):
161	        d = by_lang[lang]
162	        n = len(d["refs"])
163	        lang_wer = wer(d["refs"], d["hyps"])
164	        lang_cer = cer(d["refs"], d["hyps"])
165	        lang_wer_norm = wer(d["refs_norm"], d["hyps_norm"])
166	        lang_cer_norm = cer(d["refs_norm"], d["hyps_norm"])
167
168	        empty_count = sum(1 for h in d["hyps"] if not h.strip())
169
170	        metrics[lang] = {
171	            "n_samples": n,
172	            "wer": round(lang_wer * 100, 2),
173	            "cer": round(lang_cer * 100, 2),
174	            "wer_normalized": round(lang_wer_norm * 100, 2),
175	            "cer_normalized": round(lang_cer_norm * 100, 2),
176	            "empty_hypotheses": empty_count,
177	        }
178
179	        all_refs.extend(d["refs"])
180	        all_hyps.extend(d["hyps"])
181	        all_refs_norm.extend(d["refs_norm"])
182	        all_hyps_norm.extend(d["hyps_norm"])
183
184	    # Overall (corpus-level)
185	    metrics["__overall__"] = {
186	        "n_samples": len(all_refs),
187	        "wer": round(wer(all_refs, all_hyps) * 100, 2),
188	        "cer": round(cer(all_refs, all_hyps) * 100, 2),
189	        "wer_normalized": round(wer(all_refs_norm, all_hyps_norm) * 100, 2),
190	        "cer_normalized": round(cer(all_refs_norm, all_hyps_norm) * 100, 2),
191	    }
192
193	    # Macro average across languages
194	    lang_keys = [k for k in metrics if not k.startswith("__")]
195	    metrics["__macro_avg__"] = {
196	        "n_languages": len(lang_keys),
197	        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
198	        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
199	        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
200	        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
201	    }
202
203	    return metrics
204
205
206	def print_metrics_table(metrics: dict, checkpoint_name: str):
207	    """Pretty print metrics as a table."""
208	    print(f"\n{'='*85}")
209	    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
210	    print(f"{'='*85}")
211	    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
212	    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
213
214	    for lang in sorted(k for k in metrics if not k.startswith("__")):
215	        m = metrics[lang]
216	        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
217	              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")
218
219	    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
220
221	    ov = metrics["__overall__"]
222	    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
223	          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")
224
225	    ma = metrics["__macro_avg__"]
226	    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
227	          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
228	    print(f"{'='*85}\n")
229
230
231	def save_sample_analysis(results: list, output_dir: Path, n_per_lang: int = 5):
232	    """Save a few sample predictions per language for manual inspection."""
233	    by_lang = defaultdict(list)
234	    for r in results:
235	        by_lang[r["language"]].append(r)
236
237	    samples = []
238	    for lang in sorted(by_lang.keys()):
239	        lang_results = by_lang[lang]
240	        # Pick first n_per_lang samples
241	        for r in lang_results[:n_per_lang]:
242	            samples.append({
243	                "id": r["id"],
244	                "language": r["language"],
245	                "reference": r["reference"],
246	                "hypothesis": r["hypothesis"],
247	                "detected_language": r.get("detected_language", ""),
248	                "ref_normalized": normalize_text(r["reference"]),
249	                "hyp_normalized": normalize_text(r["hypothesis"]),
250	            })
251
252	    path = output_dir / "sample_analysis.json"
253	    with open(path, "w", encoding="utf-8") as f:
254	        json.dump(samples, f, ensure_ascii=False, indent=2)
255	    print(f"Sample analysis saved: {path}")
256
257
258	def main():
259	    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
260	    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
261	    parser.add_argument("--checkpoint-name", default=None,
262	                        help="Name for this checkpoint in output files (default: dirname)")
263	    parser.add_argument("--hf-dataset", default="BayAreaBoys/indic-asr-benchmark-6k",
264	                        help="HuggingFace dataset name")
265	    parser.add_argument("--hf-token", default=None, help="HuggingFace token (or set HF_TOKEN env)")
266	    parser.add_argument("--cache-dir", default="/home/ubuntu/training/datasets/indic-asr-benchmark-6k")
267	    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
268	    parser.add_argument("--batch-size", type=int, default=8)
269	    parser.add_argument("--max-new-tokens", type=int, default=512)
270	    parser.add_argument("--device", default="cuda:0")
271	    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",
272	                        help="Inference backend (default: vllm)")
273	    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
274	                        help="vLLM GPU memory utilization (default: 0.85)")
275	    parser.add_argument("--no-force-language", action="store_true",
276	                        help="Don't force language, let model auto-detect")
277	    parser.add_argument("--languages", nargs="+", default=None,
278	                        help="Filter to specific language codes (e.g. hi en ta)")
279	    args = parser.parse_args()
280
281	    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
282	    output_dir = Path(args.output_dir) / ckpt_name
283	    output_dir.mkdir(parents=True, exist_ok=True)
284
285	    # Load model
286	    from qwen_asr import Qwen3ASRModel
287
288	    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")
289	    t0 = time.time()
290	    if args.backend == "vllm":
291	        model = Qwen3ASRModel.LLM(
292	            model=args.checkpoint,
293	            gpu_memory_utilization=args.gpu_memory_utilization,
294	            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,
295	            max_new_tokens=args.max_new_tokens,
296	        )
297	    else:
298	        model = Qwen3ASRModel.from_pretrained(
299	            args.checkpoint,
300	            dtype=torch.bfloat16,
301	            device_map=args.device,
302	            max_inference_batch_size=args.batch_size,
303	            max_new_tokens=args.max_new_tokens,
304	        )
305	    load_time = time.time() - t0
306	    print(f"Model loaded in {load_time:.1f}s")
307
308	    # Load dataset
309	    hf_token = args.hf_token or os.environ.get("HF_TOKEN")
310	    print(f"Loading dataset: {args.hf_dataset}")
311	    dataset = hf_load_dataset(
312	        args.hf_dataset, token=hf_token, cache_dir=args.cache_dir, split="train"
313	    )
314
315	    # Filter languages if specified
316	    if args.languages:
317	        dataset = dataset.filter(lambda x: x["lang_code"] in args.languages)
318	        print(f"Filtered to {len(dataset)} samples for languages: {args.languages}")
319
320	    print(f"Dataset: {len(dataset)} samples")
321
322	    # Run inference
323	    print(f"\nRunning inference (batch_size={args.batch_size}, "
324	          f"force_language={not args.no_force_language})...")
325	    t0 = time.time()
326	    results = run_inference(model, dataset, args.batch_size,
327	                            force_language=not args.no_force_language)
328	    inference_time = time.time() - t0
329
330	    total_audio_sec = sum(r["duration"] for r in results)
331	    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0
332	    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")
333
334	    # Save raw predictions
335	    preds_file = output_dir / "predictions.json"
336	    with open(preds_file, "w", encoding="utf-8") as f:
337	        json.dump(results, f, ensure_ascii=False, indent=2)
338	    print(f"Predictions saved: {preds_file}")
339
340	    # Compute metrics
341	    metrics = compute_metrics(results)
342	    metrics["__meta__"] = {
343	        "checkpoint": args.checkpoint,
344	        "checkpoint_name": ckpt_name,
345	        "backend": args.backend,
346	        "dataset": args.hf_dataset,
347	        "batch_size": args.batch_size,
348	        "force_language": not args.no_force_language,
349	        "inference_time_sec": round(inference_time, 2),
350	        "total_audio_sec": round(total_audio_sec, 2),
351	        "rtf": round(rtf, 4),
352	    }
353
354	    metrics_file = output_dir / "metrics.json"
355	    with open(metrics_file, "w") as f:
356	        json.dump(metrics, f, indent=2, ensure_ascii=False)
357	    print(f"Metrics saved: {metrics_file}")
358
359	    # Print table
360	    print_metrics_table(metrics, ckpt_name)
361
362	    # Save sample analysis
363	    save_sample_analysis(results, output_dir)
364
365
366	if __name__ == "__main__":
367	    main()
368


1	# coding=utf-8
2	# Copyright 2026 The Alibaba Qwen team.
3	# SPDX-License-Identifier: Apache-2.0
4	#
5	# Licensed under the Apache License, Version 2.0 (the "License");
6	# you may not use this file except in compliance with the License.
7	# You may obtain a copy of the License at
8	#
9	#     http://www.apache.org/licenses/LICENSE-2.0
10	#
11	# Unless required by applicable law or agreed to in writing, software
12	# distributed under the License is distributed on an "AS IS" BASIS,
13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	# See the License for the specific language governing permissions and
15	# limitations under the License.
16	from dataclasses import dataclass
17	from typing import Any, Dict, List, Optional, Union
18
19	import numpy as np
20	import torch
21	from qwen_asr.core.transformers_backend import (
22	    Qwen3ASRConfig,
23	    Qwen3ASRForConditionalGeneration,
24	    Qwen3ASRProcessor,
25	)
26	from transformers import AutoConfig, AutoModel, AutoProcessor
27
28	AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
29	AutoModel.register(Qwen3ASRConfig, Qwen3ASRForConditionalGeneration)
30	AutoProcessor.register(Qwen3ASRConfig, Qwen3ASRProcessor)
31
32	from .qwen3_forced_aligner import Qwen3ForcedAligner
33	from .utils import (
34	    MAX_ASR_INPUT_SECONDS,
35	    MAX_FORCE_ALIGN_INPUT_SECONDS,
36	    SAMPLE_RATE,
37	    SUPPORTED_LANGUAGES,
38	    AudioChunk,
39	    AudioLike,
40	    chunk_list,
41	    merge_languages,
42	    normalize_audios,
43	    normalize_language_name,
44	    parse_asr_output,
45	    split_audio_into_chunks,
46	    validate_language,
47	)
48
49	try:
50	    from qwen_asr.core.vllm_backend import Qwen3ASRForConditionalGeneration
51	    from vllm import ModelRegistry
52	    ModelRegistry.register_model("Qwen3ASRForConditionalGeneration", Qwen3ASRForConditionalGeneration)
53	except:
54	    pass
55
56
57	@dataclass
58	class ASRTranscription:
59	    """
60	    One transcription result.
61
62	    Attributes:
63	        language (str):
64	            Merged language string for the sample, e.g. "Chinese" or "Chinese,English".
65	            Empty string if unknown or silent audio.
66	        text (str):
67	            Transcribed text.
68	        time_stamps (Optional[Any]):
69	            Forced aligner output (ForcedAlignResult).
70	            Present only when return_time_stamps=True.
71	    """
72	    language: str
73	    text: str
74	    time_stamps: Optional[Any] = None
75
76
77	@dataclass
78	class ASRStreamingState:
79	    """
80	    Streaming ASR state for one audio stream (single utterance).
81
82	    Attributes:
83	        unfixed_chunk_num (int):
84	            For the first N chunks, do not use previous ASR result as prefix prompt (reset prefix to "").
85	        unfixed_token_num (int):
86	            When chunk_id >= unfixed_chunk_num, rollback the last K tokens from the accumulated text
87	            before using it as prefix prompt, to reduce boundary jitter.
88	        chunk_size_sec (float):
89	            Chunk size in seconds. Audio will be fed to the model in increments of this length.
90	        chunk_size_samples (int):
91	            Chunk size in samples at 16kHz (derived from chunk_size_sec).
92	        chunk_id (int):
93	            Current chunk index (0-based).
94	        buffer (np.ndarray):
95	            Buffered PCM samples that are not yet consumed into a full chunk.
96	        audio_accum (np.ndarray):
97	            Accumulated audio from the beginning of the stream up to current time (no padding).
98	        prompt_raw (str):
99	            Base prompt generated by chat template (with generation prompt), without appended prefix text.
100	        context (str):
101	            Context string.
102	        force_language (Optional[str]):
103	            If provided, force output to be text-only by appending "language X<asr_text>" in prompt_raw,
104	            consistent with non-streaming transcribe().
105	        language (str):
106	            Latest parsed language (updated after each chunk decode). Empty if unknown/silent.
107	        text (str):
108	            Latest parsed transcription text (updated after each chunk decode).
109	        _raw_decoded (str):
110	            Internal accumulated decoded raw text (before parse_asr_output normalization).
111	            Used for rollback/token trimming and as prefix for prompting.
112	    """
113	    unfixed_chunk_num: int
114	    unfixed_token_num: int
115	    chunk_size_sec: float
116	    chunk_size_samples: int
117
118	    chunk_id: int
119	    buffer: np.ndarray
120	    audio_accum: np.ndarray
121
122	    prompt_raw: str
123	    context: str
124	    force_language: Optional[str]
125
126	    language: str
127	    text: str
128	    _raw_decoded: str
129
130
131	class Qwen3ASRModel:
132	    """
133	    Unified inference wrapper for Qwen3-ASR with two backends:
134	      - Transformers backend
135	      - vLLM backend
136
137	    It optionally supports time stamp output via Qwen3-ForcedAligner.
138
139	    Notes:
140	      - Each request uses a context text and exactly one audio.
141	      - If language is provided, the prompt will force the output to be text-only by appending
142	        "language {Language}<asr_text>" to the assistant prompt.
143	    """
144
145	    def __init__(
146	        self,
147	        backend: str,
148	        model: Any,
149	        processor: Any,
150	        sampling_params: Optional[Any] = None,
151	        forced_aligner: Optional[Qwen3ForcedAligner] = None,
152	        max_inference_batch_size: int = -1,
153	        max_new_tokens: int = 512,
154	    ):
155	        self.backend = backend  # "transformers" | "vllm"
156	        self.model = model
157	        self.processor = processor
158	        self.sampling_params = sampling_params
159	        self.forced_aligner = forced_aligner
160	        self.max_inference_batch_size = int(max_inference_batch_size)
161	        self.max_new_tokens = max_new_tokens
162
163	        if backend == "transformers":
164	            self.device = getattr(model, "device", None)
165	            if self.device is None:
166	                try:
167	                    self.device = next(model.parameters()).device
168	                except StopIteration:
169	                    self.device = torch.device("cpu")
170	            self.dtype = getattr(model, "dtype", torch.float32)
171	        else:
172	            self.device = None
173	            self.dtype = None
174
175	    @classmethod
176	    def from_pretrained(
177	        cls,
178	        pretrained_model_name_or_path: str,
179	        forced_aligner: Optional[str] = None,
180	        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
181	        max_inference_batch_size: int = 32,
182	        max_new_tokens: Optional[int] = 512,
183	        **kwargs,
184	    ) -> "Qwen3ASRModel":
185	        """
186	        Initialize using Transformers backend.
187
188	        Args:
189	            pretrained_model_name_or_path:
190	                HuggingFace repo id or local directory.
191	            forced_aligner:
192	                Optional forced aligner model path/repo id.
193	            forced_aligner_kwargs:
194	                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
195	            max_inference_batch_size:
196	                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
197	            max_new_tokens:
198	                Maximum number of tokens to generate.
199	            **kwargs:
200	                Forwarded to AutoModel.from_pretrained(...).
201
202	        Returns:
203	            Qwen3ASRModel
204	        """
205
206	        model = AutoModel.from_pretrained(pretrained_model_name_or_path, **kwargs)
207
208	        processor = AutoProcessor.from_pretrained(pretrained_model_name_or_path, fix_mistral_regex=True)
209
210	        forced_aligner_model = None
211	        if forced_aligner is not None:
212	            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
213	                forced_aligner, **(forced_aligner_kwargs or {})
214	            )
215
216	        return cls(
217	            backend="transformers",
218	            model=model,
219	            processor=processor,
220	            sampling_params=None,
221	            forced_aligner=forced_aligner_model,
222	            max_inference_batch_size=max_inference_batch_size,
223	            max_new_tokens=max_new_tokens,
224	        )
225
226	    @classmethod
227	    def LLM(
228	        cls,
229	        model: str,
230	        forced_aligner: Optional[str] = None,
231	        forced_aligner_kwargs: Optional[Dict[str, Any]] = None,
232	        max_inference_batch_size: int = -1,
233	        max_new_tokens: Optional[int] = 4096,
234	        **kwargs,
235	    ) -> "Qwen3ASRModel":
236	        """
237	        Initialize using vLLM backend.
238
239	        Import is isolated to keep vLLM optional.
240
241	        Args:
242	            model:
243	                Model path/repo for vLLM.
244	            forced_aligner:
245	                Optional forced aligner model path/repo id.
246	            forced_aligner_kwargs:
247	                Optional kwargs forwarded to Qwen3ForcedAligner.from_pretrained(...).
248	            max_inference_batch_size:
249	                Batch size limit for inference. -1 means no chunking. Small values can avoid OOM.
250	            max_new_tokens:
251	                Maximum number of tokens to generate.
252	            **kwargs:
253	                Forwarded to vllm.LLM(...).
254
255	        Returns:
256	            Qwen3ASRModel
257
258	        Raises:
259	            ImportError: If vLLM is not installed.
260	        """
261	        try:
262	            from vllm import LLM as vLLM
263	            from vllm import SamplingParams
264	        except Exception as e:
265	            raise ImportError(
266	                "vLLM is not available. Install with: pip install qwen-asr[vllm]"
267	            ) from e
268
269	        llm = vLLM(model=model, **kwargs)
270
271	        processor = Qwen3ASRProcessor.from_pretrained(model, fix_mistral_regex=True)
272	        sampling_params = SamplingParams(**({"temperature": 0.0, "max_tokens": max_new_tokens}))
273
274	        forced_aligner_model = None
275	        if forced_aligner is not None:
276	            forced_aligner_model = Qwen3ForcedAligner.from_pretrained(
277	                forced_aligner, **(forced_aligner_kwargs or {})
278	            )
279
280	        return cls(
281	            backend="vllm",
282	            model=llm,
283	            processor=processor,
284	            sampling_params=sampling_params,
285	            forced_aligner=forced_aligner_model,
286	            max_inference_batch_size=max_inference_batch_size,
287	            max_new_tokens=None,
288	        )
289
290	    def get_supported_languages(self) -> List[str]:
291	        """
292	        Returns the supported language list.
293
294	        Returns:
295	            List[str]: Canonical language names.
296	        """
297	        return list(SUPPORTED_LANGUAGES)
298
299	    @torch.no_grad()
300	    def transcribe(
301	        self,
302	        audio: Union[AudioLike, List[AudioLike]],
303	        context: Union[str, List[str]] = "",
304	        language: Optional[Union[str, List[Optional[str]]]] = None,
305	        return_time_stamps: bool = False,
306	    ) -> List[ASRTranscription]:
307	        """
308	        Transcribe audio with optional context and optional forced alignment timestamps.
309
310	        Args:
311	            audio:
312	                Audio input(s). Supported:
313	                  - str: local path / URL / base64 data url
314	                  - (np.ndarray, sr)
315	                  - list of above
316	            context:
317	                Context string(s). If scalar, it will be broadcast to batch size.
318	            language:
319	                Optional language(s). If provided, it must be in supported languages.
320	                If scalar, it will be broadcast to batch size.
321	                If provided, the prompt will force output to be transcription text only.
322	            return_time_stamps:
323	                If True, timestamps are produced via forced aligner and merged across chunks.
324	                This requires forced_aligner initialized.
325
326	        Returns:
327	            List[ASRTranscription]: One result per input audio.
328
329	        Raises:
330	            ValueError:
331	                - If return_time_stamps=True but forced_aligner is not provided.
332	                - If language is unsupported.
333	                - If batch sizes mismatch for context/language.
334	        """
335	        if return_time_stamps and self.forced_aligner is None:
336	            raise ValueError("return_time_stamps=True requires `forced_aligner` to be provided at initialization.")
337
338	        wavs = normalize_audios(audio)
339	        n = len(wavs)
340
341	        ctxs = context if isinstance(context, list) else [context]
342	        if len(ctxs) == 1 and n > 1:
343	            ctxs = ctxs * n
344	        if len(ctxs) != n:
345	            raise ValueError(f"Batch size mismatch: audio={n}, context={len(ctxs)}")
346
347	        langs_in: List[Optional[str]]
348	        if language is None:
349	            langs_in = [None] * n
350	        else:
351	            langs_in = language if isinstance(language, list) else [language]
352	            if len(langs_in) == 1 and n > 1:
353	                langs_in = langs_in * n
354	            if len(langs_in) != n:
355	                raise ValueError(f"Batch size mismatch: audio={n}, language={len(langs_in)}")
356
357	        langs_norm: List[Optional[str]] = []
358	        for l in langs_in:
359	            if l is None or str(l).strip() == "":
360	                langs_norm.append(None)
361	            else:
362	                ln = normalize_language_name(str(l))
363	                validate_language(ln)
364	                langs_norm.append(ln)
365
366	        max_chunk_sec = MAX_FORCE_ALIGN_INPUT_SECONDS if return_time_stamps else MAX_ASR_INPUT_SECONDS
367
368	        # chunk audios and record mapping
369	        chunks: List[AudioChunk] = []
370	        for i, wav in enumerate(wavs):
371	            parts = split_audio_into_chunks(
372	                wav=wav,
373	                sr=SAMPLE_RATE,
374	                max_chunk_sec=max_chunk_sec,
375	            )
376	            for j, (cwav, offset_sec) in enumerate(parts):
377	                chunks.append(AudioChunk(orig_index=i, chunk_index=j, wav=cwav, sr=SAMPLE_RATE, offset_sec=offset_sec))
378
379	        # run ASR on chunks
380	        chunk_ctx: List[str] = [ctxs[c.orig_index] for c in chunks]
381	        chunk_lang: List[Optional[str]] = [langs_norm[c.orig_index] for c in chunks]
382	        chunk_wavs: List[np.ndarray] = [c.wav for c in chunks]
383	        raw_outputs = self._infer_asr(chunk_ctx, chunk_wavs, chunk_lang)
384
385	        # parse outputs, prepare for optional alignment
386	        per_chunk_lang: List[str] = []
387	        per_chunk_text: List[str] = []
388	        for out, forced_lang in zip(raw_outputs, chunk_lang):
389	            lang, txt = parse_asr_output(out, user_language=forced_lang)
390	            per_chunk_lang.append(lang)
391	            per_chunk_text.append(txt)
392
393	        # forced alignment (optional)
394	        per_chunk_align: List[Optional[Any]] = [None] * len(chunks)
395	        if return_time_stamps:
396	            to_align_audio = []
397	            to_align_text = []
398	            to_align_lang = []
399	            to_align_idx = []
400
401	            for idx, (c, txt, lang_pred) in enumerate(zip(chunks, per_chunk_text, per_chunk_lang)):
402	                if txt.strip() == "":
403	                    continue
404	                to_align_audio.append((c.wav, c.sr))
405	                to_align_text.append(txt)
406	                to_align_lang.append(lang_pred)
407	                to_align_idx.append(idx)
408
409	            # batch align with max_inference_batch_size
410	            aligned_results: List[Any] = []
411	            for a_chunk, t_chunk, l_chunk in zip(
412	                chunk_list(to_align_audio, self.max_inference_batch_size),
413	                chunk_list(to_align_text, self.max_inference_batch_size),
414	                chunk_list(to_align_lang, self.max_inference_batch_size),
415	            ):
416	                aligned_results.extend(
417	                    self.forced_aligner.align(audio=a_chunk, text=t_chunk, language=l_chunk)
418	                )
419
420	            # offset fix
421	            for k, idx in enumerate(to_align_idx):
422	                c = chunks[idx]
423	                r = aligned_results[k]
424	                per_chunk_align[idx] = self._offset_align_result(r, c.offset_sec)
425
426	        # merge chunks back to original samples
427	        out_langs: List[List[str]] = [[] for _ in range(n)]
428	        out_texts: List[List[str]] = [[] for _ in range(n)]
429	        out_aligns: List[List[Any]] = [[] for _ in range(n)]
430
431	        for c, lang, txt, al in zip(chunks, per_chunk_lang, per_chunk_text, per_chunk_align):
432	            out_langs[c.orig_index].append(lang)
433	            out_texts[c.orig_index].append(txt)
434	            if return_time_stamps and al is not None:
435	                out_aligns[c.orig_index].append(al)
436
437	        results: List[ASRTranscription] = []
438	        for i in range(n):
439	            merged_text = "".join([t for t in out_texts[i] if t is not None])
440	            merged_language = merge_languages(out_langs[i])
441	            merged_align = None
442	            if return_time_stamps:
443	                merged_align = self._merge_align_results(out_aligns[i])
444	            results.append(ASRTranscription(language=merged_language, text=merged_text, time_stamps=merged_align))
445
446	        return results
447
448	    def _build_messages(self, context: str, audio_payload: Any) -> List[Dict[str, Any]]:
449	        return [
450	            {"role": "system", "content": context or ""},
451	            {"role": "user", "content": [{"type": "audio", "audio": audio_payload}]},
452	        ]
453
454	    def _build_text_prompt(self, context: str, force_language: Optional[str]) -> str:
455	        """
456	        Build the string prompt for one request.
457
458	        If force_language is provided, "language X<asr_text>" is appended after the generation prompt
459	        to request text-only output.
460	        """
461	        msgs = self._build_messages(context=context, audio_payload="")
462	        base = self.processor.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
463	        if force_language:
464	            base = base + f"language {force_language}{'<asr_text>'}"
465	        return base
466
467	    def _infer_asr(
468	        self,
469	        contexts: List[str],
470	        wavs: List[np.ndarray],
471	        languages: List[Optional[str]],
472	    ) -> List[str]:
473	        """
474	        Run backend inference for chunk-level items.
475
476	        Args:
477	            contexts: List of context strings.
478	            wavs: List of mono waveforms (np.ndarray).
479	            languages: List of forced languages or None.
480
481	        Returns:
482	            List[str]: Raw decoded strings (one per chunk).
483	        """
484	        if self.backend == "transformers":
485	            return self._infer_asr_transformers(contexts, wavs, languages)
486	        if self.backend == "vllm":
487	            return self._infer_asr_vllm(contexts, wavs, languages)
488	        raise RuntimeError(f"Unknown backend: {self.backend}")
489
490	    def _infer_asr_transformers(
491	        self,
492	        contexts: List[str],
493	        wavs: List[np.ndarray],
494	        languages: List[Optional[str]],
495	    ) -> List[str]:
496	        outs: List[str] = []
497
498	        texts = [self._build_text_prompt(context=c, force_language=fl) for c, fl in zip(contexts, languages)]
499
500	        batch_size = self.max_inference_batch_size
501	        if batch_size is None or batch_size < 0:
502	            batch_size = len(texts)
503
504	        for i in range(0, len(texts), batch_size):
505	            sub_text = texts[i : i + batch_size]
506	            sub_wavs = wavs[i : i + batch_size]
507	            inputs = self.processor(text=sub_text, audio=sub_wavs, return_tensors="pt", padding=True)
508	            inputs = inputs.to(self.model.device).to(self.model.dtype)
509
510	            text_ids = self.model.generate(**inputs, max_new_tokens=self.max_new_tokens)
511
512	            decoded = self.processor.batch_decode(
513	                text_ids.sequences[:, inputs["input_ids"].shape[1]:],
514	                skip_special_tokens=True,
515	                clean_up_tokenization_spaces=False,
516	            )
517	            outs.extend(list(decoded))
518
519	        return outs
520
521	    def _infer_asr_vllm(
522	        self,
523	        contexts: List[str],
524	        wavs: List[np.ndarray],
525	        languages: List[Optional[str]],
526	    ) -> List[str]:
527	        inputs: List[Dict[str, Any]] = []
528	        for c, w, fl in zip(contexts, wavs, languages):
529	            prompt = self._build_text_prompt(context=c, force_language=fl)
530	            inputs.append({"prompt": prompt, "multi_modal_data": {"audio": [w]}})
531
532	        outs: List[str] = []
533	        for batch in chunk_list(inputs, self.max_inference_batch_size):
534	            outputs = self.model.generate(batch, sampling_params=self.sampling_params, use_tqdm=False)
535	            for o in outputs:
536	                outs.append(o.outputs[0].text)
537	        return outs
538
539	    def _offset_align_result(self, result: Any, offset_sec: float) -> Any:
540	        """
541	        Apply time offset to a ForcedAlignResult-like object.
542
543	        This function assumes:
544	          - result has attribute `.items` which is a list of items with start_time/end_time in seconds.
545	          - dataclasses are frozen in upstream implementation, so we reconstruct by type.
546
547	        Args:
548	            result: ForcedAlignResult
549	            offset_sec: Offset in seconds
550
551	        Returns:
552	            ForcedAlignResult: New object with shifted timestamps.
553	        """
554	        if result is None:
555	            return None
556	        items = []
557	        for it in result.items:
558	            items.append(type(it)(text=it.text,
559	                                  start_time=round(it.start_time + offset_sec, 3),
560	                                  end_time=round(it.end_time + offset_sec, 3)))
561	        return type(result)(items=items)
562
563	    def _merge_align_results(self, results: List[Any]) -> Optional[Any]:
564	        """
565	        Merge multiple ForcedAlignResult objects into a single one by concatenating items.
566
567	        Args:
568	            results: List of ForcedAlignResult
569
570	        Returns:
571	            ForcedAlignResult or None
572	        """
573	        if not results:
574	            return None
575	        all_items = []
576	        for r in results:
577	            if r is None:
578	                continue
579	            all_items.extend(list(r.items))
580	        if not all_items:
581	            return None
582	        return type(results[0])(items=all_items)
583
584	    def init_streaming_state(
585	        self,
586	        context: str = "",
587	        language: Optional[str] = None,
588	        unfixed_chunk_num: int = 2,
589	        unfixed_token_num: int = 5,
590	        chunk_size_sec: float = 2.0,
591	    ) -> ASRStreamingState:
592	        """
593	        Initialize streaming ASR state for a single stream.
594
595	        Notes:
596	            - Streaming ASR is supported ONLY for vLLM backend.
597	            - Streaming ASR does NOT support timestamps (forced aligner is not used).
598	            - Batch inference is NOT supported.
599
600	        Args:
601	            context:
602	                Context string.
603	            language:
604	                Optional forced language. If provided, it must be in supported languages.
605	                Same behavior as transcribe(): forces text-only output via prompt suffix.
606	            unfixed_chunk_num:
607	                For the first N chunks, do not use previous output as prefix prompt (reset prefix to "").
608	            unfixed_token_num:
609	                Roll back the last K tokens from accumulated output when using it as prefix prompt
610	                after unfixed_chunk_num.
611	            chunk_size_sec:
612	                Chunk size in seconds (audio is 16k PCM). The function will internally convert it
613	                to sample count at 16kHz.
614
615	        Returns:
616	            ASRStreamingState: Mutable state object to be passed to streaming_transcribe() and
617	            finish_streaming_transcribe().
618
619	        Raises:
620	            ValueError:
621	                - If backend is not "vllm".
622	                - If chunk_size_sec <= 0.
623	                - If forced language is invalid (same validation rules as transcribe()).
624	        """
625	        if self.backend != "vllm":
626	            raise ValueError("Streaming ASR is supported only for vLLM backend (backend='vllm').")
627	        if chunk_size_sec is None or float(chunk_size_sec) <= 0:
628	            raise ValueError(f"chunk_size_sec must be > 0, got: {chunk_size_sec}")
629
630	        force_language = None
631	        if language is not None and str(language).strip() != "":
632	            ln = normalize_language_name(str(language))
633	            validate_language(ln)
634	            force_language = ln
635
636	        chunk_size_samples = int(round(float(chunk_size_sec) * SAMPLE_RATE))
637	        chunk_size_samples = max(1, chunk_size_samples)
638
639	        prompt_raw = self._build_text_prompt(context=context, force_language=force_language)
640
641	        return ASRStreamingState(
642	            unfixed_chunk_num=int(unfixed_chunk_num),
643	            unfixed_token_num=int(unfixed_token_num),
644	            chunk_size_sec=float(chunk_size_sec),
645	            chunk_size_samples=int(chunk_size_samples),
646	            chunk_id=0,
647	            buffer=np.zeros((0,), dtype=np.float32),
648	            audio_accum=np.zeros((0,), dtype=np.float32),
649	            prompt_raw=prompt_raw,
650	            context=context or "",
651	            force_language=force_language,
652	            language="",
653	            text="",
654	            _raw_decoded="",
655	        )
656
657	    def streaming_transcribe(self, pcm16k: np.ndarray, state: ASRStreamingState) -> ASRStreamingState:
658	        """
659	        Streaming ASR decode step.
660
661	        This function accepts an arbitrary-length 16k PCM float numpy array (mono).
662	        It buffers incoming samples, and whenever enough samples are accumulated to form one
663	        full chunk (chunk_size_sec), it runs one incremental decode step and updates:
664
665	          - state.language
666	          - state.text
667
668	        The caller only needs to keep passing audio to this function and read state.language/state.text.
669
670	        Implementation details:
671	            - Each time a new chunk is ready, we append it to audio_accum and re-feed *all* audio seen
672	              so far to the model (no padding).
673	            - We update the prompt as: state.prompt_raw + prefix_text
674	            - Prefix rollback strategy:
675	                * If chunk_id < unfixed_chunk_num: prefix_text = ""
676	                * Else: rollback last unfixed_token_num tokens from previously accumulated decoded text.
677
678	        Notes:
679	            - vLLM backend only.
680	            - No timestamps.
681	            - Single stream only (no batching).
682
683	        Args:
684	            pcm16k:
685	                16kHz mono PCM waveform (np.ndarray). Length can be any non-negative integer.
686	                dtype can be float32/float64/int16; it will be converted to float32.
687	            state:
688	                Streaming state returned by init_streaming_state().
689
690	        Returns:
691	            ASRStreamingState: The same state object (mutated) for convenience.
692
693	        Raises:
694	            ValueError:
695	                If backend is not "vllm" or state is invalid.
696	        """
697	        if self.backend != "vllm":
698	            raise ValueError("streaming_transcribe() is supported only for vLLM backend (backend='vllm').")
699	        if state is None:
700	            raise ValueError("state must not be None. Call init_streaming_state() first.")
701	        if pcm16k is None:
702	            raise ValueError("pcm16k must not be None.")
703
704	        # Ensure 1D mono
705	        x = np.asarray(pcm16k)
706	        if x.ndim != 1:
707	            x = x.reshape(-1)
708
709	        # Convert to float32 PCM in [-1, 1] if int16 provided
710	        if x.dtype == np.int16:
711	            x = (x.astype(np.float32) / 32768.0)
712	        else:
713	            x = x.astype(np.float32, copy=False)
714
715	        # Append to buffer
716	        if x.shape[0] > 0:
717	            state.buffer = np.concatenate([state.buffer, x], axis=0)
718
719	        # Consume full chunks
720	        while state.buffer.shape[0] >= state.chunk_size_samples:
721	            chunk = state.buffer[: state.chunk_size_samples]
722	            state.buffer = state.buffer[state.chunk_size_samples :]
723
724	            # Accumulate audio (re-feed from start, no padding)
725	            if state.audio_accum.shape[0] == 0:
726	                state.audio_accum = chunk
727	            else:
728	                state.audio_accum = np.concatenate([state.audio_accum, chunk], axis=0)
729
730	            # Build prefix with rollback strategy
731	            prefix = ""
732	            if state.chunk_id < state.unfixed_chunk_num:
733	                prefix = ""
734	            else:
735	                cur_ids = self.processor.tokenizer.encode(state._raw_decoded)
736	                k = int(state.unfixed_token_num)
737	                while True:
738	                    end_idx = max(0, len(cur_ids) - k)
739	                    prefix = self.processor.tokenizer.decode(cur_ids[:end_idx]) if end_idx > 0 else ""
740	                    if '\ufffd' not in prefix:
741	                        break
742	                    else:
743	                        if end_idx == 0:
744	                            prefix = ""
745	                            break
746	                        k += 1
747
748	            prompt = state.prompt_raw + prefix
749
750	            # vLLM input: single item
751	            inp = {"prompt": prompt, "multi_modal_data": {"audio": [state.audio_accum]}}
752
753	            outputs = self.model.generate([inp], sampling_params=self.sampling_params, use_tqdm=False)
754	            gen_text = outputs[0].outputs[0].text
755
756	            # Accumulate raw decoded (then parse to lang/text)
757	            state._raw_decoded = (prefix + gen_text) if prefix is not None else gen_text
758
759	            lang, txt = parse_asr_output(state._raw_decoded, user_language=state.force_language)
760	            state.language = lang
761	            state.text = txt
762
763	            state.chunk_id += 1
764
765	        return state
766
767	    def finish_streaming_transcribe(self, state: ASRStreamingState) -> ASRStreamingState:
768	        """
769	        Finish streaming ASR.
770
771	        This function flushes the remaining buffered audio in state.buffer (tail audio).
772	        It sends the remaining samples to the model even if shorter than chunk_size_sec,
773	        without padding. Then it updates state.language/state.text one last time.
774
775	        Notes:
776	            - vLLM backend only.
777	            - No timestamps.
778	            - Single stream only.
779
780	        Args:
781	            state:
782	                Streaming state.
783
784	        Returns:
785	            ASRStreamingState: Updated state (mutated).
786
787	        Raises:
788	            ValueError:
789	                If backend is not "vllm" or state is invalid.
790	        """
791	        if self.backend != "vllm":
792	            raise ValueError("finish_streaming_transcribe() is supported only for vLLM backend (backend='vllm').")
793	        if state is None:
794	            raise ValueError("state must not be None.")
795
796	        # If no remaining buffer, still return state as-is.
797	        if state.buffer is None or state.buffer.shape[0] == 0:
798	            return state
799
800	        tail = state.buffer
801	        state.buffer = np.zeros((0,), dtype=np.float32)
802
803	        # Append tail to accumulated audio
804	        if state.audio_accum.shape[0] == 0:
805	            state.audio_accum = tail
806	        else:
807	            state.audio_accum = np.concatenate([state.audio_accum, tail], axis=0)
808
809	        # Prefix rollback strategy (same as per-chunk)
810	        prefix = ""
811	        if state.chunk_id < state.unfixed_chunk_num:
812	            prefix = ""
813	        else:
814	            cur_ids = self.processor.tokenizer.encode(state._raw_decoded)
815	            end_idx = max(1, len(cur_ids) - int(state.unfixed_token_num))
816	            prefix = self.processor.tokenizer.decode(cur_ids[:end_idx])
817
818	        prompt = state.prompt_raw + prefix
819	        inp = {"prompt": prompt, "multi_modal_data": {"audio": [state.audio_accum]}}
820
821	        outputs = self.model.generate([inp], sampling_params=self.sampling_params, use_tqdm=False)
822	        gen_text = outputs[0].outputs[0].text
823
824	        state._raw_decoded = (prefix + gen_text) if prefix is not None else gen_text
825	        lang, txt = parse_asr_output(state._raw_decoded, user_language=state.force_language)
826	        state.language = lang
827	        state.text = txt
828
829	        state.chunk_id += 1
830	        return state


Error: File content (11102 tokens) exceeds maximum allowed tokens (10000). Use offset and limit parameters to read specific portions of the file, or search for specific content instead of reading the whole file.


Let me read the vllm backend file more carefully:


1	# SPDX-License-Identifier: Apache-2.0
2	# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3	# Copyright 2026 The Qwen team.
4	# Copyright 2023 The vLLM team.
5	# Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved.
6	#
7	# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
8	# and OPT implementations in this library. It has been modified from its
9	# original forms to accommodate minor architectural differences compared
10	# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
11	#
12	# Licensed under the Apache License, Version 2.0 (the "License");
13	# you may not use this file except in compliance with the License.
14	# You may obtain a copy of the License at
15	#
16	#     http://www.apache.org/licenses/LICENSE-2.0
17	#
18	# Unless required by applicable law or agreed to in writing, software
19	# distributed under the License is distributed on an "AS IS" BASIS,
20	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
21	# See the License for the specific language governing permissions and
22	# limitations under the License.
23	"""Inference-only Qwen3-ASR model."""
24
25	from collections.abc import Iterable, Mapping, Sequence
26	from typing import Any, Literal, cast
27
28	import numpy as np
29	import torch
30	import torch.nn as nn
31	import torch.nn.functional as F
32	from transformers.feature_extraction_utils import BatchFeature
33	from transformers.models.whisper import WhisperFeatureExtractor
34
35	from vllm.config import MultiModalConfig, ModelConfig, SpeechToTextConfig, VllmConfig
36	from vllm.config.multimodal import BaseDummyOptions
37	from vllm.distributed import get_tensor_model_parallel_world_size
38	from vllm.inputs.data import PromptType
39	from vllm.logger import init_logger
40	from vllm.model_executor.layers.activation import _ACTIVATION_REGISTRY
41	from vllm.model_executor.layers.attention.mm_encoder_attention import (
42	    MMEncoderAttention,
43	)
44	from vllm.model_executor.layers.linear import (
45	    ColumnParallelLinear,
46	    QKVParallelLinear,
47	    RowParallelLinear,
48	)
49	from vllm.model_executor.model_loader.weight_utils import default_weight_loader
50	from vllm.model_executor.models.interfaces import (
51	    MultiModalEmbeddings,
52	    SupportsMRoPE,
53	    SupportsMultiModal,
54	    SupportsPP,
55	    SupportsTranscription,
56	)
57	from vllm.model_executor.models.module_mapping import MultiModelKeys
58	from vllm.model_executor.models.qwen3 import Qwen3ForCausalLM
59	from vllm.model_executor.models.qwen3_omni_moe_thinker import (
60	    Qwen2_5OmniAudioFeatureInputs,
61	    Qwen3OmniMoeThinkerMultiModalProcessor,
62	)
63	from vllm.model_executor.models.utils import (
64	    AutoWeightsLoader,
65	    WeightsMapper,
66	    _merge_multimodal_embeddings,
67	    maybe_prefix,
68	)
69	from vllm.model_executor.models.whisper import ISO639_1_SUPPORTED_LANGS
70	from vllm.multimodal import MULTIMODAL_REGISTRY
71	from vllm.multimodal.inputs import (
72	    AudioItem,
73	    ModalityData,
74	    MultiModalDataDict,
75	    MultiModalFeatureSpec,
76	    MultiModalFieldConfig,
77	    MultiModalKwargsItems,
78	)
79	from vllm.multimodal.parse import (
80	    AudioProcessorItems,
81	    DictEmbeddingItems,
82	    ModalityDataItems,
83	    MultiModalDataItems,
84	    MultiModalDataParser,
85	)
86	from vllm.multimodal.processing import (
87	    BaseProcessingInfo,
88	    PromptReplacement,
89	    PromptUpdate,
90	)
91	from vllm.sequence import IntermediateTensors
92	from vllm.v1.attention.backends.registry import AttentionBackendEnum
93	from vllm.tokenizers import cached_tokenizer_from_config
94	from vllm.transformers_utils.processor import cached_processor_from_config
95	from vllm.model_executor.models.vision import (
96	    get_vit_attn_backend,
97	)
98	from ..transformers_backend.configuration_qwen3_asr import (
99	    Qwen3ASRConfig,
100	    Qwen3ASRThinkerConfig,


find /home/ubuntu/training/qwen3-asr-1.7b-phase2-sft -name "*.py" | grep -E "(transformers|config|__init__)" | head -15


Perfect! Now let me check the key files: the __init__.py, config, and transformers backend modeling:


1	# coding=utf-8
2	# Copyright 2026 The Alibaba Qwen team.
3	# SPDX-License-Identifier: Apache-2.0
4	#
5	# Licensed under the Apache License, Version 2.0 (the "License");
6	# you may not use this file except in compliance with the License.
7	# You may obtain a copy of the License at
8	#
9	#     http://www.apache.org/licenses/LICENSE-2.0
10	#
11	# Unless required by applicable law or agreed to in writing, software
12	# distributed under the License is distributed on an "AS IS" BASIS,
13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	# See the License for the specific language governing permissions and
15	# limitations under the License.
16	"""
17	qwen_asr: Qwen3-ASR package.
18	"""
19
20	from .inference.qwen3_asr import Qwen3ASRModel
21	from .inference.qwen3_forced_aligner import Qwen3ForcedAligner
22
23	from .inference.utils import parse_asr_output
24
25	__all__ = ["__version__"]
26


1	# coding=utf-8
2	# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
3	#
4	# Licensed under the Apache License, Version 2.0 (the "License");
5	# you may not use this file except in compliance with the License.
6	# You may obtain a copy of the License at
7	#
8	#     http://www.apache.org/licenses/LICENSE-2.0
9	#
10	# Unless required by applicable law or agreed to in writing, software
11	# distributed under the License is distributed on an "AS IS" BASIS,
12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	# See the License for the specific language governing permissions and
14	# limitations under the License.
15	from transformers.configuration_utils import PretrainedConfig
16	from transformers.utils import logging
17
18
19	logger = logging.get_logger(__name__)
20
21
22	class Qwen3ASRAudioEncoderConfig(PretrainedConfig):
23	    r"""
24	    This is the configuration class to store the configuration of a [`Qwen3ASRAudioEncoder`]. It is used to instantiate a
25	    Qwen3-ASR audio encoder according to the specified arguments, defining the model architecture. Instantiating a
26	    configuration with the defaults will yield a similar configuration to that of the audio encoder of the Qwen2-Audio
27	    architecture.
28
29	    e.g. [Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
30
31	    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
32	    documentation from [`PretrainedConfig`] for more information.
33
34	    Args:
35	        num_mel_bins (`int`, *optional*, defaults to 128):
36	            Number of mel features used per input features. Should correspond to the value used in the
37	            `Qwen3ASRProcessor` class.
38	        encoder_layers (`int`, *optional*, defaults to 32):
39	            Number of encoder layers.
40	        encoder_attention_heads (`int`, *optional*, defaults to 20):
41	            Number of attention heads for each attention layer in the Transformer encoder.
42	        encoder_ffn_dim (`int`, *optional*, defaults to 5120):
43	            Dimensionality of the "intermediate" (often named feed-forward) layer in encoder.
44	        d_model (`int`, *optional*, defaults to 1280):
45	            Dimensionality of the layers.
46	        dropout (`float`, *optional*, defaults to 0.0):
47	            The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
48	        attention_dropout (`float`, *optional*, defaults to 0.0):
49	            The dropout ratio for the attention probabilities.
50	        activation_function (`str`, *optional*, defaults to `"gelu"`):
51	            The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`,
52	            `"relu"`, `"silu"` and `"gelu_new"` are supported.
53	        activation_dropout (`float`, *optional*, defaults to 0.0):
54	            The dropout ratio for activations inside the fully connected layer.
55	        scale_embedding (`bool`, *optional*, defaults to `False`):
56	            Scale embeddings by diving by sqrt(d_model).
57	        initializer_range (`float`, *optional*, defaults to 0.02):
58	            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
59	        max_source_positions (`int`, *optional*, defaults to 1500):
60	            The maximum sequence length of log-mel filter-bank features that this model might ever be used with.
61	        n_window (`int`, *optional*, defaults to 100):
62	            The chunk for conv and flash attn in AudioEncoder.
63	        output_dim (`int`, *optional*, defaults to 3584):
64	            The output dimension of AudioEncoder.
65
66	    Example:
67
68	    ```python
69	    >>> from transformers import Qwen3ASRAudioEncoderConfig, Qwen3ASRAudioEncoder
70
71	    >>> # Initializing a Qwen3ASRAudioEncoderConfig
72	    >>> configuration = Qwen3ASRAudioEncoderConfig()
73
74	    >>> # Initializing a Qwen3ASRAudioEncoder (with random weights)
75	    >>> model = Qwen3ASRAudioEncoder(configuration)
76
77	    >>> # Accessing the model configuration
78	    >>> configuration = model.config
79	    ```"""
80


1	# coding=utf-8
2	# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
3	#
4	# Licensed under the Apache License, Version 2.0 (the "License");
5	# you may not use this file except in compliance with the License.
6	# You may obtain a copy of the License at
7	#
8	#     http://www.apache.org/licenses/LICENSE-2.0
9	#
10	# Unless required by applicable law or agreed to in writing, software
11	# distributed under the License is distributed on an "AS IS" BASIS,
12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	# See the License for the specific language governing permissions and
14	# limitations under the License.
15	import math
16	from dataclasses import dataclass
17	from typing import Callable, Optional, Union
18
19	import numpy as np
20	import torch
21	from torch import nn
22	from torch.nn import functional as F
23
24	from transformers.activations import ACT2FN
25	from transformers.cache_utils import Cache, DynamicCache
26	from transformers.generation import GenerationMixin
27	from transformers.integrations import use_kernel_forward_from_hub
28	from transformers.masking_utils import create_causal_mask
29	from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
30	from transformers.modeling_layers import GradientCheckpointingLayer
31	from transformers.modeling_outputs import (
32	    BaseModelOutput,
33	    BaseModelOutputWithPast,
34	    MoeCausalLMOutputWithPast,
35	)
36	from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
37	from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
38	from transformers.processing_utils import Unpack
39	from transformers.utils import auto_docstring, can_return_tuple
40	from transformers.utils.deprecation import deprecate_kwarg
41	from transformers.utils.generic import TransformersKwargs, check_model_inputs
42
43	from .configuration_qwen3_asr import (
44	    Qwen3ASRAudioEncoderConfig,
45	    Qwen3ASRConfig,
46	    Qwen3ASRThinkerConfig,
47	)
48
49
50	@use_kernel_forward_from_hub("RMSNorm")
51	class Qwen3ASRTextRMSNorm(nn.Module):
52	    def __init__(self, hidden_size, eps: float = 1e-6) -> None:
53	        """
54	        Qwen3ASRTextRMSNorm is equivalent to T5LayerNorm
55	        """
56	        super().__init__()
57	        self.weight = nn.Parameter(torch.ones(hidden_size))
58	        self.variance_epsilon = eps
59
60	    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
61	        input_dtype = hidden_states.dtype
62	        hidden_states = hidden_states.to(torch.float32)
63	        variance = hidden_states.pow(2).mean(-1, keepdim=True)
64	        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
65	        return self.weight * hidden_states.to(input_dtype)
66
67	    def extra_repr(self):
68	        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
69
70
71	def rotate_half(x):
72	    """Rotates half the hidden dims of the input."""
73	    x1 = x[..., : x.shape[-1] // 2]
74	    x2 = x[..., x.shape[-1] // 2 :]
75	    return torch.cat((-x2, x1), dim=-1)
76
77
78	def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
79	    """
80	    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
81	    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
82	    """
83	    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
84	    if n_rep == 1:
85	        return hidden_states
86	    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
87	    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
88
89
90	def eager_attention_forward(
91	    module: nn.Module,
92	    query: torch.Tensor,
93	    key: torch.Tensor,
94	    value: torch.Tensor,
95	    attention_mask: Optional[torch.Tensor],
96	    scaling: float,
97	    dropout: float = 0.0,
98	    **kwargs: Unpack[TransformersKwargs],
99	):
100	    key_states = repeat_kv(key, module.num_key_value_groups)
101	    value_states = repeat_kv(value, module.num_key_value_groups)
102
103	    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
104	    if attention_mask is not None:
105	        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
106	        attn_weights = attn_weights + causal_mask
107
108	    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
109	    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
110	    attn_output = torch.matmul(attn_weights, value_states)
111	    attn_output = attn_output.transpose(1, 2).contiguous()
112
113	    return attn_output, attn_weights
114
115
116	def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
117	    """Applies Rotary Position Embedding to the query and key tensors.
118
119	    Args:
120	        q (`torch.Tensor`): The query tensor.
121	        k (`torch.Tensor`): The key tensor.
122	        cos (`torch.Tensor`): The cosine part of the rotary embedding.
123	        sin (`torch.Tensor`): The sine part of the rotary embedding.
124	        position_ids (`torch.Tensor`, *optional*):
125	            Deprecated and unused.
126	        unsqueeze_dim (`int`, *optional*, defaults to 1):
127	            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
128	            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
129	            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
130	            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
131	            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
132	            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
133	    Returns:
134	        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
135	    """
136	    cos = cos.unsqueeze(unsqueeze_dim)
137	    sin = sin.unsqueeze(unsqueeze_dim)
138	    q_embed = (q * cos) + (rotate_half(q) * sin)
139	    k_embed = (k * cos) + (rotate_half(k) * sin)
140	    return q_embed, k_embed
141
142
143	class Qwen3ASRTextAttention(nn.Module):
144	    """Multi-headed attention from 'Attention Is All You Need' paper"""
145
146	    def __init__(self, config: Qwen3ASRConfig, layer_idx: int):
147	        super().__init__()
148	        self.config = config
149	        self.layer_idx = layer_idx
150	        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)


Excellent! Now let me search for the actual model class definition:


1311:class Qwen3ASRForConditionalGeneration(Qwen3ASRPreTrainedModel, GenerationMixin):


1000	            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
1001
1002	        # torch.jit.trace() doesn't support cache objects in the output
1003	        if use_cache and past_key_values is None and not torch.jit.is_tracing():
1004	            past_key_values = DynamicCache(config=self.config)
1005
1006	        if inputs_embeds is None:
1007	            inputs_embeds = self.embed_tokens(input_ids)
1008
1009	        if cache_position is None:
1010	            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
1011	            cache_position = torch.arange(
1012	                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
1013	            )
1014
1015	        # the hard coded `3` is for temporal, height and width.
1016	        if position_ids is None:
1017	            position_ids = cache_position.view(1, 1, -1).expand(3, inputs_embeds.shape[0], -1)
1018	        elif position_ids.ndim == 2:
1019	            position_ids = position_ids[None, ...].expand(3, position_ids.shape[0], -1)
1020
1021	        if position_ids.ndim == 3 and position_ids.shape[0] == 4:
1022	            text_position_ids = position_ids[0]
1023	            position_ids = position_ids[1:]
1024	        else:
1025	            text_position_ids = position_ids[0]
1026
1027	        attention_mask = create_causal_mask(
1028	            config=self.config,
1029	            input_embeds=inputs_embeds,
1030	            attention_mask=attention_mask,
1031	            cache_position=cache_position,
1032	            past_key_values=past_key_values,
1033	            position_ids=text_position_ids,
1034	        )
1035
1036	        hidden_states = inputs_embeds
1037
1038	        # create position embeddings to be shared across the decoder layers
1039	        position_embeddings = self.rotary_emb(hidden_states, position_ids)
1040
1041	        # decoder layers
1042	        for layer_idx, decoder_layer in enumerate(self.layers):
1043	            layer_outputs = decoder_layer(
1044	                hidden_states,
1045	                attention_mask=attention_mask,
1046	                position_ids=text_position_ids,
1047	                past_key_values=past_key_values,
1048	                cache_position=cache_position,
1049	                position_embeddings=position_embeddings,
1050	                **kwargs,
1051	            )
1052	            hidden_states = layer_outputs
1053
1054	        hidden_states = self.norm(hidden_states)
1055
1056	        return BaseModelOutputWithPast(
1057	            last_hidden_state=hidden_states,
1058	            past_key_values=past_key_values,
1059	        )
1060
1061
1062	@auto_docstring(
1063	    custom_intro="""
1064	    The Qwen3ASRThinker model which consists of a audio backbone and a language model.
1065	    """
1066	)
1067	class Qwen3ASRThinkerForConditionalGeneration(Qwen3ASRPreTrainedModelForConditionalGeneration, GenerationMixin):
1068	    config: Qwen3ASRThinkerConfig
1069	    base_model_prefix = "thinker"
1070	    _tied_weights_keys = ["model.embed_tokens.weight", "lm_head.weight"]
1071	    _no_split_modules = [
1072	        "Qwen3ASRAudioEncoderLayer",
1073	        "Qwen3ASRThinkerTextDecoderLayer",
1074	    ]
1075	    _can_record_outputs = {
1076	        "hidden_states": Qwen3ASRThinkerTextDecoderLayer,
1077	        "attentions": Qwen3ASRThinkerTextAttention,
1078	    }
1079
1080	    def __init__(self, config):
1081	        super().__init__(config)
1082	        self.audio_tower = Qwen3ASRAudioEncoder._from_config(config.audio_config)
1083	        self.vocab_size = config.text_config.vocab_size
1084	        self.model = Qwen3ASRThinkerTextModel._from_config(config.text_config)
1085	        if "forced_aligner" in config.model_type:
1086	            self.lm_head = nn.Linear(config.text_config.hidden_size, config.classify_num, bias=False)
1087	        else:
1088	            self.lm_head = nn.Linear(config.text_config.hidden_size, config.text_config.vocab_size, bias=False)
1089	        self.pad_token_id = self.config.pad_token_id if self.config.pad_token_id is not None else -1
1090	        self.rope_deltas = None
1091	        self.post_init()
1092
1093	    def get_input_embeddings(self):
1094	        return self.model.get_input_embeddings()
1095
1096	    def set_input_embeddings(self, value):
1097	        self.model.set_input_embeddings(value)
1098
1099	    def get_audio_features(
1100	        self,
1101	        input_features: torch.FloatTensor,
1102	        feature_attention_mask: Optional[torch.LongTensor] = None,
1103	        audio_feature_lengths: Optional[torch.LongTensor] = None,
1104	    ):
1105	        """
1106	        Encodes audios into continuous embeddings that can be forwarded to the language model.
1107
1108	        Args:
1109	            input_features (`torch.FloatTensor`):
1110	                The tensors corresponding to the input audios.
1111	            feature_attention_mask (`torch.LongTensor`, *optional*):
1112	                Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`:
1113	            audio_feature_lengths (`torch.LongTensor` of shape `(num_audios)`, *optional*):
1114	                The length of feature shape of each audio in LLM.
1115	        """
1116	        if feature_attention_mask is not None:
1117	            audio_feature_lengths = torch.sum(feature_attention_mask, dim=1)
1118	        else:
1119	            audio_feature_lengths = None
1120	        feature_lens = audio_feature_lengths if audio_feature_lengths is not None else feature_attention_mask.sum(-1)
1121
1122	        # audio encoder do not support batch inference to keep precision
1123	        audio_features = []
1124	        for input_feature, feature_len in zip(input_features, feature_lens):
1125	            audio_output = self.audio_tower(
1126	                input_feature[:, :feature_len],
1127	                feature_lens=feature_len.unsqueeze(0),
1128	            )
1129	            audio_feature = audio_output.last_hidden_state
1130	            audio_features.append(audio_feature)
1131	        audio_features = torch.cat(audio_features, dim=0)
1132
1133	        return audio_features
1134
1135	    def get_placeholder_mask(
1136	        self,
1137	        input_ids: torch.LongTensor,
1138	        inputs_embeds: torch.FloatTensor,
1139	    ):
1140	        """
1141	        Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is
1142	        equal to the length of multimodal features. If the lengths are different, an error is raised.
1143	        """
1144	        if input_ids is None:
1145	            special_audio_mask = (
1146	                inputs_embeds
1147	                == self.get_input_embeddings()(
1148	                    torch.tensor(self.config.audio_token_id, dtype=torch.long, device=inputs_embeds.device)
1149	                )


Perfect! Let me get the full class definition:


1310
1311	class Qwen3ASRForConditionalGeneration(Qwen3ASRPreTrainedModel, GenerationMixin):
1312	    config_class = Qwen3ASRConfig
1313
1314	    def __init__(self, config: Qwen3ASRConfig):
1315	        super().__init__(config)
1316	        self.config = config
1317
1318	        self.thinker = Qwen3ASRThinkerForConditionalGeneration._from_config(config.thinker_config)
1319	        self.post_init()
1320
1321	    def get_support_languages(self):
1322	        return self.config.support_languages
1323
1324	    @torch.no_grad()
1325	    def generate(
1326	        self,
1327	        input_ids: Optional[torch.Tensor] = None,
1328	        max_new_tokens: int = 4096,
1329	        eos_token_id: int | list[int] = [151645, 151643],
1330	        **kwargs,
1331	    ):
1332	        shared_kwargs = {}
1333	        thinker_kwargs = {
1334	            "max_new_tokens": max_new_tokens,
1335	            "eos_token_id": eos_token_id,
1336	        }
1337
1338	        for key, value in kwargs.items():
1339	            # Process special input values
1340	            if key == "feature_attention_mask":
1341	                thinker_kwargs[key] = value
1342	            elif key in ("input_features", "attention_mask"):
1343	                thinker_kwargs[key] = value
1344	            # Put other key to shared kwargs
1345	            else:
1346	                shared_kwargs[key] = value
1347
1348	        # Merge kwargs
1349	        for key, value in shared_kwargs.items():
1350	            if key not in thinker_kwargs:
1351	                thinker_kwargs[key] = value
1352
1353	        thinker_result = self.thinker.generate(input_ids=input_ids, return_dict_in_generate=True, **thinker_kwargs)
1354
1355	        return thinker_result
1356
1357
1358	__all__ = [
1359	    "Qwen3ASRForConditionalGeneration",
1360	    "Qwen3ASRThinkerTextModel",
1361	    "Qwen3ASRThinkerForConditionalGeneration",
1362	    "Qwen3ASRPreTrainedModel",
1363	    "Qwen3ASRPreTrainedModelForConditionalGeneration",
1364	    "Qwen3ASRThinkerTextPreTrainedModel",
1365	]
1366


1	# coding=utf-8
2	# Copyright 2026 The Alibaba Qwen team.
3	# SPDX-License-Identifier: Apache-2.0
4	#
5	# Licensed under the Apache License, Version 2.0 (the "License");
6	# you may not use this file except in compliance with the License.
7	# You may obtain a copy of the License at
8	#
9	#     http://www.apache.org/licenses/LICENSE-2.0
10	#
11	# Unless required by applicable law or agreed to in writing, software
12	# distributed under the License is distributed on an "AS IS" BASIS,
13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	# See the License for the specific language governing permissions and
15	# limitations under the License.
16	from .configuration_qwen3_asr import Qwen3ASRConfig
17	from .modeling_qwen3_asr import Qwen3ASRForConditionalGeneration
18	from .processing_qwen3_asr import Qwen3ASRProcessor


Perfect! Now let me check the example usage and the processing file:


1	# coding=utf-8
2	# Copyright 2026 The Alibaba Qwen team.
3	# SPDX-License-Identifier: Apache-2.0
4	#
5	# Licensed under the Apache License, Version 2.0 (the "License");
6	# you may not use this file except in compliance with the License.
7	# You may obtain a copy of the License at
8	#
9	#     http://www.apache.org/licenses/LICENSE-2.0
10	#
11	# Unless required by applicable law or agreed to in writing, software
12	# distributed under the License is distributed on an "AS IS" BASIS,
13	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14	# See the License for the specific language governing permissions and
15	# limitations under the License.
16	"""
17	Examples for Qwen3ASRModel (Transformers backend).
18
19	Covers:
20	  - single-sample inference (URL audio)
21	  - batch inference (mixed URL / base64 / (np.ndarray, sr))
22	  - forcing language (text-only output)
23	  - returning time_stamps (single + batch) via Qwen3ForcedAligner
24	"""
25
26	import base64
27	import io
28	import urllib.request
29	from typing import Tuple
30
31	import numpy as np
32	import soundfile as sf
33	import torch
34
35	from qwen_asr import Qwen3ASRModel
36
37
38	ASR_MODEL_PATH = "Qwen/Qwen3-ASR-1.7B"
39	FORCED_ALIGNER_PATH = "Qwen/Qwen3-ForcedAligner-0.6B"
40
41	URL_ZH = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_zh.wav"
42	URL_EN = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
43
44
45	def _download_audio_bytes(url: str, timeout: int = 30) -> bytes:
46	    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
47	    with urllib.request.urlopen(req, timeout=timeout) as resp:
48	        return resp.read()
49
50
51	def _read_wav_from_bytes(audio_bytes: bytes) -> Tuple[np.ndarray, int]:
52	    with io.BytesIO(audio_bytes) as f:
53	        wav, sr = sf.read(f, dtype="float32", always_2d=False)
54	    return np.asarray(wav, dtype=np.float32), int(sr)
55
56
57	def _to_data_url_base64(audio_bytes: bytes, mime: str = "audio/wav") -> str:
58	    b64 = base64.b64encode(audio_bytes).decode("utf-8")
59	    return f"data:{mime};base64,{b64}"
60
61
62	def _print_result(title: str, results) -> None:
63	    print(f"\n===== {title} =====")
64	    for i, r in enumerate(results):
65	        print(f"[sample {i}] language={r.language!r}")
66	        print(f"[sample {i}] text={r.text!r}")
67	        if r.time_stamps is not None and len(r.time_stamps) > 0:
68	            head = r.time_stamps[0]
69	            tail = r.time_stamps[-1]
70	            print(f"[sample {i}] ts_first: {head.text!r} {head.start_time}->{head.end_time} s")
71	            print(f"[sample {i}] ts_last : {tail.text!r} {tail.start_time}->{tail.end_time} s")
72
73
74	def test_single_url(asr: Qwen3ASRModel) -> None:
75	    results = asr.transcribe(
76	        audio=URL_ZH,
77	        language=None,
78	        return_time_stamps=False,
79	    )
80	    assert isinstance(results, list) and len(results) == 1
81	    _print_result("single-url (no forced language, no timestamps)", results)
82
83
84	def test_batch_mixed(asr: Qwen3ASRModel) -> None:
85	    zh_bytes = _download_audio_bytes(URL_ZH)
86	    en_bytes = _download_audio_bytes(URL_EN)
87
88	    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
89	    en_wav, en_sr = _read_wav_from_bytes(en_bytes)
90
91	    results = asr.transcribe(
92	        audio=[URL_ZH, zh_b64, (en_wav, en_sr)],
93	        context=["", "交易 停滞", ""],
94	        language=[None, "Chinese", "English"],
95	        return_time_stamps=False,
96	    )
97	    assert len(results) == 3
98	    _print_result("batch-mixed (forced language for some)", results)
99
100
101	def test_single_with_timestamps(asr: Qwen3ASRModel) -> None:
102	    results = asr.transcribe(
103	        audio=URL_EN,
104	        language="English",
105	        return_time_stamps=True,
106	    )
107	    assert len(results) == 1
108	    assert results[0].time_stamps is not None
109	    _print_result("single-url (forced language + timestamps)", results)
110
111
112	def test_batch_with_timestamps(asr: Qwen3ASRModel) -> None:
113	    zh_bytes = _download_audio_bytes(URL_ZH)
114	    zh_b64 = _to_data_url_base64(zh_bytes, mime="audio/wav")
115
116	    results = asr.transcribe(
117	        audio=[URL_ZH, zh_b64, URL_EN],
118	        context=["", "交易 停滞", ""],
119	        language=["Chinese", "Chinese", "English"],
120	        return_time_stamps=True,
121	    )
122	    assert len(results) == 3
123	    assert all(r.time_stamps is not None for r in results)
124	    _print_result("batch (forced language + timestamps)", results)
125
126
127	def main() -> None:
128	    asr = Qwen3ASRModel.from_pretrained(
129	        ASR_MODEL_PATH,
130	        dtype=torch.bfloat16,
131	        device_map="cuda:0",
132	        # attn_implementation="flash_attention_2",
133	        forced_aligner=FORCED_ALIGNER_PATH,
134	        forced_aligner_kwargs=dict(
135	            dtype=torch.bfloat16,
136	            device_map="cuda:0",
137	            # attn_implementation="flash_attention_2",
138	        ),
139	        max_inference_batch_size=32,
140	        max_new_tokens=256,
141	    )
142
143	    test_single_url(asr)
144	    test_batch_mixed(asr)
145	    test_single_with_timestamps(asr)
146	    test_batch_with_timestamps(asr)
147
148
149	if __name__ == "__main__":
150	    main()
151


1	# coding=utf-8
2	# Copyright 2026 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
3	#
4	# Licensed under the Apache License, Version 2.0 (the "License");
5	# you may not use this file except in compliance with the License.
6	# You may obtain a copy of the License at
7	#
8	#     http://www.apache.org/licenses/LICENSE-2.0
9	#
10	# Unless required by applicable law or agreed to in writing, software
11	# distributed under the License is distributed on an "AS IS" BASIS,
12	# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13	# See the License for the specific language governing permissions and
14	# limitations under the License.
15	import re
16
17	import numpy as np
18
19	from transformers.audio_utils import AudioInput
20	from transformers.feature_extraction_utils import BatchFeature
21	from transformers.processing_utils import ProcessingKwargs, ProcessorMixin
22	from transformers.tokenization_utils_base import TextInput
23
24
25	class Qwen3ASRProcessorKwargs(ProcessingKwargs, total=False):
26	    _defaults = {
27	        "text_kwargs": {
28	            "padding": False,
29	            "padding_side": "left",
30	        },
31	        "audio_kwargs": {
32	            "sampling_rate": 16000,
33	            "padding": True,
34	            "return_attention_mask": True,
35	        },
36	    }
37
38
39	def _get_feat_extract_output_lengths(input_lengths):
40	    """
41	    Computes the output length of the convolutional layers and the output length of the audio encoder
42	    """
43
44	    input_lengths_leave = input_lengths % 100
45	    feat_lengths = (input_lengths_leave - 1) // 2 + 1
46	    output_lengths = ((feat_lengths - 1) // 2 + 1 - 1) // 2 + 1 + (input_lengths // 100) * 13
47	    return output_lengths
48
49
50	class Qwen3ASRProcessor(ProcessorMixin):
51	    r"""
52	    Constructs a Qwen3ASR processor.
53	    [`Qwen3ASRProcessor`] offers all the functionalities of [`WhisperFeatureExtractor`], and [`Qwen2TokenizerFast`]. See the
54	    [`~Qwen3ASRProcessor.__call__`] and [`~Qwen3ASRProcessor.decode`] for more information.
55
56	    Args:
57	        feature_extractor ([`WhisperFeatureExtractor`], *optional*):
58	            The audio feature extractor.
59	        tokenizer ([`Qwen2TokenizerFast`], *optional*):
60	            The text tokenizer.
61	        chat_template (`Optional[str]`, *optional*):
62	            The Jinja template to use for formatting the conversation. If not provided, the default chat template is used.
63	    """
64
65	    attributes = ["feature_extractor", "tokenizer"]
66	    feature_extractor_class = "WhisperFeatureExtractor"
67	    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
68
69	    def __init__(
70	        self, feature_extractor=None, tokenizer=None, chat_template=None
71	    ):
72	        super().__init__(feature_extractor, tokenizer, chat_template=chat_template)
73	        self.audio_token = self.tokenizer.audio_token
74	        self.audio_bos_token = self.tokenizer.audio_bos_token
75	        self.audio_eos_token = self.tokenizer.audio_eos_token
76
77	    def __call__(
78	        self,
79	        text: TextInput = None,
80	        audio: AudioInput = None,


Excellent! Now I have a comprehensive view. Let me create a summary document for you:

## Summary: Qwen3-ASR Loading and Inference Pattern

Based on my analysis of the codebase at `/home/ubuntu/training/`, here's the complete information you requested:

### 1. **Benchmark Script Location**
`/home/ubuntu/training/benchmark_qwen3_asr.py` - Primary benchmark script for evaluating Qwen3-ASR on indic-asr-benchmark-6k dataset

### 2. **Model Loading & Inference Pattern**

The model uses **HuggingFace Transformers** with two supported backends:

#### **Transformers Backend (Default)**
```python
from qwen_asr import Qwen3ASRModel

model = Qwen3ASRModel.from_pretrained(
    args.checkpoint,
    dtype=torch.bfloat16,
    device_map=args.device,
    max_inference_batch_size=args.batch_size,
    max_new_tokens=args.max_new_tokens,
)
```

#### **vLLM Backend** (for optimized inference)
```python
model = Qwen3ASRModel.LLM(
    model=args.checkpoint,
    gpu_memory_utilization=args.gpu_memory_utilization,
    max_inference_batch_size=args.batch_size,
    max_new_tokens=args.max_new_tokens,
)
```

Both are called via the unified `Qwen3ASRModel` wrapper (at `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/qwen3_asr.py`).

### 3. **Model Class Architecture**

**Main Model Class:** `Qwen3ASRForConditionalGeneration`
- Located: `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/core/transformers_backend/modeling_qwen3_asr.py` (line 1311)
- Inherits from: `Qwen3ASRPreTrainedModel` + `GenerationMixin`
- Wraps: `Qwen3ASRThinkerForConditionalGeneration`

**Internal Architecture:**
- `Qwen3ASRThinkerForConditionalGeneration` (the actual model, line 1067)
  - `audio_tower`: `Qwen3ASRAudioEncoder` - encodes mel-spectrogram features to embeddings
  - `model`: `Qwen3ASRThinkerTextModel` - LLM decoder
  - `lm_head`: Linear layer for token generation

### 4. **Checkpoint Loading - HuggingFace Format**

Yes, checkpoints are **standard HuggingFace transformers format** with:
- `config.json` - Model configuration (defined in `Qwen3ASRConfig`)
- `pytorch_model.bin` (or `.safetensors`) - Model weights
- `tokenizer_config.json` + `tokenizer.json` - Qwen2 tokenizer files
- `preprocessor_config.json` - Audio feature extractor config (WhisperFeatureExtractor)

**Loading chain:**
```python
# From benchmark_qwen3_asr.py line 298-304
model = AutoModel.from_pretrained(args.checkpoint, **kwargs)
processor = AutoProcessor.from_pretrained(args.checkpoint, fix_mistral_regex=True)
```

**Auto-registration** (lines 28-30 in qwen3_asr.py):
```python
AutoConfig.register("qwen3_asr", Qwen3ASRConfig)
AutoModel.register(Qwen3ASRConfig, Qwen3ASRForConditionalGeneration)
AutoProcessor.register(Qwen3ASRConfig, Qwen3ASRProcessor)
```

### 5. **Inference Workflow**

**Key Method:** `Qwen3ASRModel.transcribe()`

**Audio Input Support:**
- File paths, URLs, base64 data URLs, numpy arrays `(wav, sr)`, or lists of above
- Sample rate: 16kHz (mono PCM float32)
- Automatic chunking for long audio (MAX_ASR_INPUT_SECONDS = 300s)

**Transcription Call** (benchmark_qwen3_asr.py, lines 95-99):
```python
transcriptions = model.transcribe(
    audio=audio_inputs,      # List of (np.ndarray, sr) tuples
    language=forced_langs,   # Optional: ["Hindi", "English", ...]
    return_time_stamps=False,
)
```

**Output:** List of `ASRTranscription` objects with:
- `language`: Detected/forced language name
- `text`: Transcribed text
- `time_stamps`: Optional forced alignment results

### 6. **Processor Details**

**Class:** `Qwen3ASRProcessor` (processing_qwen3_asr.py)
- Combines:
  - `WhisperFeatureExtractor` - converts audio → mel-spectrogram
  - `Qwen2TokenizerFast` - text tokenization
  - Chat template application

**Usage in inference:**
```python
inputs = processor(
    text=prompts,        # Chat template formatted
    audio=wav_arrays,    # 16kHz float32 mono
    return_tensors="pt",
    padding=True
)
```

### 7. **Key Files Overview**

| File | Purpose |
|------|---------|
| `/home/ubuntu/training/benchmark_qwen3_asr.py` | Main benchmark/evaluation script |
| `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/qwen3_asr.py` | Unified inference API (transformers + vLLM) |
| `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/core/transformers_backend/modeling_qwen3_asr.py` | HF model class definitions |
| `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/core/transformers_backend/configuration_qwen3_asr.py` | Config schema |
| `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/core/transformers_backend/processing_qwen3_asr.py` | Processor implementation |
| `/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/examples/example_qwen3_asr_transformers.py` | Complete usage examples |

This is a clean HuggingFace transformers-based ASR model (NOT NeMo), with dual backend support for inference optimization.