Search the codebase at /home/ubuntu/training/ for any existing normalization utilities used in ASR benchmarking. Specifically look for:
1. Any text normalization functions (Unicode NFC/NFKC, punctuation removal, case folding, number canonicalization)
2. Files in benchmark_outputs/ or nearby that contain benchmark scripts
3. Any existing benchmark/eval scripts that compute WER with jiwer

Check directories: benchmark_outputs/, any scripts/ dirs, and search for files containing "jiwer" or "normalize" or "wer_norm". Be quick - just find the key files/functions.


I'll search the codebase for existing normalization utilities and benchmark/eval scripts. Let me start with a broad search across the repository.


Bash

find /home/ubuntu/training -type f -name "*.py" | head -20


Now let me search for files containing "jiwer", "normalize", or "wer_norm" and check for benchmark directories.


Found 32 files
api_results/recompute_ours.py
api_results/compute_metrics.py
convert_to_schema_v1.py
generate_schema_outputs.py
benchmark_gemma3n_asr.py
benchmark_qwen3_asr.py
qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/utils.py
gemma3n-E2B-asr-ft/vendor/LlamaFactory/scripts/convert_ckpt/llamafy_baichuan2.py
qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/qwen3_forced_aligner.py
qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/qwen3_asr.py
qwen3-asr-1.7b-phase2-sft/qwen_asr/cli/demo.py
maya-tts-llama32-1b-pretraining/tts/training/rlhf/reward_utils.py
maya-tts-llama32-1b-pretraining/tts/training/rlhf/rewards.py
maya-tts-llama32-1b-pretraining/tts/training/rlhf/rlhf_main.py
maya-tts-llama32-1b-pretraining/tts/data/tts_datasets.py
maya-tts-llama32-1b-pretraining/tts/inference/quality_validation.py
maya-tts-llama32-1b-pretraining/tts/training/codec/codec_datasets.py
maya-tts-llama32-1b-pretraining/tts/training/main.py
maya-tts-llama32-1b-pretraining/tts/data/datasets/finetuning.py
maya-tts-llama32-1b-pretraining/tts/data/datasets/pretraining.py
maya-tts-llama32-1b-pretraining/tts/data/datasets/rlhf.py
maya-tts-llama32-1b-pretraining/tts/data/pretraining_contract.py
maya-tts-llama32-1b-pretraining/tts/data/text_corpus.py
maya-tts-llama32-1b-pretraining/tts/data/text_normalization.py
maya-tts-llama32-1b-pretraining/tools/data/process_raw_text_to_jsonl.py
maya-tts-llama32-1b-pretraining/tests/unit/test_tts_datasets.py
maya-tts-llama32-1b-pretraining/tools/analysis/run_inference_analysis.py
maya-tts-llama32-1b-pretraining/tools/analysis/run_pre_sft_gate.py
maya-tts-llama32-1b-pretraining/tools/data/build_text_corpus.py
maya-tts-llama32-1b-pretraining/tests/unit/test_pretraining_datasets.py
maya-tts-llama32-1b-pretraining/tests/unit/test_text_corpus.py
maya-tts-llama32-1b-pretraining/llamafactory_experiment/scripts/maya_lf_train.py


Great! I found several key benchmark files. Let me read the main benchmark scripts and the text normalization file.


     1→#!/usr/bin/env python3
     2→"""
     3→Benchmark Gemma3n-E2B-ASR checkpoints on indic-asr-benchmark-6k dataset.
     4→
     5→Usage:
     6→    python benchmark_gemma3n_asr.py \
     7→        --checkpoint /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-10000 \
     8→        --checkpoint-name ckpt-10000 \
     9→        --output-dir /home/ubuntu/training/benchmark_results_gemma3n \
    10→        --batch-size 1
    11→
    12→Metrics computed per language and overall:
    13→    - WER  (Word Error Rate)
    14→    - CER  (Character Error Rate)
    15→    - WER-norm (WER after unicode/punctuation/space normalization)
    16→    - CER-norm (CER after normalization)
    17→"""
    18→
    19→import argparse
    20→import json
    21→import os
    22→import re
    23→import sys
    24→import time
    25→import unicodedata
    26→from collections import defaultdict
    27→from pathlib import Path
    28→
    29→import numpy as np
    30→import torch
    31→import torchaudio
    32→from datasets import load_dataset as hf_load_dataset
    33→from jiwer import cer, wer
    34→from transformers import AutoModelForImageTextToText, AutoProcessor
    35→
    36→# Language code → display name mapping
    37→LANG_CODE_TO_NAME = {
    38→    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    39→    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    40→    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
    41→}
    42→
    43→
    44→# ── Text normalization (standard ASR eval practice) ──────────────────────
    45→
    46→def normalize_text(text: str) -> str:
    47→    """Normalize text for fair WER/CER comparison.
    48→    Unicode NFC → lowercase → remove punctuation → collapse whitespace → strip.
    49→    """
    50→    text = unicodedata.normalize("NFC", text)
    51→    text = text.lower()
    52→    text = re.sub(r"[^\w\s]", "", text, flags=re.UNICODE)
    53→    text = re.sub(r"\s+", " ", text).strip()
    54→    return text
    55→
    56→
    57→def load_audio_array(sample, target_sr=16000):
    58→    """Extract audio as numpy float32 array at target sample rate."""
    59→    audio = sample["audio"]
    60→    wav = np.array(audio["array"], dtype=np.float32)
    61→    sr = audio["sampling_rate"]
    62→
    63→    if sr != target_sr:
    64→        wav_tensor = torch.from_numpy(wav).unsqueeze(0)
    65→        wav_tensor = torchaudio.functional.resample(wav_tensor, sr, target_sr)
    66→        wav = wav_tensor.squeeze(0).numpy()
    67→
    68→    return wav
    69→
    70→
    71→def run_inference(model, processor, dataset, batch_size: int = 16, max_new_tokens: int = 256):
    72→    """Run batched inference on HF dataset and return list of result dicts."""
    73→    results = []
    74→    total = len(dataset)
    75→
    76→    for start in range(0, total, batch_size):
    77→        end = min(start + batch_size, total)
    78→        batch = dataset.select(range(start, end))
    79→
    80→        # Prepare batch: audio arrays + prompts
    81→        wavs = []
    82→        prompts = []
    83→        batch_meta = []
    84→        for sample in batch:
    85→            lang_name = LANG_CODE_TO_NAME.get(sample["lang_code"], sample["language"].capitalize())
    86→            user_content = f"<audio_soft_token>\nTranscribe the audio.\nLanguage: {lang_name}"
    87→            messages = [{"role": "user", "content": user_content}]
    88→
    89→            wavs.append(load_audio_array(sample))
    90→            prompts.append(processor.apply_chat_template(
    91→                messages, add_generation_prompt=True, tokenize=False
    92→            ))
    93→            batch_meta.append({
    94→                "id": sample["id"],
    95→                "language": sample["language"],
    96→                "lang_code": sample["lang_code"],
    97→                "source": sample["source"],
    98→                "duration": sample["duration"],
    99→                "reference": sample["reference"],
   100→            })
   101→
   102→        try:
   103→            # Process batch
   104→            inputs = processor(
   105→                text=prompts, audio=wavs, sampling_rate=16000,
   106→                return_tensors="pt", padding=True,
   107→            )
   108→            inputs = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v
   109→                      for k, v in inputs.items()}
   110→            input_len = inputs["input_ids"].shape[1]
   111→
   112→            # Generate
   113→            with torch.no_grad():
   114→                output_ids = model.generate(
   115→                    **inputs, max_new_tokens=max_new_tokens, do_sample=False,
   116→                )
   117→
   118→            # Decode each sample in the batch
   119→            for i in range(len(batch_meta)):
   120→                gen_ids = output_ids[i][input_len:]
   121→                hypothesis = processor.tokenizer.decode(gen_ids, skip_special_tokens=True).strip()
   122→                results.append({**batch_meta[i], "hypothesis": hypothesis})
   123→
   124→        except Exception as e:
   125→            print(f"  Batch {start}-{end} FAILED ({e}), falling back to sequential...", flush=True)
   126→            for i, sample in enumerate(batch):
   127→                try:
   128→                    inp = processor(
   129→                        text=[prompts[i]], audio=[wavs[i]], sampling_rate=16000,
   130→                        return_tensors="pt",
   131→                    )
   132→                    inp = {k: v.to(model.device) if isinstance(v, torch.Tensor) else v
   133→                           for k, v in inp.items()}
   134→                    with torch.no_grad():
   135→                        out = model.generate(**inp, max_new_tokens=max_new_tokens, do_sample=False)
   136→                    gen_ids = out[0][inp["input_ids"].shape[1]:]
   137→                    hypothesis = processor.tokenizer.decode(gen_ids, skip_special_tokens=True).strip()
   138→                except Exception as e2:
   139→                    print(f"  Sample {start+i} ({batch_meta[i]['id']}) FAILED: {e2}")
   140→                    hypothesis = ""
   141→                results.append({**batch_meta[i], "hypothesis": hypothesis})
   142→
   143→        elapsed_pct = end / total * 100
   144→        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)
   145→
   146→    return results
   147→
   148→
   149→def compute_metrics(results: list) -> dict:
   150→    """Compute WER, CER, normalized-WER/CER per language and overall."""
   151→    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})
   152→
   153→    for r in results:
   154→        lang = r["language"]
   155→        ref = r["reference"]
   156→        hyp = r.get("hypothesis", "")
   157→
   158→        ref_norm = normalize_text(ref)
   159→        hyp_norm = normalize_text(hyp)
   160→
   161→        if not ref_norm:
   162→            continue
   163→
   164→        by_lang[lang]["refs"].append(ref)
   165→        by_lang[lang]["hyps"].append(hyp)
   166→        by_lang[lang]["refs_norm"].append(ref_norm)
   167→        by_lang[lang]["hyps_norm"].append(hyp_norm)
   168→
   169→    metrics = {}
   170→    all_refs, all_hyps = [], []
   171→    all_refs_norm, all_hyps_norm = [], []
   172→
   173→    for lang in sorted(by_lang.keys()):
   174→        d = by_lang[lang]
   175→        n = len(d["refs"])
   176→        lang_wer = wer(d["refs"], d["hyps"])
   177→        lang_cer = cer(d["refs"], d["hyps"])
   178→        lang_wer_norm = wer(d["refs_norm"], d["hyps_norm"])
   179→        lang_cer_norm = cer(d["refs_norm"], d["hyps_norm"])
   180→
   181→        empty_count = sum(1 for h in d["hyps"] if not h.strip())
   182→
   183→        metrics[lang] = {
   184→            "n_samples": n,
   185→            "wer": round(lang_wer * 100, 2),
   186→            "cer": round(lang_cer * 100, 2),
   187→            "wer_normalized": round(lang_wer_norm * 100, 2),
   188→            "cer_normalized": round(lang_cer_norm * 100, 2),
   189→            "empty_hypotheses": empty_count,
   190→        }
   191→
   192→        all_refs.extend(d["refs"])
   193→        all_hyps.extend(d["hyps"])
   194→        all_refs_norm.extend(d["refs_norm"])
   195→        all_hyps_norm.extend(d["hyps_norm"])
   196→
   197→    # Overall (corpus-level)
   198→    metrics["__overall__"] = {
   199→        "n_samples": len(all_refs),
   200→        "wer": round(wer(all_refs, all_hyps) * 100, 2),
   201→        "cer": round(cer(all_refs, all_hyps) * 100, 2),
   202→        "wer_normalized": round(wer(all_refs_norm, all_hyps_norm) * 100, 2),
   203→        "cer_normalized": round(cer(all_refs_norm, all_hyps_norm) * 100, 2),
   204→    }
   205→
   206→    # Macro average across languages
   207→    lang_keys = [k for k in metrics if not k.startswith("__")]
   208→    metrics["__macro_avg__"] = {
   209→        "n_languages": len(lang_keys),
   210→        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
   211→        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
   212→        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
   213→        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
   214→    }
   215→
   216→    return metrics
   217→
   218→
   219→def print_metrics_table(metrics: dict, checkpoint_name: str):
   220→    """Pretty print metrics as a table."""
   221→    print(f"\n{'='*85}")
   222→    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
   223→    print(f"{'='*85}")
   224→    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
   225→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   226→
   227→    for lang in sorted(k for k in metrics if not k.startswith("__")):
   228→        m = metrics[lang]
   229→        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
   230→              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")
   231→
   232→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   233→
   234→    ov = metrics["__overall__"]
   235→    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
   236→          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")
   237→
   238→    ma = metrics["__macro_avg__"]
   239→    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
   240→          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
   241→    print(f"{'='*85}\n")
   242→
   243→
   244→def save_sample_analysis(results: list, output_dir: Path, n_per_lang: int = 5):
   245→    """Save a few sample predictions per language for manual inspection."""
   246→    by_lang = defaultdict(list)
   247→    for r in results:
   248→        by_lang[r["language"]].append(r)
   249→
   250→    samples = []
   251→    for lang in sorted(by_lang.keys()):
   252→        lang_results = by_lang[lang]
   253→        for r in lang_results[:n_per_lang]:
   254→            samples.append({
   255→                "id": r["id"],
   256→                "language": r["language"],
   257→                "reference": r["reference"],
   258→                "hypothesis": r["hypothesis"],
   259→                "ref_normalized": normalize_text(r["reference"]),
   260→                "hyp_normalized": normalize_text(r["hypothesis"]),
   261→            })
   262→
   263→    path = output_dir / "sample_analysis.json"
   264→    with open(path, "w", encoding="utf-8") as f:
   265→        json.dump(samples, f, ensure_ascii=False, indent=2)
   266→    print(f"Sample analysis saved: {path}")
   267→
   268→
   269→def main():
   270→    parser = argparse.ArgumentParser(description="Benchmark Gemma3n-E2B-ASR on Indic ASR dataset")
   271→    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
   272→    parser.add_argument("--checkpoint-name", default=None,
   273→                        help="Name for this checkpoint in output files (default: dirname)")
   274→    parser.add_argument("--hf-dataset", default="BayAreaBoys/indic-asr-benchmark-6k",
   275→                        help="HuggingFace dataset name")
   276→    parser.add_argument("--hf-token", default=None, help="HuggingFace token (or set HF_TOKEN env)")
   277→    parser.add_argument("--cache-dir", default="/home/ubuntu/training/datasets/indic-asr-benchmark-6k")
   278→    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results_gemma3n")
   279→    parser.add_argument("--batch-size", type=int, default=16,
   280→                        help="Samples per batch for offline batched inference")
   281→    parser.add_argument("--max-new-tokens", type=int, default=256)
   282→    parser.add_argument("--device", default="cuda:0")
   283→    parser.add_argument("--languages", nargs="+", default=None,
   284→                        help="Filter to specific language codes (e.g. hi en ta)")
   285→    args = parser.parse_args()
   286→
   287→    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
   288→    output_dir = Path(args.output_dir) / ckpt_name
   289→    output_dir.mkdir(parents=True, exist_ok=True)
   290→
   291→    # Load model and processor
   292→    print(f"Loading checkpoint: {args.checkpoint}")
   293→    t0 = time.time()
   294→
   295→    # Load tokenizer first, patch missing audio_token_id (version mismatch workaround)
   296→    from transformers import AutoTokenizer, AutoConfig
   297→    config = AutoConfig.from_pretrained(args.checkpoint, trust_remote_code=True)
   298→    tokenizer = AutoTokenizer.from_pretrained(args.checkpoint, trust_remote_code=True, padding_side="left")
   299→
   300→    # Gemma3nProcessor expects special token attrs on the tokenizer (added in transformers 5.x)
   301→    # Patch them from config for compatibility with transformers 4.x
   302→    _token_patches = {
   303→        "audio_token_id": getattr(config, "audio_token_id", 262273),
   304→        "audio_token": "<audio_soft_token>",
   305→        "boa_token_id": getattr(config, "boa_token_id", 256000),
   306→        "boa_token": "<start_of_audio>",
   307→        "eoa_token_id": getattr(config, "eoa_token_id", 262272),
   308→        "eoa_token": "<end_of_audio>",
   309→        "boi_token_id": getattr(config, "boi_token_id", 255999),
   310→        "boi_token": "<start_of_image>",
   311→        "eoi_token_id": getattr(config, "eoi_token_id", 262144),
   312→        "eoi_token": "<end_of_image>",
   313→        "image_token_id": getattr(config, "image_token_id", 262145),
   314→        "image_token": "<image_soft_token>",
   315→    }
   316→    for attr, val in _token_patches.items():
   317→        if not hasattr(tokenizer, attr):
   318→            setattr(tokenizer, attr, val)
   319→
   320→    processor = AutoProcessor.from_pretrained(
   321→        args.checkpoint, trust_remote_code=True, tokenizer=tokenizer
   322→    )
   323→    model = AutoModelForImageTextToText.from_pretrained(
   324→        args.checkpoint,
   325→        torch_dtype=torch.bfloat16,
   326→        device_map=args.device,
   327→        trust_remote_code=True,
   328→        attn_implementation="sdpa",
   329→    )
   330→    model.eval()
   331→    # Enable KV cache for faster autoregressive generation (training disables it)
   332→    model.config.use_cache = True
   333→    if hasattr(model.config, "text_config"):
   334→        model.config.text_config.use_cache = True
   335→
   336→    load_time = time.time() - t0
   337→    print(f"Model loaded in {load_time:.1f}s")
   338→
   339→    # Load dataset
   340→    hf_token = args.hf_token or os.environ.get("HF_TOKEN")
   341→    print(f"Loading dataset: {args.hf_dataset}")
   342→    dataset = hf_load_dataset(
   343→        args.hf_dataset, token=hf_token, cache_dir=args.cache_dir, split="train"
   344→    )
   345→
   346→    # Filter languages if specified
   347→    if args.languages:
   348→        dataset = dataset.filter(lambda x: x["lang_code"] in args.languages)
   349→        print(f"Filtered to {len(dataset)} samples for languages: {args.languages}")
   350→
   351→    print(f"Dataset: {len(dataset)} samples")
   352→
   353→    # Run inference
   354→    print(f"\nRunning inference (batch_size={args.batch_size})...")
   355→    t0 = time.time()
   356→    results = run_inference(model, processor, dataset, args.batch_size, args.max_new_tokens)
   357→    inference_time = time.time() - t0
   358→
   359→    total_audio_sec = sum(r["duration"] for r in results)
   360→    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0
   361→    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")
   362→
   363→    # Save raw predictions
   364→    preds_file = output_dir / "predictions.json"
   365→    with open(preds_file, "w", encoding="utf-8") as f:
   366→        json.dump(results, f, ensure_ascii=False, indent=2)
   367→    print(f"Predictions saved: {preds_file}")
   368→
   369→    # Compute metrics
   370→    metrics = compute_metrics(results)
   371→    metrics["__meta__"] = {
   372→        "checkpoint": args.checkpoint,
   373→        "checkpoint_name": ckpt_name,
   374→        "model_type": "gemma3n-E2B-asr",
   375→        "dataset": args.hf_dataset,
   376→        "batch_size": args.batch_size,
   377→        "inference_time_sec": round(inference_time, 2),
   378→        "total_audio_sec": round(total_audio_sec, 2),
   379→        "rtf": round(rtf, 4),
   380→    }
   381→
   382→    metrics_file = output_dir / "metrics.json"
   383→    with open(metrics_file, "w") as f:
   384→        json.dump(metrics, f, indent=2, ensure_ascii=False)
   385→    print(f"Metrics saved: {metrics_file}")
   386→
   387→    # Print table
   388→    print_metrics_table(metrics, ckpt_name)
   389→
   390→    # Save sample analysis
   391→    save_sample_analysis(results, output_dir)
   392→
   393→
   394→if __name__ == "__main__":
   395→    main()
   396→


     1→#!/usr/bin/env python3
     2→"""
     3→Benchmark Qwen3-ASR checkpoints on indic-asr-benchmark-6k dataset.
     4→
     5→Usage:
     6→    python benchmark_qwen3_asr.py \
     7→        --checkpoint /home/ubuntu/training/checkpoints/qwen3-asr-ckpt-24000 \
     8→        --checkpoint-name ckpt-24000 \
     9→        --output-dir /home/ubuntu/training/benchmark_results \
    10→        --batch-size 8
    11→
    12→Metrics computed per language and overall:
    13→    - WER  (Word Error Rate)
    14→    - CER  (Character Error Rate)
    15→    - WER-norm (WER after unicode/punctuation/space normalization)
    16→    - CER-norm (CER after normalization)
    17→"""
    18→
    19→import argparse
    20→import json
    21→import os
    22→import re
    23→import sys
    24→import time
    25→import unicodedata
    26→from collections import defaultdict
    27→from pathlib import Path
    28→
    29→import numpy as np
    30→import torch
    31→from datasets import load_dataset as hf_load_dataset
    32→from jiwer import cer, wer
    33→
    34→# Add the repo to path for local imports
    35→sys.path.insert(0, "/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft")
    36→
    37→# Language code → Qwen3-ASR language name mapping
    38→LANG_CODE_TO_NAME = {
    39→    "as": "Assamese", "bn": "Bengali", "en": "English", "gu": "Gujarati",
    40→    "hi": "Hindi", "kn": "Kannada", "ml": "Malayalam", "mr": "Marathi",
    41→    "or": "Odia", "pa": "Punjabi", "ta": "Tamil", "te": "Telugu",
    42→}
    43→
    44→# Languages the model supports (base + Indic from Phase 2 fine-tuning)
    45→_MODEL_SUPPORTED_LANGS = {
    46→    "Chinese", "English", "Cantonese", "Arabic", "German", "French", "Spanish",
    47→    "Portuguese", "Indonesian", "Italian", "Korean", "Russian", "Thai", "Vietnamese",
    48→    "Japanese", "Turkish", "Hindi", "Malay", "Dutch", "Swedish", "Danish", "Finnish",
    49→    "Polish", "Czech", "Filipino", "Persian", "Greek", "Romanian", "Hungarian", "Macedonian",
    50→    # Indic languages (Phase 2 fine-tuning)
    51→    "Assamese", "Bengali", "Gujarati", "Kannada", "Malayalam",
    52→    "Marathi", "Odia", "Punjabi", "Tamil", "Telugu",
    53→}
    54→
    55→
    56→# ── Text normalization (standard ASR eval practice) ──────────────────────
    57→
    58→def normalize_text(text: str) -> str:
    59→    """Normalize text for fair WER/CER comparison.
    60→    Unicode NFC → lowercase → remove punctuation → collapse whitespace → strip.
    61→    """
    62→    text = unicodedata.normalize("NFC", text)
    63→    text = text.lower()
    64→    text = re.sub(r"[^\w\s]", "", text, flags=re.UNICODE)
    65→    text = re.sub(r"\s+", " ", text).strip()
    66→    return text
    67→
    68→
    69→def run_inference(model, dataset, batch_size: int, force_language: bool = True):
    70→    """Run batched inference on HF dataset and return list of result dicts."""
    71→    results = []
    72→    total = len(dataset)
    73→
    74→    for start in range(0, total, batch_size):
    75→        end = min(start + batch_size, total)
    76→        batch = dataset.select(range(start, end))
    77→
    78→        audio_inputs = []
    79→        for sample in batch:
    80→            audio = sample["audio"]
    81→            wav = np.array(audio["array"], dtype=np.float32)
    82→            sr = audio["sampling_rate"]
    83→            audio_inputs.append((wav, sr))
    84→
    85→        # Force language if supported
    86→        if force_language:
    87→            forced_langs = []
    88→            for sample in batch:
    89→                lang_name = LANG_CODE_TO_NAME.get(sample["lang_code"])
    90→                forced_langs.append(lang_name)
    91→        else:
    92→            forced_langs = None
    93→
    94→        try:
    95→            transcriptions = model.transcribe(
    96→                audio=audio_inputs,
    97→                language=forced_langs,
    98→                return_time_stamps=False,
    99→            )
   100→        except Exception as e:
   101→            print(f"  Batch {start}-{end} failed ({e}), retrying without forced language...")
   102→            try:
   103→                transcriptions = model.transcribe(
   104→                    audio=audio_inputs, language=None, return_time_stamps=False,
   105→                )
   106→            except Exception as e2:
   107→                print(f"  Batch {start}-{end} FAILED: {e2}, trying one-by-one...")
   108→                transcriptions = []
   109→                for ai in audio_inputs:
   110→                    try:
   111→                        r = model.transcribe(audio=ai, language=None, return_time_stamps=False)
   112→                        transcriptions.extend(r)
   113→                    except Exception as e3:
   114→                        from qwen_asr.inference.qwen3_asr import ASRTranscription
   115→                        transcriptions.append(ASRTranscription(language="", text=""))
   116→
   117→        for i, t in enumerate(transcriptions):
   118→            sample = batch[i]
   119→            results.append({
   120→                "id": sample["id"],
   121→                "language": sample["language"],
   122→                "lang_code": sample["lang_code"],
   123→                "source": sample["source"],
   124→                "duration": sample["duration"],
   125→                "reference": sample["reference"],
   126→                "hypothesis": t.text,
   127→                "detected_language": t.language,
   128→            })
   129→
   130→        elapsed_pct = end / total * 100
   131→        print(f"  [{end}/{total}] ({elapsed_pct:.0f}%) processed", flush=True)
   132→
   133→    return results
   134→
   135→
   136→def compute_metrics(results: list) -> dict:
   137→    """Compute WER, CER, normalized-WER/CER per language and overall."""
   138→    by_lang = defaultdict(lambda: {"refs": [], "hyps": [], "refs_norm": [], "hyps_norm": []})
   139→
   140→    for r in results:
   141→        lang = r["language"]
   142→        ref = r["reference"]
   143→        hyp = r.get("hypothesis", "")
   144→
   145→        ref_norm = normalize_text(ref)
   146→        hyp_norm = normalize_text(hyp)
   147→
   148→        if not ref_norm:
   149→            continue
   150→
   151→        by_lang[lang]["refs"].append(ref)
   152→        by_lang[lang]["hyps"].append(hyp)
   153→        by_lang[lang]["refs_norm"].append(ref_norm)
   154→        by_lang[lang]["hyps_norm"].append(hyp_norm)
   155→
   156→    metrics = {}
   157→    all_refs, all_hyps = [], []
   158→    all_refs_norm, all_hyps_norm = [], []
   159→
   160→    for lang in sorted(by_lang.keys()):
   161→        d = by_lang[lang]
   162→        n = len(d["refs"])
   163→        lang_wer = wer(d["refs"], d["hyps"])
   164→        lang_cer = cer(d["refs"], d["hyps"])
   165→        lang_wer_norm = wer(d["refs_norm"], d["hyps_norm"])
   166→        lang_cer_norm = cer(d["refs_norm"], d["hyps_norm"])
   167→
   168→        empty_count = sum(1 for h in d["hyps"] if not h.strip())
   169→
   170→        metrics[lang] = {
   171→            "n_samples": n,
   172→            "wer": round(lang_wer * 100, 2),
   173→            "cer": round(lang_cer * 100, 2),
   174→            "wer_normalized": round(lang_wer_norm * 100, 2),
   175→            "cer_normalized": round(lang_cer_norm * 100, 2),
   176→            "empty_hypotheses": empty_count,
   177→        }
   178→
   179→        all_refs.extend(d["refs"])
   180→        all_hyps.extend(d["hyps"])
   181→        all_refs_norm.extend(d["refs_norm"])
   182→        all_hyps_norm.extend(d["hyps_norm"])
   183→
   184→    # Overall (corpus-level)
   185→    metrics["__overall__"] = {
   186→        "n_samples": len(all_refs),
   187→        "wer": round(wer(all_refs, all_hyps) * 100, 2),
   188→        "cer": round(cer(all_refs, all_hyps) * 100, 2),
   189→        "wer_normalized": round(wer(all_refs_norm, all_hyps_norm) * 100, 2),
   190→        "cer_normalized": round(cer(all_refs_norm, all_hyps_norm) * 100, 2),
   191→    }
   192→
   193→    # Macro average across languages
   194→    lang_keys = [k for k in metrics if not k.startswith("__")]
   195→    metrics["__macro_avg__"] = {
   196→        "n_languages": len(lang_keys),
   197→        "wer": round(np.mean([metrics[k]["wer"] for k in lang_keys]), 2),
   198→        "cer": round(np.mean([metrics[k]["cer"] for k in lang_keys]), 2),
   199→        "wer_normalized": round(np.mean([metrics[k]["wer_normalized"] for k in lang_keys]), 2),
   200→        "cer_normalized": round(np.mean([metrics[k]["cer_normalized"] for k in lang_keys]), 2),
   201→    }
   202→
   203→    return metrics
   204→
   205→
   206→def print_metrics_table(metrics: dict, checkpoint_name: str):
   207→    """Pretty print metrics as a table."""
   208→    print(f"\n{'='*85}")
   209→    print(f"  BENCHMARK RESULTS: {checkpoint_name}")
   210→    print(f"{'='*85}")
   211→    print(f"{'Language':<14} {'N':>5} {'WER%':>8} {'CER%':>8} {'WER-n%':>8} {'CER-n%':>8}")
   212→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   213→
   214→    for lang in sorted(k for k in metrics if not k.startswith("__")):
   215→        m = metrics[lang]
   216→        print(f"{lang:<14} {m['n_samples']:>5} {m['wer']:>8.2f} {m['cer']:>8.2f} "
   217→              f"{m['wer_normalized']:>8.2f} {m['cer_normalized']:>8.2f}")
   218→
   219→    print(f"{'-'*14} {'-'*5} {'-'*8} {'-'*8} {'-'*8} {'-'*8}")
   220→
   221→    ov = metrics["__overall__"]
   222→    print(f"{'OVERALL':<14} {ov['n_samples']:>5} {ov['wer']:>8.2f} {ov['cer']:>8.2f} "
   223→          f"{ov['wer_normalized']:>8.2f} {ov['cer_normalized']:>8.2f}")
   224→
   225→    ma = metrics["__macro_avg__"]
   226→    print(f"{'MACRO-AVG':<14} {ma['n_languages']:>5} {ma['wer']:>8.2f} {ma['cer']:>8.2f} "
   227→          f"{ma['wer_normalized']:>8.2f} {ma['cer_normalized']:>8.2f}")
   228→    print(f"{'='*85}\n")
   229→
   230→
   231→def save_sample_analysis(results: list, output_dir: Path, n_per_lang: int = 5):
   232→    """Save a few sample predictions per language for manual inspection."""
   233→    by_lang = defaultdict(list)
   234→    for r in results:
   235→        by_lang[r["language"]].append(r)
   236→
   237→    samples = []
   238→    for lang in sorted(by_lang.keys()):
   239→        lang_results = by_lang[lang]
   240→        # Pick first n_per_lang samples
   241→        for r in lang_results[:n_per_lang]:
   242→            samples.append({
   243→                "id": r["id"],
   244→                "language": r["language"],
   245→                "reference": r["reference"],
   246→                "hypothesis": r["hypothesis"],
   247→                "detected_language": r.get("detected_language", ""),
   248→                "ref_normalized": normalize_text(r["reference"]),
   249→                "hyp_normalized": normalize_text(r["hypothesis"]),
   250→            })
   251→
   252→    path = output_dir / "sample_analysis.json"
   253→    with open(path, "w", encoding="utf-8") as f:
   254→        json.dump(samples, f, ensure_ascii=False, indent=2)
   255→    print(f"Sample analysis saved: {path}")
   256→
   257→
   258→def main():
   259→    parser = argparse.ArgumentParser(description="Benchmark Qwen3-ASR on Indic ASR dataset")
   260→    parser.add_argument("--checkpoint", required=True, help="Path to model checkpoint directory")
   261→    parser.add_argument("--checkpoint-name", default=None,
   262→                        help="Name for this checkpoint in output files (default: dirname)")
   263→    parser.add_argument("--hf-dataset", default="BayAreaBoys/indic-asr-benchmark-6k",
   264→                        help="HuggingFace dataset name")
   265→    parser.add_argument("--hf-token", default=None, help="HuggingFace token (or set HF_TOKEN env)")
   266→    parser.add_argument("--cache-dir", default="/home/ubuntu/training/datasets/indic-asr-benchmark-6k")
   267→    parser.add_argument("--output-dir", default="/home/ubuntu/training/benchmark_results")
   268→    parser.add_argument("--batch-size", type=int, default=8)
   269→    parser.add_argument("--max-new-tokens", type=int, default=512)
   270→    parser.add_argument("--device", default="cuda:0")
   271→    parser.add_argument("--backend", choices=["transformers", "vllm"], default="vllm",
   272→                        help="Inference backend (default: vllm)")
   273→    parser.add_argument("--gpu-memory-utilization", type=float, default=0.85,
   274→                        help="vLLM GPU memory utilization (default: 0.85)")
   275→    parser.add_argument("--no-force-language", action="store_true",
   276→                        help="Don't force language, let model auto-detect")
   277→    parser.add_argument("--languages", nargs="+", default=None,
   278→                        help="Filter to specific language codes (e.g. hi en ta)")
   279→    args = parser.parse_args()
   280→
   281→    ckpt_name = args.checkpoint_name or Path(args.checkpoint).name
   282→    output_dir = Path(args.output_dir) / ckpt_name
   283→    output_dir.mkdir(parents=True, exist_ok=True)
   284→
   285→    # Load model
   286→    from qwen_asr import Qwen3ASRModel
   287→
   288→    print(f"Loading checkpoint: {args.checkpoint} (backend={args.backend})")
   289→    t0 = time.time()
   290→    if args.backend == "vllm":
   291→        model = Qwen3ASRModel.LLM(
   292→            model=args.checkpoint,
   293→            gpu_memory_utilization=args.gpu_memory_utilization,
   294→            max_inference_batch_size=args.batch_size if args.batch_size > 0 else -1,
   295→            max_new_tokens=args.max_new_tokens,
   296→        )
   297→    else:
   298→        model = Qwen3ASRModel.from_pretrained(
   299→            args.checkpoint,
   300→            dtype=torch.bfloat16,
   301→            device_map=args.device,
   302→            max_inference_batch_size=args.batch_size,
   303→            max_new_tokens=args.max_new_tokens,
   304→        )
   305→    load_time = time.time() - t0
   306→    print(f"Model loaded in {load_time:.1f}s")
   307→
   308→    # Load dataset
   309→    hf_token = args.hf_token or os.environ.get("HF_TOKEN")
   310→    print(f"Loading dataset: {args.hf_dataset}")
   311→    dataset = hf_load_dataset(
   312→        args.hf_dataset, token=hf_token, cache_dir=args.cache_dir, split="train"
   313→    )
   314→
   315→    # Filter languages if specified
   316→    if args.languages:
   317→        dataset = dataset.filter(lambda x: x["lang_code"] in args.languages)
   318→        print(f"Filtered to {len(dataset)} samples for languages: {args.languages}")
   319→
   320→    print(f"Dataset: {len(dataset)} samples")
   321→
   322→    # Run inference
   323→    print(f"\nRunning inference (batch_size={args.batch_size}, "
   324→          f"force_language={not args.no_force_language})...")
   325→    t0 = time.time()
   326→    results = run_inference(model, dataset, args.batch_size,
   327→                            force_language=not args.no_force_language)
   328→    inference_time = time.time() - t0
   329→
   330→    total_audio_sec = sum(r["duration"] for r in results)
   331→    rtf = inference_time / total_audio_sec if total_audio_sec > 0 else 0
   332→    print(f"Inference done in {inference_time:.1f}s for {total_audio_sec:.0f}s audio (RTF={rtf:.4f})")
   333→
   334→    # Save raw predictions
   335→    preds_file = output_dir / "predictions.json"
   336→    with open(preds_file, "w", encoding="utf-8") as f:
   337→        json.dump(results, f, ensure_ascii=False, indent=2)
   338→    print(f"Predictions saved: {preds_file}")
   339→
   340→    # Compute metrics
   341→    metrics = compute_metrics(results)
   342→    metrics["__meta__"] = {
   343→        "checkpoint": args.checkpoint,
   344→        "checkpoint_name": ckpt_name,
   345→        "backend": args.backend,
   346→        "dataset": args.hf_dataset,
   347→        "batch_size": args.batch_size,
   348→        "force_language": not args.no_force_language,
   349→        "inference_time_sec": round(inference_time, 2),
   350→        "total_audio_sec": round(total_audio_sec, 2),
   351→        "rtf": round(rtf, 4),
   352→    }
   353→
   354→    metrics_file = output_dir / "metrics.json"
   355→    with open(metrics_file, "w") as f:
   356→        json.dump(metrics, f, indent=2, ensure_ascii=False)
   357→    print(f"Metrics saved: {metrics_file}")
   358→
   359→    # Print table
   360→    print_metrics_table(metrics, ckpt_name)
   361→
   362→    # Save sample analysis
   363→    save_sample_analysis(results, output_dir)
   364→
   365→
   366→if __name__ == "__main__":
   367→    main()
   368→


     1→"""Input text normalization."""
     2→
     3→import abc
     4→import logging as base_logging
     5→
     6→import lingua
     7→import unidecode
     8→from nemo_text_processing.text_normalization import normalize
     9→
    10→base_logging.getLogger("NeMo-text-processing").setLevel(base_logging.CRITICAL)
    11→
    12→# Language codes for text normalization.
    13→_ENGLISH = "en"
    14→_JAPANESE = "ja"
    15→_CHINESE = "zh"
    16→_SPANISH = "es"
    17→_FRENCH = "fr"
    18→_GERMAN = "de"
    19→
    20→
    21→class TextNormalizer(metaclass=abc.ABCMeta):
    22→    """Text normalization class for normalizers to implement."""
    23→
    24→    @abc.abstractmethod
    25→    def normalize(self, text: str) -> str:
    26→        """Normalize"""
    27→        raise NotImplementedError("|normalize| is not implemented.")
    28→
    29→    @abc.abstractmethod
    30→    def get_supported_languages(self) -> list[str]:
    31→        """Get supported languages."""
    32→        raise NotImplementedError("|get_supported_languages| is not implemented.")
    33→
    34→    @abc.abstractmethod
    35→    def normalize_with_language(self, text: str, language: str) -> str:
    36→        """Normalize text with a specific language."""
    37→        raise NotImplementedError("|normalize_with_language| is not implemented.")
    38→
    39→
    40→class NoOpTextNormalizer(TextNormalizer):
    41→    """No-op text normalizer."""
    42→
    43→    def normalize(self, text: str) -> str:
    44→        return text
    45→
    46→    def get_supported_languages(self) -> list[str]:
    47→        return []
    48→
    49→    def normalize_with_language(self, text: str, language: str) -> str:
    50→        return text
    51→
    52→
    53→class NemoTextNormalizer(TextNormalizer):
    54→    """Text normalizer for different languages using Nvidias NeMo text normalization
    55→    library."""
    56→
    57→    def __init__(self):
    58→        super().__init__()
    59→        self._supported_languages = [
    60→            _ENGLISH,
    61→            _JAPANESE,
    62→            _CHINESE,
    63→            _SPANISH,
    64→            _FRENCH,
    65→            _GERMAN,
    66→        ]
    67→        self._normalize_text = {
    68→            lang: normalize.Normalizer(input_case="cased", lang=lang)
    69→            for lang in self._supported_languages
    70→        }
    71→        self.lang_detector = None
    72→
    73→    def init_lang_detector(self):
    74→        self.lang_detector = lingua.LanguageDetectorBuilder.from_languages(
    75→            lingua.Language.KOREAN,
    76→            lingua.Language.JAPANESE,
    77→            lingua.Language.CHINESE,
    78→            lingua.Language.ENGLISH,
    79→            lingua.Language.SPANISH,
    80→            lingua.Language.FRENCH,
    81→            lingua.Language.GERMAN,
    82→        ).build()
    83→
    84→    def convert_to_ascii(self, text: str) -> str:
    85→        return unidecode.unidecode(text)
    86→
    87→    def get_supported_languages(self) -> list[str]:
    88→        return self._supported_languages
    89→
    90→    def normalize(self, text: str) -> str:
    91→        # detect language and normalize text
    92→        try:
    93→            # Only initialize the language detector if it's not already initialized
    94→            # (dynamic language detection).
    95→            if self.lang_detector is None:
    96→                self.init_lang_detector()
    97→            language = self.lang_detector.detect_language_of(text)
    98→            if language == lingua.Language.ENGLISH:
    99→                return self.normalize_with_language(text, _ENGLISH)
   100→            elif language == lingua.Language.JAPANESE:
   101→                return self.normalize_with_language(text, _JAPANESE)
   102→            elif language == lingua.Language.CHINESE:
   103→                return self.normalize_with_language(text, _CHINESE)
   104→            elif language == lingua.Language.SPANISH:
   105→                return self.normalize_with_language(text, _SPANISH)
   106→            elif language == lingua.Language.FRENCH:
   107→                return self.normalize_with_language(text, _FRENCH)
   108→            elif language == lingua.Language.GERMAN:
   109→                return self.normalize_with_language(text, _GERMAN)
   110→            else:
   111→                return text
   112→        except Exception:
   113→            return text
   114→
   115→    def normalize_with_language(self, text: str, language: str) -> str:
   116→        if language not in self._supported_languages:
   117→            return text
   118→
   119→        if language == _ENGLISH:
   120→            text = self.convert_to_ascii(text)
   121→
   122→        try:
   123→            text = self._normalize_text[language].normalize(text)
   124→        except Exception:
   125→            # return the unnormalized text if error
   126→            return text
   127→        return text
   128→
   129→
   130→def create_text_normalizer(enable_text_normalization: bool) -> TextNormalizer:
   131→    """Create text normalizer for NemoNormalizer."""
   132→    if enable_text_normalization:
   133→        return NemoTextNormalizer()
   134→    else:
   135→        return NoOpTextNormalizer()
   136→


     1→"""
     2→Compute all 4 metric tiers for API baseline models.
     3→Normalization version: v1
     4→"""
     5→import json
     6→import re
     7→import unicodedata
     8→import sys
     9→from collections import defaultdict
    10→from pathlib import Path
    11→from datetime import datetime, timezone
    12→
    13→from jiwer import wer as compute_wer, cer as compute_cer
    14→
    15→# ── Normalization pipeline ──
    16→
    17→# Zero-width characters to strip
    18→ZW_CHARS = re.compile(r'[\u200b-\u200f\u2028-\u202f\ufeff\u00ad]')
    19→
    20→# Punctuation sets
    21→# Common punctuation (ASCII + Unicode general)
    22→PUNCT_COMMON = re.compile(r'[!"#$%&\'()*+,\-./:;<=>?@\[\\\]^_`{|}~]')
    23→# Indic punctuation (dandas, etc.) — keep single danda for segmentation? No, remove for norm.
    24→PUNCT_INDIC = re.compile(r'[\u0964\u0965\u0970\u0971]')  # danda, double danda, abbreviation signs
    25→# Extended punctuation
    26→PUNCT_EXTENDED = re.compile(r'[\u2010-\u2027\u2030-\u205e\u2e00-\u2e4f]')
    27→# Quotes normalization
    28→QUOTES = re.compile(r'[\u2018\u2019\u201a\u201b\u201c\u201d\u201e\u201f\u00ab\u00bb]')
    29→
    30→# Parenthetical annotations like "(coughing)", "(two seconds pause)"
    31→ANNOTATIONS = re.compile(r'\([^)]*\)')
    32→
    33→def norm_raw(text: str) -> str:
    34→    """Step for wer_raw: NFC unicode + trim only."""
    35→    text = unicodedata.normalize('NFC', text)
    36→    text = text.strip()
    37→    return text
    38→
    39→def norm_standard(text: str) -> str:
    40→    """Steps for wer_norm: NFKC + strip ZW + whitespace + punct + case fold."""
    41→    # Step 1: NFKC
    42→    text = unicodedata.normalize('NFKC', text)
    43→    # Step 2: Strip zero-width chars
    44→    text = ZW_CHARS.sub('', text)
    45→    # Step 2.5: Remove parenthetical annotations
    46→    text = ANNOTATIONS.sub('', text)
    47→    # Step 3: Normalize whitespace
    48→    text = ' '.join(text.split())
    49→    # Step 4: Standardize quotes/dashes
    50→    text = QUOTES.sub("'", text)
    51→    text = text.replace('\u2014', '-').replace('\u2013', '-').replace('\u2012', '-')
    52→    # Step 5: Remove punctuation
    53→    text = PUNCT_COMMON.sub('', text)
    54→    text = PUNCT_INDIC.sub('', text)
    55→    text = PUNCT_EXTENDED.sub('', text)
    56→    # Step 6: Case fold (affects English only, Indic scripts have no case)
    57→    text = text.lower()
    58→    # Final whitespace cleanup
    59→    text = ' '.join(text.split()).strip()
    60→    return text
    61→
    62→def norm_numcanon(text: str) -> str:
    63→    """Steps for wer_numcanon: norm_standard + number canonicalization."""
    64→    text = norm_standard(text)
    65→    # Remove commas in numbers (e.g., "25,000" -> "25000")
    66→    text = re.sub(r'(\d),(\d)', r'\1\2', text)
    67→    # Normalize Devanagari digits to ASCII
    68→    for offset, base in [(0x0966, '0'), (0x09E6, '0'), (0x0A66, '0'),
    69→                          (0x0AE6, '0'), (0x0B66, '0'), (0x0BE6, '0'),
    70→                          (0x0C66, '0'), (0x0CE6, '0'), (0x0D66, '0')]:
    71→        for i in range(10):
    72→            text = text.replace(chr(offset + i), str(i))
    73→    return text
    74→
    75→# ── Metric computation ──
    76→
    77→def compute_metrics_for_samples(samples: list[dict]) -> dict:
    78→    """Compute all 4 metric tiers for a list of samples."""
    79→    by_lang = defaultdict(list)
    80→    for s in samples:
    81→        by_lang[s['language']].append(s)
    82→
    83→    result = {}
    84→    all_ref_raw, all_hyp_raw = [], []
    85→    all_ref_norm, all_hyp_norm = [], []
    86→    all_ref_nc, all_hyp_nc = [], []
    87→
    88→    for lang in sorted(by_lang.keys()):
    89→        lang_samples = by_lang[lang]
    90→        refs_raw, hyps_raw = [], []
    91→        refs_norm, hyps_norm = [], []
    92→        refs_nc, hyps_nc = [], []
    93→        empty_count = 0
    94→
    95→        for s in lang_samples:
    96→            ref = s['reference']
    97→            hyp = s.get('transcript', '') or ''
    98→
    99→            if not hyp.strip():
   100→                empty_count += 1
   101→
   102→            rr, hr = norm_raw(ref), norm_raw(hyp)
   103→            rn, hn = norm_standard(ref), norm_standard(hyp)
   104→            rnc, hnc = norm_numcanon(ref), norm_numcanon(hyp)
   105→
   106→            # Skip empty refs (shouldn't happen but safety)
   107→            if not rr.strip():
   108→                continue
   109→
   110→            refs_raw.append(rr); hyps_raw.append(hr if hr.strip() else '<empty>')
   111→            refs_norm.append(rn if rn else '<empty>'); hyps_norm.append(hn if hn else '<empty>')
   112→            refs_nc.append(rnc if rnc else '<empty>'); hyps_nc.append(hnc if hnc else '<empty>')
   113→
   114→        w_raw = compute_wer(refs_raw, hyps_raw) * 100
   115→        w_norm = compute_wer(refs_norm, hyps_norm) * 100
   116→        w_nc = compute_wer(refs_nc, hyps_nc) * 100
   117→        c_norm = compute_cer(refs_norm, hyps_norm) * 100
   118→
   119→        result[lang] = {
   120→            'n_samples': len(lang_samples),
   121→            'wer_raw': round(w_raw, 2),
   122→            'wer_norm': round(w_norm, 2),
   123→            'wer_numcanon': round(w_nc, 2),
   124→            'cer_norm': round(c_norm, 2),
   125→            'empty_hypotheses': empty_count,
   126→            'normalization_delta': {
   127→                'raw_to_norm': round(w_raw - w_norm, 2),
   128→                'norm_to_numcanon': round(w_norm - w_nc, 2),
   129→            }
   130→        }
   131→
   132→        all_ref_raw.extend(refs_raw); all_hyp_raw.extend(hyps_raw)
   133→        all_ref_norm.extend(refs_norm); all_hyp_norm.extend(hyps_norm)
   134→        all_ref_nc.extend(refs_nc); all_hyp_nc.extend(hyps_nc)
   135→
   136→    # Overall (micro-average)
   137→    result['__overall__'] = {
   138→        'n_samples': len(samples),
   139→        'wer_raw': round(compute_wer(all_ref_raw, all_hyp_raw) * 100, 2),
   140→        'wer_norm': round(compute_wer(all_ref_norm, all_hyp_norm) * 100, 2),
   141→        'wer_numcanon': round(compute_wer(all_ref_nc, all_hyp_nc) * 100, 2),
   142→        'cer_norm': round(compute_cer(all_ref_norm, all_hyp_norm) * 100, 2),
   143→    }
   144→
   145→    # Macro average
   146→    lang_keys = [k for k in result if not k.startswith('_')]
   147→    n_langs = len(lang_keys)
   148→    result['__macro_avg__'] = {
   149→        'n_languages': n_langs,
   150→        'wer_raw': round(sum(result[l]['wer_raw'] for l in lang_keys) / n_langs, 2),
   151→        'wer_norm': round(sum(result[l]['wer_norm'] for l in lang_keys) / n_langs, 2),
   152→        'wer_numcanon': round(sum(result[l]['wer_numcanon'] for l in lang_keys) / n_langs, 2),
   153→        'cer_norm': round(sum(result[l]['cer_norm'] for l in lang_keys) / n_langs, 2),
   154→    }
   155→
   156→    return result
   157→
   158→
   159→def build_sample_analysis(samples: list[dict]) -> list[dict]:
   160→    """Build sample_analysis.json entries."""
   161→    out = []
   162→    for s in samples:
   163→        ref = s['reference']
   164→        hyp = s.get('transcript', '') or ''
   165→        rn = norm_standard(ref)
   166→        hn = norm_standard(hyp)
   167→        rnc = norm_numcanon(ref)
   168→        hnc = norm_numcanon(hyp)
   169→
   170→        flags = []
   171→        if ref == hyp:
   172→            flags.append('exact_match')
   173→        if rn == hn:
   174→            flags.append('exact_match_norm')
   175→        if ref != hyp and rn == hn:
   176→            flags.append('punctuation_only_diff')
   177→        if not hyp.strip():
   178→            flags.append('empty_hypothesis')
   179→        # Numeric mismatch: digits present in either
   180→        if re.search(r'\d', ref) or re.search(r'\d', hyp):
   181→            if rn != hn:
   182→                flags.append('numeric_mismatch')
   183→
   184→        out.append({
   185→            'id': s['id'],
   186→            'language': s['language'],
   187→            'reference': ref,
   188→            'hypothesis': hyp,
   189→            'ref_norm': rn,
   190→            'hyp_norm': hn,
   191→            'ref_numcanon': rnc,
   192→            'hyp_numcanon': hnc,
   193→            'detected_language': s.get('detected_lang', ''),
   194→            'flags': flags,
   195→        })
   196→    return out
   197→
   198→
   199→# ── Model registry ──
   200→
   201→MODELS = {
   202→    'elevenlabs_full_6k.jsonl': {
   203→        'model_id': 'elevenlabs-scribe-v2',
   204→        'display_name': 'ElevenLabs Scribe v2',
   205→        'model_type': 'elevenlabs-scribe-v2',
   206→    },
   207→    'elevenlabs_v1_full_6k.jsonl': {
   208→        'model_id': 'elevenlabs-scribe-v1',
   209→        'display_name': 'ElevenLabs Scribe v1',
   210→        'model_type': 'elevenlabs-scribe-v1',
   211→    },
   212→    'gemini_3_flash_strict_full_6k.jsonl': {
   213→        'model_id': 'gemini-3-flash-strict',
   214→        'display_name': 'Gemini 3 Flash (Strict)',
   215→        'model_type': 'gemini-3-flash-preview',
   216→    },
   217→    'gemini_gemini_3_flash_preview_full_6k.jsonl': {
   218→        'model_id': 'gemini-3-flash-preview',
   219→        'display_name': 'Gemini 3 Flash (Preview)',
   220→        'model_type': 'gemini-3-flash-preview',
   221→    },
   222→    'gemini_flash_v2_full_6k.jsonl': {
   223→        'model_id': 'gemini-2.5-flash',
   224→        'display_name': 'Gemini 2.5 Flash',
   225→        'model_type': 'gemini-2.5-flash',
   226→    },
   227→    'gemini_gemini_3_1_pro_preview_full_6k.jsonl': {
   228→        'model_id': 'gemini-3.1-pro',
   229→        'display_name': 'Gemini 3.1 Pro',
   230→        'model_type': 'gemini-3.1-pro-preview',
   231→    },
   232→    'sarvam_full_6k.jsonl': {
   233→        'model_id': 'sarvam-saaras-v3',
   234→        'display_name': 'Sarvam Saaras v3',
   235→        'model_type': 'saaras:v3',
   236→    },
   237→}
   238→
   239→OUTPUT_BASE = Path('/home/ubuntu/training/benchmark_outputs')
   240→
   241→def process_model(filename: str, info: dict):
   242→    model_id = info['model_id']
   243→    print(f'\n{"="*60}')
   244→    print(f'Processing: {info["display_name"]} ({filename})')
   245→    print(f'{"="*60}')
   246→
   247→    with open(filename) as f:
   248→        samples = [json.loads(line) for line in f]
   249→
   250→    print(f'  Loaded {len(samples)} samples')
   251→
   252→    # Compute metrics
   253→    metrics = compute_metrics_for_samples(samples)
   254→
   255→    # Add meta
   256→    total_audio = sum(s.get('duration', 0) for s in samples)
   257→    metrics['__meta__'] = {
   258→        'checkpoint': f'api/{model_id}',
   259→        'checkpoint_name': 'baseline',
   260→        'model_id': model_id,
   261→        'model_type': info['model_type'],
   262→        'dataset': 'BayAreaBoys/indic-asr-benchmark-6k',
   263→        'batch_size': 1,
   264→        'inference_time_sec': 0,
   265→        'total_audio_sec': round(total_audio, 2),
   266→        'rtf': 0,
   267→        'timestamp': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
   268→        'normalization_version': 'v1',
   269→        'framework': 'api',
   270→    }
   271→
   272→    # Print summary
   273→    o = metrics['__overall__']
   274→    print(f'  Overall: wer_raw={o["wer_raw"]:.2f}  wer_norm={o["wer_norm"]:.2f}  wer_numcanon={o["wer_numcanon"]:.2f}  cer_norm={o["cer_norm"]:.2f}')
   275→
   276→    # Build sample analysis
   277→    sample_analysis = build_sample_analysis(samples)
   278→
   279→    # Write outputs
   280→    out_dir = OUTPUT_BASE / model_id / 'baseline'
   281→    out_dir.mkdir(parents=True, exist_ok=True)
   282→
   283→    with open(out_dir / 'metrics.json', 'w') as f:
   284→        json.dump(metrics, f, indent=2, ensure_ascii=False)
   285→
   286→    with open(out_dir / 'sample_analysis.json', 'w') as f:
   287→        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)
   288→
   289→    print(f'  Written to {out_dir}/')
   290→    return metrics
   291→
   292→
   293→if __name__ == '__main__':
   294→    results = {}
   295→    for filename, info in MODELS.items():
   296→        fpath = Path(filename)
   297→        if not fpath.exists():
   298→            fpath = Path('/home/ubuntu/training/api_results') / filename
   299→        if not fpath.exists():
   300→            print(f'SKIP: {filename} not found')
   301→            continue
   302→        results[info['model_id']] = process_model(str(fpath), info)
   303→
   304→    # Print comparison table
   305→    print(f'\n\n{"="*80}')
   306→    print('COMPARISON TABLE (all models)')
   307→    print(f'{"="*80}')
   308→    print(f'{"Model":<30} {"WER Raw":>8} {"WER Norm":>9} {"WER Num":>8} {"CER Norm":>9}')
   309→    print('-' * 70)
   310→    for mid, m in sorted(results.items(), key=lambda x: x[1]['__overall__']['wer_norm']):
   311→        o = m['__overall__']
   312→        print(f'{mid:<30} {o["wer_raw"]:>7.2f}% {o["wer_norm"]:>8.2f}% {o["wer_numcanon"]:>7.2f}% {o["cer_norm"]:>8.2f}%')
   313→


Perfect! Let me also check if there are any other related ASR utilities in the qwen3 inference utils.


     1→# coding=utf-8
     2→# Copyright 2026 The Alibaba Qwen team.
     3→# SPDX-License-Identifier: Apache-2.0
     4→#
     5→# Licensed under the Apache License, Version 2.0 (the "License");
     6→# you may not use this file except in compliance with the License.
     7→# You may obtain a copy of the License at
     8→#
     9→#     http://www.apache.org/licenses/LICENSE-2.0
    10→#
    11→# Unless required by applicable law or agreed to in writing, software
    12→# distributed under the License is distributed on an "AS IS" BASIS,
    13→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    14→# See the License for the specific language governing permissions and
    15→# limitations under the License.
    16→import base64
    17→import io
    18→import urllib.request
    19→from dataclasses import dataclass
    20→from typing import Any, Iterable, List, Optional, Tuple, Union
    21→from urllib.parse import urlparse
    22→
    23→import librosa
    24→import numpy as np
    25→import soundfile as sf
    26→
    27→AudioLike = Union[
    28→    str,                      # wav path / URL / base64
    29→    Tuple[np.ndarray, int],   # (waveform, sr)
    30→]
    31→MaybeList = Union[Any, List[Any]]
    32→
    33→SAMPLE_RATE = 16000
    34→MAX_ASR_INPUT_SECONDS = 1200
    35→MAX_FORCE_ALIGN_INPUT_SECONDS = 180
    36→MIN_ASR_INPUT_SECONDS = 0.5
    37→SUPPORTED_LANGUAGES: List[str] = [
    38→    "Chinese",
    39→    "English",
    40→    "Cantonese",
    41→    "Arabic",
    42→    "German",
    43→    "French",
    44→    "Spanish",
    45→    "Portuguese",
    46→    "Indonesian",
    47→    "Italian",
    48→    "Korean",
    49→    "Russian",
    50→    "Thai",
    51→    "Vietnamese",
    52→    "Japanese",
    53→    "Turkish",
    54→    "Hindi",
    55→    "Malay",
    56→    "Dutch",
    57→    "Swedish",
    58→    "Danish",
    59→    "Finnish",
    60→    "Polish",
    61→    "Czech",
    62→    "Filipino",
    63→    "Persian",
    64→    "Greek",
    65→    "Romanian",
    66→    "Hungarian",
    67→    "Macedonian",
    68→    # Indic languages (added via Phase 2 fine-tuning)
    69→    "Assamese",
    70→    "Bengali",
    71→    "Gujarati",
    72→    "Kannada",
    73→    "Malayalam",
    74→    "Marathi",
    75→    "Odia",
    76→    "Punjabi",
    77→    "Tamil",
    78→    "Telugu",
    79→]
    80→_ASR_TEXT_TAG = "<asr_text>"
    81→_LANG_PREFIX = "language "
    82→
    83→
    84→def normalize_language_name(language: str) -> str:
    85→    """
    86→    Normalize language name to the canonical format used by Qwen3-ASR:
    87→    first letter uppercase, the rest lowercase (e.g., 'cHINese' -> 'Chinese').
    88→
    89→    Args:
    90→        language (str): Input language name.
    91→
    92→    Returns:
    93→        str: Normalized language name.
    94→
    95→    Raises:
    96→        ValueError: If language is empty.
    97→    """
    98→    if language is None:
    99→        raise ValueError("language is None")
   100→    s = str(language).strip()
   101→    if not s:
   102→        raise ValueError("language is empty")
   103→    return s[:1].upper() + s[1:].lower()
   104→
   105→
   106→def validate_language(language: str) -> None:
   107→    """
   108→    Validate the language is supported.
   109→
   110→    Args:
   111→        language (str): Canonical language name.
   112→
   113→    Raises:
   114→        ValueError: If unsupported.
   115→    """
   116→    if language not in SUPPORTED_LANGUAGES:
   117→        raise ValueError(f"Unsupported language: {language}. Supported: {SUPPORTED_LANGUAGES}")
   118→
   119→
   120→def ensure_list(x: MaybeList) -> List[Any]:
   121→        return x if isinstance(x, list) else [x]
   122→
   123→
   124→def is_url(s: str) -> bool:
   125→    try:
   126→        u = urlparse(s)
   127→        return u.scheme in ("http", "https") and bool(u.netloc)
   128→    except Exception:
   129→        return False
   130→
   131→
   132→def is_probably_base64(s: str) -> bool:
   133→    if s.startswith("data:audio"):
   134→        return True
   135→    if ("/" not in s and "\\" not in s) and len(s) > 256:
   136→        return True
   137→    return False
   138→
   139→
   140→def decode_base64_bytes(b64: str) -> bytes:
   141→    if "," in b64 and b64.strip().startswith("data:"):
   142→        b64 = b64.split(",", 1)[1]
   143→    return base64.b64decode(b64)
   144→
   145→
   146→def load_audio_any(x: str) -> Tuple[np.ndarray, int]:
   147→    if is_url(x):
   148→        with urllib.request.urlopen(x) as resp:
   149→            audio_bytes = resp.read()
   150→        with io.BytesIO(audio_bytes) as f:
   151→            audio, sr = sf.read(f, dtype="float32", always_2d=False)
   152→    elif is_probably_base64(x):
   153→        audio_bytes = decode_base64_bytes(x)
   154→        with io.BytesIO(audio_bytes) as f:
   155→            audio, sr = sf.read(f, dtype="float32", always_2d=False)
   156→    else:
   157→        audio, sr = librosa.load(x, sr=None, mono=False)
   158→
   159→    audio = np.asarray(audio, dtype=np.float32)
   160→    sr = int(sr)
   161→    return audio, sr
   162→
   163→
   164→def to_mono(audio: np.ndarray) -> np.ndarray:
   165→    if audio.ndim == 1:
   166→        return audio
   167→    # soundfile can return shape (T, C); some pipelines use (C, T)
   168→    if audio.ndim == 2:
   169→        if audio.shape[0] <= 8 and audio.shape[1] > audio.shape[0]:
   170→            audio = audio.T
   171→        return np.mean(audio, axis=-1).astype(np.float32)
   172→    raise ValueError(f"Unsupported audio ndim={audio.ndim}")
   173→
   174→
   175→def float_range_normalize(audio: np.ndarray) -> np.ndarray:
   176→    audio = audio.astype(np.float32)
   177→    if audio.size == 0:
   178→        return audio
   179→    peak = float(np.max(np.abs(audio)))
   180→    if peak == 0.0:
   181→        return audio
   182→    # If decoded audio is int-like scaled or out-of-range, normalize conservatively.
   183→    if peak > 1.0:
   184→        audio = audio / peak
   185→    audio = np.clip(audio, -1.0, 1.0)
   186→    return audio
   187→
   188→
   189→def normalize_audio_input(a: AudioLike) -> np.ndarray:
   190→    """
   191→    Normalize one audio input to mono 16k float32 waveform in [-1, 1].
   192→
   193→    Supported inputs:
   194→        - str: local file path / https URL / base64 audio string
   195→        - (np.ndarray, sr): waveform and sampling rate
   196→
   197→    Returns:
   198→        np.ndarray:
   199→            Mono 16k float32 waveform in [-1, 1].
   200→    """
   201→    if isinstance(a, str):
   202→        audio, sr = load_audio_any(a)
   203→    elif isinstance(a, tuple) and len(a) == 2 and isinstance(a[0], np.ndarray):
   204→        audio, sr = a[0], int(a[1])
   205→    else:
   206→        raise TypeError(f"Unsupported audio input type: {type(a)}")
   207→
   208→    audio = to_mono(np.asarray(audio))
   209→    if sr != SAMPLE_RATE:
   210→        audio = librosa.resample(audio, orig_sr=sr, target_sr=SAMPLE_RATE).astype(np.float32)
   211→    audio = float_range_normalize(audio)
   212→    return audio
   213→
   214→
   215→def normalize_audios(audios: Union[AudioLike, List[AudioLike]]) -> List[np.ndarray]:
   216→    items = ensure_list(audios)
   217→    return [normalize_audio_input(a) for a in items]
   218→
   219→
   220→def chunk_list(xs: List[Any], chunk_size: int) -> Iterable[List[Any]]:
   221→    """
   222→    Yield chunks of a list.
   223→
   224→    Args:
   225→        xs (List[Any]): Input list.
   226→        chunk_size (int): Chunk size.
   227→
   228→    Yields:
   229→        List[Any]: Slices of xs.
   230→    """
   231→    if chunk_size <= 0:
   232→        yield xs
   233→        return
   234→    for i in range(0, len(xs), chunk_size):
   235→        yield xs[i : i + chunk_size]
   236→
   237→
   238→@dataclass(frozen=True)
   239→class AudioChunk:
   240→    """
   241→    One chunk cut from an original audio.
   242→
   243→    Attributes:
   244→        orig_index: Index of the original sample in the input batch.
   245→        chunk_index: Index of this chunk within the original sample.
   246→        wav: Mono float32 waveform.
   247→        sr: Sampling rate.
   248→        offset_sec: Start offset of this chunk in the original audio, in seconds.
   249→    """
   250→    orig_index: int
   251→    chunk_index: int
   252→    wav: np.ndarray
   253→    sr: int
   254→    offset_sec: float
   255→
   256→
   257→def split_audio_into_chunks(
   258→    wav: np.ndarray,
   259→    sr: int,
   260→    max_chunk_sec: float,
   261→    search_expand_sec: float = 5.0,
   262→    min_window_ms: float = 100.0,
   263→) -> List[Tuple[np.ndarray, float]]:
   264→    """
   265→    Split a long audio into chunks close to max_chunk_sec, using a low-energy boundary.
   266→
   267→    This implementation guarantees:
   268→      - Concatenating all returned chunks reproduces the original audio exactly
   269→        (total number of samples is identical, no overlaps, no gaps).
   270→
   271→    Args:
   272→        wav: Mono waveform float32.
   273→        sr: Sampling rate.
   274→        max_chunk_sec: Target max chunk duration in seconds.
   275→        search_expand_sec: Boundary search half-window in seconds.
   276→        min_window_ms: Sliding window in milliseconds for energy estimation.
   277→
   278→    Returns:
   279→        List[Tuple[np.ndarray, float]]: List of (chunk_wav, offset_sec).
   280→    """
   281→    wav = np.asarray(wav, dtype=np.float32)
   282→    if wav.ndim > 1:
   283→        wav = np.mean(wav, axis=-1).astype(np.float32)
   284→
   285→    total_len = int(wav.shape[0])
   286→    total_sec = total_len / float(sr)
   287→    if total_sec <= max_chunk_sec:
   288→        return [(wav, 0.0)]
   289→
   290→    max_len = int(max_chunk_sec * sr)
   291→    expand = int(search_expand_sec * sr)
   292→    win = max(4, int((min_window_ms / 1000.0) * sr))
   293→
   294→    chunks: List[Tuple[np.ndarray, float]] = []
   295→
   296→    start = 0
   297→    offset_sec = 0.0
   298→
   299→    while (total_len - start) > max_len:
   300→        cut = start + max_len
   301→
   302→        left = max(start, cut - expand)
   303→        right = min(total_len, cut + expand)
   304→
   305→        if right - left <= win:
   306→            boundary = cut
   307→        else:
   308→            seg = wav[left:right]
   309→            seg_abs = np.abs(seg)
   310→
   311→            window_sums = np.convolve(seg_abs, np.ones(win, dtype=np.float32), mode="valid")
   312→
   313→            min_pos = int(np.argmin(window_sums))
   314→
   315→            wstart = min_pos
   316→            wend = min_pos + win
   317→            local = seg_abs[wstart:wend]
   318→            inner = int(np.argmin(local))
   319→            boundary = left + wstart + inner
   320→
   321→        boundary = int(max(boundary, start + 1))
   322→        boundary = int(min(boundary, total_len))
   323→
   324→        chunk = wav[start:boundary]
   325→        chunks.append((chunk, offset_sec))
   326→
   327→        offset_sec += (boundary - start) / float(sr)
   328→        start = boundary
   329→
   330→    tail = wav[start:total_len]
   331→    chunks.append((tail, offset_sec))
   332→
   333→    # Pad too-short chunks to at least MIN_ASR_INPUT_SECONDS (zero-padding at tail)
   334→    min_len = int(MIN_ASR_INPUT_SECONDS * sr)
   335→    padded: List[Tuple[np.ndarray, float]] = []
   336→    for c, off in chunks:
   337→        if c.shape[0] < min_len:
   338→            pad = min_len - int(c.shape[0])
   339→            c = np.pad(c, (0, pad), mode="constant", constant_values=0.0).astype(np.float32)
   340→        padded.append((c, off))
   341→    chunks = padded
   342→
   343→    return chunks
   344→
   345→
   346→def detect_and_fix_repetitions(text, threshold=20):
   347→    def fix_char_repeats(s, thresh):
   348→        res = []
   349→        i = 0
   350→        n = len(s)
   351→        while i < n:
   352→            count = 1
   353→            while i + count < n and s[i + count] == s[i]:
   354→                count += 1
   355→
   356→            if count > thresh:
   357→                res.append(s[i])
   358→                i += count
   359→            else:
   360→                res.append(s[i:i+count])
   361→                i += count
   362→        return ''.join(res)
   363→
   364→    def fix_pattern_repeats(s, thresh, max_len=20):
   365→        n = len(s)
   366→        min_repeat_chars = thresh * 2
   367→        if n < min_repeat_chars:
   368→            return s
   369→
   370→        i = 0
   371→        result = []
   372→        while i <= n - min_repeat_chars:
   373→            found = False
   374→            for k in range(1, max_len + 1):
   375→                if i + k * thresh > n:
   376→                    break
   377→
   378→                pattern = s[i:i+k]
   379→                valid = True
   380→                for rep in range(1, thresh):
   381→                    start_idx = i + rep * k
   382→                    if s[start_idx:start_idx+k] != pattern:
   383→                        valid = False
   384→                        break
   385→
   386→                if valid:
   387→                    total_rep = thresh
   388→                    end_index = i + thresh * k
   389→                    while end_index + k <= n and s[end_index:end_index+k] == pattern:
   390→                        total_rep += 1
   391→                        end_index += k
   392→                    result.append(pattern)
   393→                    result.append(fix_pattern_repeats(s[end_index:], thresh, max_len))
   394→                    i = n
   395→                    found = True
   396→                    break
   397→
   398→            if found:
   399→                break
   400→            else:
   401→                result.append(s[i])
   402→                i += 1
   403→
   404→        if not found:
   405→            result.append(s[i:])
   406→        return ''.join(result)
   407→
   408→    text_raw = text
   409→    text = fix_char_repeats(text_raw, threshold)
   410→    text = fix_pattern_repeats(text, threshold)
   411→    return text
   412→
   413→
   414→def parse_asr_output(
   415→    raw: str,
   416→    user_language: Optional[str] = None,
   417→) -> Tuple[str, str]:
   418→    """
   419→    Parse Qwen3-ASR raw output into (language, text).
   420→
   421→    Cases:
   422→      - With tag: "language Chinese<asr_text>...."
   423→      - With newlines: "language Chinese\\n...\\n<asr_text>...."
   424→      - No tag: treat whole string as text.
   425→      - "language None<asr_text>": treat as empty audio -> ("", "")
   426→
   427→    If user_language is provided, language is forced to user_language and raw is treated as text-only
   428→    (the model is expected to output plain transcription without metadata).
   429→
   430→    Args:
   431→        raw: Raw decoded string.
   432→        user_language: Canonical language name if user forced language.
   433→
   434→    Returns:
   435→        Tuple[str, str]: (language, text)
   436→    """
   437→    if raw is None:
   438→        return "", ""
   439→    s = str(raw).strip()
   440→    if not s:
   441→        return "", ""
   442→
   443→    s = detect_and_fix_repetitions(s)
   444→
   445→    if user_language:
   446→        # user explicitly forced language => model output is treated as pure text
   447→        return user_language, s
   448→
   449→    meta_part = s
   450→    text_part = ""
   451→    has_tag = _ASR_TEXT_TAG in s
   452→    if has_tag:
   453→        meta_part, text_part = s.split(_ASR_TEXT_TAG, 1)
   454→    else:
   455→        # no tag => pure text
   456→        return "", s.strip()
   457→
   458→    meta_lower = meta_part.lower()
   459→
   460→    # empty audio heuristic
   461→    if "language none" in meta_lower:
   462→        t = text_part.strip()
   463→        if not t:
   464→            return "", ""
   465→        # if model still returned something, keep it but language unknown
   466→        return "", t
   467→
   468→    # extract "language xxx" from meta
   469→    lang = ""
   470→    for line in meta_part.splitlines():
   471→        line = line.strip()
   472→        if not line:
   473→            continue
   474→        low = line.lower()
   475→        if low.startswith(_LANG_PREFIX):
   476→            val = line[len(_LANG_PREFIX):].strip()
   477→            if val:
   478→                lang = normalize_language_name(val)
   479→            break
   480→
   481→    return lang, text_part.strip()
   482→
   483→
   484→def merge_languages(langs: List[str]) -> str:
   485→    """
   486→    Merge per-chunk languages into a compact comma-separated string,
   487→    keeping order and removing consecutive duplicates and empty entries.
   488→
   489→    Example:
   490→      ["Chinese", "English", "English"] -> "Chinese,English"
   491→
   492→    Args:
   493→        langs: List of canonical language names.
   494→
   495→    Returns:
   496→        str: Merged language string.
   497→    """
   498→    out: List[str] = []
   499→    prev = None
   500→    for x in langs:
   501→        x = (x or "").strip()
   502→        if not x:
   503→            continue
   504→        if x == prev:
   505→            continue
   506→        out.append(x)
   507→        prev = x
   508→    return ",".join(out)
   509→


Excellent! Now I have a complete picture. Let me summarize the key findings:

## Summary of Existing Normalization Utilities and Benchmark Scripts

I found comprehensive existing normalization and benchmarking infrastructure in the codebase:

### 1. **Primary Benchmark Scripts** (with WER/CER using jiwer):

**`/home/ubuntu/training/benchmark_gemma3n_asr.py`** (397 lines)
- Uses `jiwer` library for WER/CER computation
- Implements `normalize_text()` function (lines 46-54):
  - Unicode NFC normalization
  - Lowercase conversion
  - Punctuation removal (`[^\w\s]`)
  - Whitespace collapsing
- Computes 4 metrics per language: WER, CER, WER-normalized, CER-normalized
- Outputs: metrics.json, predictions.json, sample_analysis.json

**`/home/ubuntu/training/benchmark_qwen3_asr.py`** (367 lines)
- Nearly identical normalization approach to gemma3n
- Same `normalize_text()` function implementation
- Per-language + overall + macro-average metrics

### 2. **Advanced Normalization Pipeline** (compute_metrics.py):

**`/home/ubuntu/training/api_results/compute_metrics.py`** (313 lines)
- **Most comprehensive normalization** with 4 tiers:
  1. `norm_raw()`: NFC + trim only
  2. `norm_standard()`: NFKC + zero-width stripping + annotations removal + punctuation removal + case folding
  3. `norm_numcanon()`: Adds number canonicalization (removes commas, converts Devanagari digits to ASCII)
  4. Devanagari digit mapping for 9 scripts (offsets 0x0966, 0x09E6, 0x0A66, etc.)

- Uses jiwer for WER/CER computation
- Detailed regex patterns for:
  - Zero-width characters: `[\u200b-\u200f\u2028-\u202f\ufeff\u00ad]`
  - Common punctuation: ASCII + Unicode
  - Indic punctuation (dandas): `[\u0964\u0965\u0970\u0971]`
  - Extended punctuation: `[\u2010-\u2027\u2030-\u205e\u2e00-\u2e4f]`
  - Parenthetical annotations: `\([^)]*\)`

### 3. **TTS Text Normalization** (different purpose but has utilities):

**`/home/ubuntu/training/maya-tts-llama32-1b-pretraining/tts/data/text_normalization.py`** (136 lines)
- Abstract `TextNormalizer` class
- `NemoTextNormalizer`: Uses NVIDIA NeMo text processing library
- `NoOpTextNormalizer`: No-op variant
- Language-specific normalization for: en, ja, zh, es, fr, de
- Uses lingua for language detection + unidecode for ASCII conversion

### 4. **ASR Utilities** (supporting functions):

**`/home/ubuntu/training/qwen3-asr-1.7b-phase2-sft/qwen_asr/inference/utils.py`** (509 lines)
- Helper functions for ASR inference:
  - `normalize_language_name()`: Canonical language name formatting
  - `normalize_audio_input()`: Audio preprocessing (mono conversion, resampling to 16kHz, normalization to [-1, 1])
  - `parse_asr_output()`: Parse raw ASR output with language detection
  - `detect_and_fix_repetitions()`: Fix repetition artifacts (character & pattern repeats)

### 5. **Benchmark Outputs Directory**:

**`/home/ubuntu/training/benchmark_outputs/`** contains:
- Structured results for multiple models (qwen3-asr, gemma3n-e2b, elevenlabs, gemini, sarvam)
- Each with: `metrics.json`, `sample_analysis.json`, `error_analysis.json`

---

## Key Takeaways for ASR Benchmarking:

1. **Simple approach** (benchmark_*.py): NFC + lowercase + remove non-word chars + collapse whitespace
2. **Advanced approach** (api_results/compute_metrics.py): NFKC + zero-width chars + annotations + punctuation + case-folding + number canonicalization
3. **All use jiwer library** for computing WER/CER from reference/hypothesis lists
4. **Devanagari digit support** for Indic languages (9 different scripts with different Unicode offsets)
5. **Language-specific handling** available via NeMo (NVIDIA's text processing library)