● Done. Private repo pushed:

  https://github.com/Maya-AI-ML-Trainings-Cookbooks/cohere-transcribe-indic (private)

  Contents:
  - 35 files, full source code + configs + extended tokenizer artifacts
  - README with training, data, and inference instructions
  - Model weights excluded (in R2 checkpoints, too large for git) there is a benchmark data in this workspace here locally. and htere is a dashboard mechanism where you can drop the generated benchmarks and i can view it on the dashboard with scores. your task is to clone the code repo locally here, then pull this ckpt  r2:ptcheckpoints/cohere-transcribe/04-05-2026/ckpt-10000/, then run the benchmarks onn this ckpt.moving forward we will be running more benchmarks here on further ckpts but for now let's get this 10k done.


Let me explore the workspace to understand the existing benchmark and dashboard setup, then get started.


api_results
asr-benchmark-dashboard
benchmark_gemma3n_asr.py
benchmark_maya_asr_tdt.py
benchmark_outputs
benchmark_qwen3_asr.py
benchmark_qwen3_mixed_76k.sh
benchmark_qwen3_mixed.sh
benchmark_results
benchmark_results_gemma3n
benchmark_schema
benchmark_tokenizers.py
checkpoints
convert_to_schema_v1.py
datasets
generate_schema_outputs.py
generate_schema_v1.py
indic-asr-benchmark-6k
maya-asr-hybrid-fastconformer-rnnt-stage1
__pycache__
qwen3-asr-1.7b-phase2-sft
transcribe_300k.py
transcriptions_300k


Bash

ls /home/ubuntu/training/asr-benchmark-dashboard/ 2>/dev/null || echo "no dashboard dir"


AGENTS.md
CLAUDE.md
eslint.config.mjs
next.config.ts
next-env.d.ts
node_modules
package.json
package-lock.json
postcss.config.mjs
public
README.md
src
tsconfig.json


benchmark_gemma3n_asr.py
benchmark_tokenizers.py
benchmark_maya_asr_tdt.py
benchmark_qwen3_asr.py


ls /home/ubuntu/training/benchmark_outputs/ | head -20


elevenlabs-scribe-v1
elevenlabs-scribe-v2
gemini-2.5-flash
gemini-3.1-pro
gemini-3-flash-preview
gemini-3-flash-strict
gemma3n-e2b
parakeet-1.1b
parakeet-tdt-1.1b-lang
qwen3-asr
qwen3-asr-mixed
qwen3-asr-mixed-v2
sarvam-saaras-v3


ls /home/ubuntu/training/benchmark_results/ | head -20


analysis_report.md
ckpt-100000-vllm
ckpt-120000-mixed
ckpt-150000-mixed
ckpt-170000-vllm
ckpt-200000-vllm
ckpt-24000
ckpt-24000-vllm
ckpt-246000-mixed
ckpt-250000-vllm
ckpt-300000-vllm
ckpt-400000-vllm
ckpt-50000
ckpt-72000
ckpt-72000-vllm
ckpt-76000
mixed-v2-ckpt-100000
mixed-v2-ckpt-130000
mixed-v2-ckpt-150000
mixed-v2-ckpt-200000


ls /home/ubuntu/training/benchmark_schema/ | head -20


1	@AGENTS.md
2

@AGENTS.md


1	# ASR Benchmark Schema
2
3	Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.
4
5	---
6
7	## Directory Layout
8
9	```
10	/home/ubuntu/training/benchmark_outputs/
11	  <model_id>/
12	    <checkpoint_name>/
13	      metrics.json
14	      sample_analysis.json
15	      error_analysis.json
16	```
17
18	## Model Registry
19
20	| Model ID         | Display Name                    | Checkpoints              |
21	|------------------|---------------------------------|--------------------------|
22	| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
23	| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
24	| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |
25
26	## Languages
27
28	| Language   | Code | Source                  | Samples |
29	|------------|------|-------------------------|---------|
30	| assamese   | as   | IndicVoices (valid)     | 500     |
31	| bengali    | bn   | Kathbath (test_known)   | 500     |
32	| english    | en   | Svarah                  | 500     |
33	| gujarati   | gu   | Kathbath (test_known)   | 500     |
34	| hindi      | hi   | Kathbath (test_known)   | 500     |
35	| kannada    | kn   | Kathbath (test_known)   | 500     |
36	| malayalam  | ml   | Kathbath (test_known)   | 500     |
37	| marathi    | mr   | Kathbath (test_known)   | 500     |
38	| odia       | or   | Kathbath (test_known)   | 500     |
39	| punjabi    | pa   | Kathbath (test_known)   | 500     |
40	| tamil      | ta   | Kathbath (test_known)   | 500     |
41	| telugu     | te   | Kathbath (test_known)   | 500     |
42
43	---
44
45	## Metric Tiers
46
47	Six metric tiers are computed. Each builds on standard normalization with one additional step.
48
49	| Metric           | Unicode | Whitespace     | Punctuation Removal | Case Fold | Number Canon | Purpose                              |
50	|------------------|---------|----------------|---------------------|-----------|--------------|--------------------------------------|
51	| `wer_raw`        | NFC     | trim only      | No                  | No        | No           | Strict transcript fidelity           |
52	| `wer_norm`       | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**               |
53	| `wer_numcanon`   | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness                   |
54	| `space_norm_wer` | NFKC    | **removed for alignment** | Yes (lang-aware) | Yes* | No   | Word-level after space-insensitive alignment |
55	| `mer`            | NFKC    | **all removed**| Yes (lang-aware)    | Yes*      | No           | Meaningful Error Rate (char-level)   |
56	| `cer_norm`       | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic             |
57
58	*Case fold only for scripts with case (Latin). Indic scripts are unaffected.
59
60	### MER — Meaningful Error Rate (Space-Normalized WER)
61
62	MER measures whether the model got the **content** right regardless of word boundary placement. It answers: "Did the model produce the right characters in the right order, even if it split or merged words incorrectly?"
63
64	**How it works**: Remove ALL spaces from both reference and hypothesis after standard normalization, then compute WER on the resulting single-token strings. This means:
65
66	```
67	"india"     == "indi a"      ✓ (split word — not penalized)
68	"good"      == "go od"       ✓ (split word — not penalized)
69	"hello world" == "helloworld" ✓ (merged words — not penalized)
70	"நிலையான"    == "நிலை யான"    ✓ (Indic word split — not penalized)
71	"india"     != "indonesia"   ✗ (different content — still an error)
72	```
73
74	**Why this matters for Indic ASR**: Many Indic scripts have ambiguous word boundaries. Models frequently produce correct character sequences but with incorrect spacing — especially for agglutinative languages (Tamil, Malayalam, Kannada) and compound words in Hindi/Marathi. Standard WER penalizes these as multiple errors (insertion + substitution), inflating the score. MER isolates true recognition errors from tokenization/segmentation errors.
75
76	**Computation**:
77	```python
78	def space_normalize(text: str) -> str:
79	    """Remove all spaces after standard normalization."""
80	    return norm_standard(text, language).replace(" ", "")
81
82	mer = wer(space_normalize(ref), space_normalize(hyp))
83	```
84
85	Note: Since space-normalized text is a single "word", MER is effectively identical to CER on the space-stripped text. But we report it as a WER-scale metric for direct comparison with other WER tiers.
86
87	**Interpretation**:
88	- If `wer_norm` is much worse than `mer`, the model has a **word segmentation problem** — it knows the content but not the boundaries.
89	- If `wer_norm` ≈ `mer`, spacing is not a significant error source.
90	- The gap `wer_norm - mer` quantifies exactly how much WER is inflated by spacing errors.
91
92	### Normalization Pipeline (applied in order)
93
94	```
95	Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon/mer)
96	Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
97	Step 3: Normalize whitespace (collapse runs, trim)
98	Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
99	Step 5: Remove punctuation (norm/numcanon/mer only, using language-aware set)
100	Step 6: Case fold (English only)
101	Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
102	Step 8: Space removal (mer only — remove ALL spaces to produce a single character stream)
103	Step 9: Space-insensitive word alignment (space_norm_wer only — see below)
104	```
105
106	### space_norm_wer — Space-Normalized Word Error Rate
107
108	Word error rate after forgiving whitespace boundaries by doing character alignment on space-stripped text and then counting how many reference words are touched by real content edits.
109
110	**Algorithm per sample:**
111	1. Split `ref_norm` into `ref_words`. If empty, return `(0, 0)`.
112	2. Remove all spaces: `ref_nospace = ref_norm.replace(" ", "")`, same for hyp.
113	3. If `ref_nospace == hyp_nospace`: return `(0, len(ref_words))` — pure spacing difference = zero errors.
114	4. Build `char_to_word` mapping: for each character position in `ref_nospace`, record which reference word index it belongs to.
115	5. Compute Levenshtein edit distance DP over characters of `ref_nospace` vs `hyp_nospace`.
116	6. Backtrack through the DP matrix. For each edit operation:
117	   - **Match**: mark nothing.
118	   - **Substitution**: mark the reference word owning that character.
119	   - **Deletion**: mark the reference word owning that deleted character.
120	   - **Insertion**: mark the nearest reference word (previous if available, else next).
121	7. Count distinct reference words touched by any edit. Each word counts at most once.
122	8. Return `(error_words, total_words)`.
123
124	**Corpus-level aggregation**: Micro-average — sum `error_words` and `total_words` across all samples, then divide.
125
126	**Key properties:**
127	- Forgives pure spacing differences (split/merge words) — `"new york" == "newyork"` scores 0 errors.
128	- One wrong character makes the entire reference word count as wrong (coarser than MER).
129	- Stricter than MER, more lenient than wer_norm.
130	- Hierarchy: `mer < space_norm_wer < wer_norm`
131
132	```python
133	# Example:
134	# ref: "भद्रादी कोत्तागुडेम और करीमनगर"
135	# hyp: "भद्रादी कोत्ता गुड़ेम और करीम नगर"
136	# → ref_nospace vs hyp_nospace differ only by ड→ड़ (one char edit)
137	# → touches word "कोत्तागुडेम" → error_words=1, total=4 → space_norm_wer=25%
138	# → wer_norm would be much higher (split words = multiple errors)
139	# → mer would be lower (just counts the one character)
140	```
141
142	---
143
144	## Schema: `metrics.json`
145
146	```jsonc
147	{
148	  "<language_name>": {
149	    "n_samples": 500,              // int
150	    "wer_raw": 42.10,             // float: % — minimal normalization
151	    "wer_norm": 34.07,            // float: % — primary metric
152	    "wer_numcanon": 32.50,        // float: % — with number canonicalization
153	    "space_norm_wer": 26.20,     // float: % — word-level after space-insensitive char alignment
154	    "mer": 12.40,                 // float: % — Meaningful Error Rate (CER on space-stripped text)
155	    "cer_norm": 12.30,            // float: % — character-level normalized
156	    "empty_hypotheses": 0,        // int: samples where model produced no output
157	    "normalization_delta": {
158	      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm
159	      "norm_to_numcanon": -1.57,  // float: wer_norm - wer_numcanon
160	      "norm_to_space_norm": -7.87, // float: wer_norm - space_norm_wer (pure spacing inflation)
161	      "norm_to_mer": -21.67       // float: wer_norm - mer (spacing + word-granularity inflation)
162	    }
163	  },
164	  // ... one key per language (12 total) ...
165
166	  "__overall__": {
167	    "n_samples": 6000,
168	    "wer_raw": 40.75,
169	    "wer_norm": 32.07,
170	    "wer_numcanon": 30.80,
171	    "space_norm_wer": 24.10,
172	    "mer": 11.50,
173	    "cer_norm": 11.34
174	  },
175
176	  "__macro_avg__": {
177	    "n_languages": 12,
178	    "wer_raw": 40.52,
179	    "wer_norm": 31.76,
180	    "wer_numcanon": 30.20,
181	    "space_norm_wer": 23.80,
182	    "mer": 11.20,
183	    "cer_norm": 11.03
184	  },
185
186	  "__meta__": {
187	    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
188	    "checkpoint_name": "ckpt-20000",
189	    "model_id": "gemma3n-e2b",
190	    "model_type": "gemma3n-E2B-asr",
191	    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
192	    "batch_size": 128,
193	    "inference_time_sec": 723.7,
194	    "total_audio_sec": 40354.46,
195	    "rtf": 0.0179,
196	    "timestamp": "2026-03-26T14:30:00Z",
197	    "gpu": "NVIDIA A100 80GB",
198	    "framework": "transformers",
199	    "normalization_version": "v1",
200	    "jiwer_version": "3.1.0"
201	  }
202	}
203	```
204
205	### Required Fields
206
207	| Section          | Required                                                                                |
208	|------------------|--------------------------------------------------------------------------------------------------|
209	| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
210	| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`         |
211	| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`       |
212	| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |
213
214	---
215
216	## Schema: `sample_analysis.json`
217
218	Array of per-sample objects. One entry per sample (6000 total).
219
220	```jsonc
221	[
222	  {
223	    "id": "as_0000",
224	    "language": "assamese",
225	    "reference": "<ground truth>",
226	    "hypothesis": "<model output>",
227	    "ref_norm": "<after norm pipeline>",
228	    "hyp_norm": "<after norm pipeline>",
229	    "ref_numcanon": "<after numcanon pipeline>",
230	    "hyp_numcanon": "<after numcanon pipeline>",
231	    "ref_mer": "<after space removal>",
232	    "hyp_mer": "<after space removal>",
233	    "detected_language": "Assamese",
234	    "wer_raw": 45.0,
235	    "wer_norm": 30.0,
236	    "mer": 22.0,
237	    "flags": ["numeric_mismatch", "punctuation_only_diff", "spacing_error"]
238	  }
239	]
240	```
241
242	### Per-sample fields
243
244	| Field              | Type     | Required | Description                                              |
245	|--------------------|----------|----------|----------------------------------------------------------|
246	| `id`               | string   | Yes      | Unique sample ID                                         |
247	| `language`         | string   | Yes      | Language name, lowercase                                 |
248	| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
249	| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
250	| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
251	| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
252	| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
253	| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
254	| `ref_mer`          | string   | No       | Reference after space removal (single char stream)       |
255	| `hyp_mer`          | string   | No       | Hypothesis after space removal (single char stream)      |
256	| `detected_language`| string   | No       | Language detected by model                               |
257	| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
258	| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
259	| `mer`              | float    | No       | Per-sample MER — Meaningful Error Rate (%)               |
260	| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |
261
262	### Flag Vocabulary
263
264	Agents SHOULD tag samples with applicable flags:
265
266	| Flag                    | Meaning                                                      |
267	|-------------------------|--------------------------------------------------------------|
268	| `exact_match`           | reference == hypothesis (raw)                                |
269	| `exact_match_norm`      | ref_norm == hyp_norm                                         |
270	| `numeric_mismatch`      | Difference involves digits or number words                   |
271	| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
272	| `empty_hypothesis`      | Model produced no output                                     |
273	| `script_mismatch`       | Hypothesis uses a different script than reference            |
274	| `lang_confusion`        | detected_language != expected language                        |
275	| `high_wer`              | wer_norm > 80%                                               |
276	| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |
277	| `spacing_error`         | wer_norm > mer — content is correct but word boundaries wrong|
278
279	---
280
281	## Schema: `error_analysis.json`
282
283	Per-language error breakdown. One file per checkpoint.
284
285	```jsonc
286	{
287	  "<language_name>": {
288	    "top_substitutions": [
289	      {"ref": "word_a", "hyp": "word_b", "count": 42},
290	      // top 20 by count
291	    ],
292	    "top_insertions": [
293	      {"word": "um", "count": 15},
294	      // top 20
295	    ],
296	    "top_deletions": [
297	      {"word": "the", "count": 30},
298	      // top 20
299	    ],
300	    "error_buckets": {
301	      "numeric_mismatch_count": 23,
302	      "punctuation_only_count": 45,
303	      "spacing_tokenization_count": 12,
304	      "entity_mismatch_count": 8,
305	      "script_confusion_count": 0,
306	      "empty_hypothesis_count": 0
307	    },
308	    "examples": {
309	      "worst_samples": ["as_0047", "as_0231", "as_0419"],
310	      "best_samples": ["as_0001", "as_0155", "as_0302"],
311	      "numeric_mismatch_samples": ["as_0089", "as_0201"],
312	      "entity_mismatch_samples": ["as_0134"]
313	    }
314	  },
315	  // ... one key per language ...
316
317	  "__summary__": {
318	    "model_diagnosis": "recognition-limited",
319	    "primary_error_source": "recognition",
320	    "numeric_verbalization_impact": "moderate",
321	    "formatting_impact": "low",
322	    "worst_languages": ["assamese", "malayalam", "kannada"],
323	    "best_languages": ["hindi", "english", "gujarati"]
324	  }
325	}
326	```
327
328	### `__summary__.model_diagnosis` values
329
330	| Value                 | Meaning                                           |
331	|-----------------------|---------------------------------------------------|
332	| `recognition-limited` | Core ASR errors dominate, normalization helps little |
333	| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
334	| `numeric-limited`     | Number verbalization is the primary error source   |
335	| `mixed`               | No single dominant error category                  |
336
337	### Required Fields
338
339	| Section        | Required                                                                    |
340	|----------------|-----------------------------------------------------------------------------|
341	| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
342	| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |
343
344	---
345
346	## Agent Instructions
347
348	### Output directory
349
350	```
351	/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
352	```
353
354	### Files to generate per checkpoint
355
356	| File                   | Required | Content                     |
357	|------------------------|----------|-----------------------------|
358	| `metrics.json`         | Yes      | All six metric tiers        |
359	| `sample_analysis.json` | Yes      | Per-sample predictions      |
360	| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |
361
362	### Critical Rules
363
364	1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
365	2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
366	3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
367	4. All six metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`) are mandatory per language and in aggregates.
368	5. `normalization_delta` must include `raw_to_norm`, `norm_to_numcanon`, `norm_to_space_norm`, and `norm_to_mer`.
369	6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
370	7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.
371
372	### Model-Specific Notes
373
374	| Model            | model_id        | Checkpoint paths                                              |
375	|------------------|-----------------|---------------------------------------------------------------|
376	| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
377	| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
378	| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |
379
380	### Metric Roles
381
382	| Role                           | Metric           | Use for                                              |
383	|--------------------------------|------------------|------------------------------------------------------|
384	| Primary research metric        | `wer_norm`       | Model comparison, paper reporting, dashboard default  |
385	| Strict production metric       | `wer_raw`        | Transcript fidelity, formatting quality               |
386	| Numeric robustness metric      | `wer_numcanon`   | Isolating number verbalization errors                 |
387	| Space-insensitive word metric  | `space_norm_wer` | Word accuracy ignoring segmentation — spacing forgiven, content errors still word-level |
388	| Content accuracy metric        | `mer`            | CER on space-stripped text — purest content accuracy  |
389	| Script-sensitive metric        | `cer_norm`       | Indic script evaluation, character-level quality      |
390
391	---
392
393	## Dashboard Integration
394
395	### nginx endpoint (port 80 on 216.81.248.184)
396
397	```
398	GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
399	GET /api/benchmarks/<model_id>/                        → checkpoint listing
400	GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
401	GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
402	GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
403	```
404
405	### Next.js API routes (on Vercel)
406
407	```
408	GET /api/models                                → aggregated model + checkpoint list
409	GET /api/metrics?model=X&checkpoint=Y          → metrics.json
410	GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
411	GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
412	GET /api/compare                               → cross-model WER comparison
413	```
414
415	Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.
416
417	---
418
419	## Migration from Old Schema
420
421	Old schema fields map to new as follows:
422
423	| Old Field          | New Field        |
424	|--------------------|------------------|
425	| `wer`              | `wer_raw`        |
426	| `cer`              | (dropped — use `cer_norm`) |
427	| `wer_normalized`   | `wer_norm`       |
428	| `cer_normalized`   | `cer_norm`       |
429	| (did not exist)    | `wer_numcanon`   |
430	| (did not exist)    | `space_norm_wer` |
431	| (did not exist)    | `mer`            |
432	| (did not exist)    | `normalization_delta` |
433	| (did not exist)    | `error_analysis.json` |
434
435	Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.
436

# ASR Benchmark Schema

Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

---

## Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  <model_id>/
    <checkpoint_name>/
      metrics.json
      sample_analysis.json
      error_analysis.json
```

## Model Registry

| Model ID         | Display Name                    | Checkpoints              |
|------------------|---------------------------------|--------------------------|
| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

## Languages

| Language   | Code | Source                  | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Metric Tiers

Six metric tiers are computed. Each builds on standard normalization with one additional step.

| Metric           | Unicode | Whitespace     | Punctuation Removal | Case Fold | Number Canon | Purpose                              |
|------------------|---------|----------------|---------------------|-----------|--------------|--------------------------------------|
| `wer_raw`        | NFC     | trim only      | No                  | No        | No           | Strict transcript fidelity           |
| `wer_norm`       | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**               |
| `wer_numcanon`   | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness                   |
| `space_norm_wer` | NFKC    | **removed for alignment** | Yes (lang-aware) | Yes* | No   | Word-level after space-insensitive alignment |
| `mer`            | NFKC    | **all removed**| Yes (lang-aware)    | Yes*      | No           | Meaningful Error Rate (char-level)   |
| `cer_norm`       | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic             |

*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

### MER — Meaningful Error Rate (Space-Normalized WER)

MER measures whether the model got the **content** right regardless of word boundary placement. It answers: "Did the model produce the right characters in the right order, even if it split or merged words incorrectly?"

**How it works**: Remove ALL spaces from both reference and hypothesis after standard normalization, then compute WER on the resulting single-token strings. This means:

```
"india"     == "indi a"      ✓ (split word — not penalized)
"good"      == "go od"       ✓ (split word — not penalized)
"hello world" == "helloworld" ✓ (merged words — not penalized)
"நிலையான"    == "நிலை யான"    ✓ (Indic word split — not penalized)
"india"     != "indonesia"   ✗ (different content — still an error)
```

**Why this matters for Indic ASR**: Many Indic scripts have ambiguous word boundaries. Models frequently produce correct character sequences but with incorrect spacing — especially for agglutinative languages (Tamil, Malayalam, Kannada) and compound words in Hindi/Marathi. Standard WER penalizes these as multiple errors (insertion + substitution), inflating the score. MER isolates true recognition errors from tokenization/segmentation errors.

**Computation**:
```python
def space_normalize(text: str) -> str:
    """Remove all spaces after standard normalization."""
    return norm_standard(text, language).replace(" ", "")

mer = wer(space_normalize(ref), space_normalize(hyp))
```

Note: Since space-normalized text is a single "word", MER is effectively identical to CER on the space-stripped text. But we report it as a WER-scale metric for direct comparison with other WER tiers.

**Interpretation**:
- If `wer_norm` is much worse than `mer`, the model has a **word segmentation problem** — it knows the content but not the boundaries.
- If `wer_norm` ≈ `mer`, spacing is not a significant error source.
- The gap `wer_norm - mer` quantifies exactly how much WER is inflated by spacing errors.

### Normalization Pipeline (applied in order)

```
Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon/mer)
Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
Step 3: Normalize whitespace (collapse runs, trim)
Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
Step 5: Remove punctuation (norm/numcanon/mer only, using language-aware set)
Step 6: Case fold (English only)
Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
Step 8: Space removal (mer only — remove ALL spaces to produce a single character stream)
Step 9: Space-insensitive word alignment (space_norm_wer only — see below)
```

### space_norm_wer — Space-Normalized Word Error Rate

Word error rate after forgiving whitespace boundaries by doing character alignment on space-stripped text and then counting how many reference words are touched by real content edits.

**Algorithm per sample:**
1. Split `ref_norm` into `ref_words`. If empty, return `(0, 0)`.
2. Remove all spaces: `ref_nospace = ref_norm.replace(" ", "")`, same for hyp.
3. If `ref_nospace == hyp_nospace`: return `(0, len(ref_words))` — pure spacing difference = zero errors.
4. Build `char_to_word` mapping: for each character position in `ref_nospace`, record which reference word index it belongs to.
5. Compute Levenshtein edit distance DP over characters of `ref_nospace` vs `hyp_nospace`.
6. Backtrack through the DP matrix. For each edit operation:
   - **Match**: mark nothing.
   - **Substitution**: mark the reference word owning that character.
   - **Deletion**: mark the reference word owning that deleted character.
   - **Insertion**: mark the nearest reference word (previous if available, else next).
7. Count distinct reference words touched by any edit. Each word counts at most once.
8. Return `(error_words, total_words)`.

**Corpus-level aggregation**: Micro-average — sum `error_words` and `total_words` across all samples, then divide.

**Key properties:**
- Forgives pure spacing differences (split/merge words) — `"new york" == "newyork"` scores 0 errors.
- One wrong character makes the entire reference word count as wrong (coarser than MER).
- Stricter than MER, more lenient than wer_norm.
- Hierarchy: `mer < space_norm_wer < wer_norm`

```python
# Example:
# ref: "भद्रादी कोत्तागुडेम और करीमनगर"
# hyp: "भद्रादी कोत्ता गुड़ेम और करीम नगर"
# → ref_nospace vs hyp_nospace differ only by ड→ड़ (one char edit)
# → touches word "कोत्तागुडेम" → error_words=1, total=4 → space_norm_wer=25%
# → wer_norm would be much higher (split words = multiple errors)
# → mer would be lower (just counts the one character)
```

---

## Schema: `metrics.json`

```jsonc
{
  "<language_name>": {
    "n_samples": 500,              // int
    "wer_raw": 42.10,             // float: % — minimal normalization
    "wer_norm": 34.07,            // float: % — primary metric
    "wer_numcanon": 32.50,        // float: % — with number canonicalization
    "space_norm_wer": 26.20,     // float: % — word-level after space-insensitive char alignment
    "mer": 12.40,                 // float: % — Meaningful Error Rate (CER on space-stripped text)
    "cer_norm": 12.30,            // float: % — character-level normalized
    "empty_hypotheses": 0,        // int: samples where model produced no output
    "normalization_delta": {
      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm
      "norm_to_numcanon": -1.57,  // float: wer_norm - wer_numcanon
      "norm_to_space_norm": -7.87, // float: wer_norm - space_norm_wer (pure spacing inflation)
      "norm_to_mer": -21.67       // float: wer_norm - mer (spacing + word-granularity inflation)
    }
  },
  // ... one key per language (12 total) ...

  "__overall__": {
    "n_samples": 6000,
    "wer_raw": 40.75,
    "wer_norm": 32.07,
    "wer_numcanon": 30.80,
    "space_norm_wer": 24.10,
    "mer": 11.50,
    "cer_norm": 11.34
  },

  "__macro_avg__": {
    "n_languages": 12,
    "wer_raw": 40.52,
    "wer_norm": 31.76,
    "wer_numcanon": 30.20,
    "space_norm_wer": 23.80,
    "mer": 11.20,
    "cer_norm": 11.03
  },

  "__meta__": {
    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
    "checkpoint_name": "ckpt-20000",
    "model_id": "gemma3n-e2b",
    "model_type": "gemma3n-E2B-asr",
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
    "batch_size": 128,
    "inference_time_sec": 723.7,
    "total_audio_sec": 40354.46,
    "rtf": 0.0179,
    "timestamp": "2026-03-26T14:30:00Z",
    "gpu": "NVIDIA A100 80GB",
    "framework": "transformers",
    "normalization_version": "v1",
    "jiwer_version": "3.1.0"
  }
}
```

### Required Fields

| Section          | Required                                                                                |
|------------------|--------------------------------------------------------------------------------------------------|
| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`         |
| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`       |
| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

---

## Schema: `sample_analysis.json`

Array of per-sample objects. One entry per sample (6000 total).

```jsonc
[
  {
    "id": "as_0000",
    "language": "assamese",
    "reference": "<ground truth>",
    "hypothesis": "<model output>",
    "ref_norm": "<after norm pipeline>",
    "hyp_norm": "<after norm pipeline>",
    "ref_numcanon": "<after numcanon pipeline>",
    "hyp_numcanon": "<after numcanon pipeline>",
    "ref_mer": "<after space removal>",
    "hyp_mer": "<after space removal>",
    "detected_language": "Assamese",
    "wer_raw": 45.0,
    "wer_norm": 30.0,
    "mer": 22.0,
    "flags": ["numeric_mismatch", "punctuation_only_diff", "spacing_error"]
  }
]
```

### Per-sample fields

| Field              | Type     | Required | Description                                              |
|--------------------|----------|----------|----------------------------------------------------------|
| `id`               | string   | Yes      | Unique sample ID                                         |
| `language`         | string   | Yes      | Language name, lowercase                                 |
| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
| `ref_mer`          | string   | No       | Reference after space removal (single char stream)       |
| `hyp_mer`          | string   | No       | Hypothesis after space removal (single char stream)      |
| `detected_language`| string   | No       | Language detected by model                               |
| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
| `mer`              | float    | No       | Per-sample MER — Meaningful Error Rate (%)               |
| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

### Flag Vocabulary

Agents SHOULD tag samples with applicable flags:

| Flag                    | Meaning                                                      |
|-------------------------|--------------------------------------------------------------|
| `exact_match`           | reference == hypothesis (raw)                                |
| `exact_match_norm`      | ref_norm == hyp_norm                                         |
| `numeric_mismatch`      | Difference involves digits or number words                   |
| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
| `empty_hypothesis`      | Model produced no output                                     |
| `script_mismatch`       | Hypothesis uses a different script than reference            |
| `lang_confusion`        | detected_language != expected language                        |
| `high_wer`              | wer_norm > 80%                                               |
| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |
| `spacing_error`         | wer_norm > mer — content is correct but word boundaries wrong|

---

## Schema: `error_analysis.json`

Per-language error breakdown. One file per checkpoint.

```jsonc
{
  "<language_name>": {
    "top_substitutions": [
      {"ref": "word_a", "hyp": "word_b", "count": 42},
      // top 20 by count
    ],
    "top_insertions": [
      {"word": "um", "count": 15},
      // top 20
    ],
    "top_deletions": [
      {"word": "the", "count": 30},
      // top 20
    ],
    "error_buckets": {
      "numeric_mismatch_count": 23,
      "punctuation_only_count": 45,
      "spacing_tokenization_count": 12,
      "entity_mismatch_count": 8,
      "script_confusion_count": 0,
      "empty_hypothesis_count": 0
    },
    "examples": {
      "worst_samples": ["as_0047", "as_0231", "as_0419"],
      "best_samples": ["as_0001", "as_0155", "as_0302"],
      "numeric_mismatch_samples": ["as_0089", "as_0201"],
      "entity_mismatch_samples": ["as_0134"]
    }
  },
  // ... one key per language ...

  "__summary__": {
    "model_diagnosis": "recognition-limited",
    "primary_error_source": "recognition",
    "numeric_verbalization_impact": "moderate",
    "formatting_impact": "low",
    "worst_languages": ["assamese", "malayalam", "kannada"],
    "best_languages": ["hindi", "english", "gujarati"]
  }
}
```

### `__summary__.model_diagnosis` values

| Value                 | Meaning                                           |
|-----------------------|---------------------------------------------------|
| `recognition-limited` | Core ASR errors dominate, normalization helps little |
| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
| `numeric-limited`     | Number verbalization is the primary error source   |
| `mixed`               | No single dominant error category                  |

### Required Fields

| Section        | Required                                                                    |
|----------------|-----------------------------------------------------------------------------|
| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

---

## Agent Instructions

### Output directory

```
/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
```

### Files to generate per checkpoint

| File                   | Required | Content                     |
|------------------------|----------|-----------------------------|
| `metrics.json`         | Yes      | All six metric tiers        |
| `sample_analysis.json` | Yes      | Per-sample predictions      |
| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

### Critical Rules

1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
4. All six metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `space_norm_wer`, `mer`, `cer_norm`) are mandatory per language and in aggregates.
5. `normalization_delta` must include `raw_to_norm`, `norm_to_numcanon`, `norm_to_space_norm`, and `norm_to_mer`.
6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

### Model-Specific Notes

| Model            | model_id        | Checkpoint paths                                              |
|------------------|-----------------|---------------------------------------------------------------|
| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

### Metric Roles

| Role                           | Metric           | Use for                                              |
|--------------------------------|------------------|------------------------------------------------------|
| Primary research metric        | `wer_norm`       | Model comparison, paper reporting, dashboard default  |
| Strict production metric       | `wer_raw`        | Transcript fidelity, formatting quality               |
| Numeric robustness metric      | `wer_numcanon`   | Isolating number verbalization errors                 |
| Space-insensitive word metric  | `space_norm_wer` | Word accuracy ignoring segmentation — spacing forgiven, content errors still word-level |
| Content accuracy metric        | `mer`            | CER on space-stripped text — purest content accuracy  |
| Script-sensitive metric        | `cer_norm`       | Indic script evaluation, character-level quality      |

---

## Dashboard Integration

### nginx endpoint (port 80 on 216.81.248.184)

```
GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
GET /api/benchmarks/<model_id>/                        → checkpoint listing
GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
```

### Next.js API routes (on Vercel)

```
GET /api/models                                → aggregated model + checkpoint list
GET /api/metrics?model=X&checkpoint=Y          → metrics.json
GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
GET /api/compare                               → cross-model WER comparison
```

Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

---

## Migration from Old Schema

Old schema fields map to new as follows:

| Old Field          | New Field        |
|--------------------|------------------|
| `wer`              | `wer_raw`        |
| `cer`              | (dropped — use `cer_norm`) |
| `wer_normalized`   | `wer_norm`       |
| `cer_normalized`   | `cer_norm`       |
| (did not exist)    | `wer_numcanon`   |
| (did not exist)    | `space_norm_wer` |
| (did not exist)    | `mer`            |
| (did not exist)    | `normalization_delta` |
| (did not exist)    | `error_analysis.json` |

Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.


ls /home/ubuntu/training/asr-benchmark-dashboard/public/ | head -20


file.svg
globe.svg
next.svg
vercel.svg
window.svg


1	#!/usr/bin/env python3
2	"""
3	Generate benchmark_outputs conforming to BENCHMARK_SCHEMA.md v1.
4
5	Reads raw predictions from benchmark_results/ and produces:
6	  - metrics.json       (4 metric tiers + normalization_delta + meta)
7	  - sample_analysis.json  (per-sample with flags)
8	  - error_analysis.json   (top subs/ins/del + error buckets + diagnosis)
9	"""
10
11	import json
12	import re
13	import sys
14	import unicodedata
15	from collections import Counter, defaultdict
16	from datetime import datetime, timezone
17	from pathlib import Path
18
19	import numpy as np
20	from jiwer import cer, wer
21
22	# ── Constants ────────────────────────────────────────────────────────────
23
24	NORMALIZATION_VERSION = "v1"
25	GPU_NAME = "NVIDIA H200 80GB"
26	MODEL_ID = "qwen3-asr"
27	MODEL_TYPE = "Qwen3-ASR-1.7B"
28	DATASET = "BayAreaBoys/indic-asr-benchmark-6k"
29
30	LATIN_SCRIPT_LANGS = {"english"}
31
32	# Zero-width characters to strip
33	ZW_CHARS = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff\u00ad]")
34
35	# Language-aware punctuation sets
36	# Base punctuation (safe to remove for all languages)
37	BASE_PUNCT = set("!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~")
38	# Indic punctuation
39	INDIC_PUNCT = {
40	    "\u0964",  # Devanagari danda ।
41	    "\u0965",  # Devanagari double danda ॥
42	    "\u0970",  # Devanagari abbreviation sign
43	}
44	# We preserve danda for Indic but remove double-danda and others
45	INDIC_REMOVE_PUNCT = {"\u0965", "\u0970"}
46
47	# Curly quotes / special dashes to normalize before removal
48	QUOTE_MAP = {
49	    "\u2018": "'", "\u2019": "'", "\u201c": '"', "\u201d": '"',
50	    "\u2013": "-", "\u2014": "-", "\u2015": "-",
51	    "\u2026": "...",
52	}
53
54	# Number word mappings for English
55	EN_NUMBER_WORDS = {
56	    "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
57	    "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
58	    "ten": "10", "eleven": "11", "twelve": "12", "thirteen": "13",
59	    "fourteen": "14", "fifteen": "15", "sixteen": "16", "seventeen": "17",
60	    "eighteen": "18", "nineteen": "19", "twenty": "20", "thirty": "30",
61	    "forty": "40", "fifty": "50", "sixty": "60", "seventy": "70",
62	    "eighty": "80", "ninety": "90", "hundred": "100", "thousand": "1000",
63	    "lakh": "100000", "crore": "10000000",
64	}
65
66	# Hindi number words (Devanagari)
67	HI_NUMBER_WORDS = {
68	    "शून्य": "0", "एक": "1", "दो": "2", "तीन": "3", "चार": "4",
69	    "पांच": "5", "छह": "6", "सात": "7", "आठ": "8", "नौ": "9",
70	    "दस": "10", "ग्यारह": "11", "बारह": "12", "तेरह": "13",
71	    "चौदह": "14", "पंद्रह": "15", "सोलह": "16", "सत्रह": "17",
72	    "अठारह": "18", "उन्नीस": "19", "बीस": "20",
73	    "सौ": "100", "हज़ार": "1000", "हजार": "1000",
74	    "लाख": "100000", "करोड़": "10000000",
75	}
76
77
78	# ── Normalization Pipeline ───────────────────────────────────────────────
79
80	def norm_raw(text: str) -> str:
81	    """Tier 1: NFC unicode + trim only."""
82	    text = unicodedata.normalize("NFC", text)
83	    text = text.strip()
84	    return text
85
86
87	def norm_standard(text: str, language: str = "") -> str:
88	    """Tier 2 (wer_norm): NFKC + strip ZW + normalize whitespace + standardize punct + remove punct + case fold."""
89	    # Step 1: NFKC
90	    text = unicodedata.normalize("NFKC", text)
91	    # Step 2: Strip zero-width chars
92	    text = ZW_CHARS.sub("", text)
93	    # Step 3: Normalize whitespace
94	    text = re.sub(r"\s+", " ", text).strip()
95	    # Step 4: Standardize punctuation variants
96	    for old, new in QUOTE_MAP.items():
97	        text = text.replace(old, new)
98	    # Normalize double danda to single danda
99	    text = text.replace("\u0965", "\u0964")
100	    # Step 5: Remove punctuation (language-aware)
101	    # For Indic: keep danda (।) as it's segmentation-relevant... actually schema says remove
102	    # Schema: "Remove punctuation (norm/numcanon only, using language-aware set)"
103	    # Remove all base punctuation
104	    text = "".join(c for c in text if c not in BASE_PUNCT)
105	    # Remove Indic punctuation too (danda, double danda, abbreviation)
106	    text = "".join(c for c in text if c not in INDIC_PUNCT and c not in INDIC_REMOVE_PUNCT)
107	    # Step 6: Case fold (English/Latin only)
108	    if language in LATIN_SCRIPT_LANGS:
109	        text = text.lower()
110	    else:
111	        text = text.lower()  # safe: Indic scripts have no case, lowering is a no-op
112	    # Collapse whitespace again after removals
113	    text = re.sub(r"\s+", " ", text).strip()
114	    return text
115
116
117	def norm_mer(text: str, language: str = "") -> str:
118	    """Tier 4 (MER): norm_standard + remove ALL spaces."""
119	    text = norm_standard(text, language)
120	    text = text.replace(" ", "")
121	    return text
122
123
124	# ── space_norm_wer computation ───────────────────────────────────────
125
126	def space_norm_wer_sample(ref_norm: str, hyp_norm: str):
127	    """
128	    Word error rate after space-insensitive character alignment.
129	    Returns (error_words, total_words).
130	    """
131	    ref_words = ref_norm.split()
132	    if not ref_words:
133	        return (0, 0)
134
135	    ref_nospace = ref_norm.replace(" ", "")
136	    hyp_nospace = hyp_norm.replace(" ", "")
137	    total_words = len(ref_words)
138
139	    if ref_nospace == hyp_nospace:
140	        return (0, total_words)
141
142	    # char_to_word: ref_nospace[i] → word index
143	    char_to_word = []
144	    for word_idx, word in enumerate(ref_words):
145	        for _ in word:
146	            char_to_word.append(word_idx)
147
148	    n = len(ref_nospace)
149	    m = len(hyp_nospace)
150
151	    # Levenshtein DP
152	    dp = [[0] * (m + 1) for _ in range(n + 1)]
153	    for i in range(n + 1):
154	        dp[i][0] = i
155	    for j in range(m + 1):
156	        dp[0][j] = j
157	    for i in range(1, n + 1):
158	        for j in range(1, m + 1):
159	            if ref_nospace[i - 1] == hyp_nospace[j - 1]:
160	                dp[i][j] = dp[i - 1][j - 1]
161	            else:
162	                dp[i][j] = 1 + min(dp[i - 1][j - 1], dp[i - 1][j], dp[i][j - 1])
163
164	    # Backtrack and mark touched words
165	    touched = set()
166	    i, j = n, m
167	    while i > 0 or j > 0:
168	        if i > 0 and j > 0 and ref_nospace[i - 1] == hyp_nospace[j - 1] and dp[i][j] == dp[i - 1][j - 1]:
169	            i -= 1; j -= 1
170	        elif i > 0 and j > 0 and dp[i][j] == dp[i - 1][j - 1] + 1:
171	            touched.add(char_to_word[i - 1]); i -= 1; j -= 1
172	        elif i > 0 and dp[i][j] == dp[i - 1][j] + 1:
173	            touched.add(char_to_word[i - 1]); i -= 1
174	        elif j > 0 and dp[i][j] == dp[i][j - 1] + 1:
175	            if i > 0:
176	                touched.add(char_to_word[i - 1])
177	            elif i < n:
178	                touched.add(char_to_word[i])
179	            j -= 1
180	        else:
181	            break
182
183	    return (len(touched), total_words)
184
185
186	def norm_numcanon(text: str, language: str = "") -> str:
187	    """Tier 3 (wer_numcanon): norm_standard + number canonicalization."""
188	    text = norm_standard(text, language)
189	    # Remove digit grouping (commas in numbers)
190	    text = re.sub(r"(\d),(\d)", r"\1\2", text)
191	    text = re.sub(r"(\d),(\d)", r"\1\2", text)  # repeat for 1,00,000 style
192	    # For English, replace simple number words with digits
193	    if language == "english":
194	        words = text.split()
195	        out = []
196	        for w in words:
197	            if w in EN_NUMBER_WORDS:
198	                out.append(EN_NUMBER_WORDS[w])
199	            else:
200	                out.append(w)
201	        text = " ".join(out)
202	    # For Hindi
203	    if language == "hindi":
204	        words = text.split()
205	        out = []
206	        for w in words:
207	            if w in HI_NUMBER_WORDS:
208	                out.append(HI_NUMBER_WORDS[w])
209	            else:
210	                out.append(w)
211	        text = " ".join(out)
212	    return text
213
214
215	# ── Per-sample WER ───────────────────────────────────────────────────────
216
217	def sample_wer(ref: str, hyp: str) -> float:
218	    """Compute WER for a single sample. Returns 0-100 float."""
219	    if not ref.strip():
220	        return 0.0 if not hyp.strip() else 100.0
221	    if not hyp.strip():
222	        return 100.0
223	    return round(wer(ref, hyp) * 100, 2)
224
225
226	# ── Flag detection ───────────────────────────────────────────────────────
227
228	def detect_flags(ref: str, hyp: str, ref_n: str, hyp_n: str,
229	                 detected_lang: str, expected_lang: str, wer_norm_val: float) -> list:
230	    flags = []
231	    if ref == hyp:
232	        flags.append("exact_match")
233	    if ref_n == hyp_n:
234	        flags.append("exact_match_norm")
235	    if not hyp.strip():
236	        flags.append("empty_hypothesis")
237	    if ref != hyp and ref_n == hyp_n:
238	        flags.append("punctuation_only_diff")
239	    # Numeric mismatch: digits present in either but differ
240	    ref_digits = re.findall(r"\d+", ref_n)
241	    hyp_digits = re.findall(r"\d+", hyp_n)
242	    if ref_digits != hyp_digits and (ref_digits or hyp_digits):
243	        flags.append("numeric_mismatch")
244	    # Script mismatch: check if hypothesis has different unicode blocks than reference
245	    if hyp.strip() and ref.strip():
246	        ref_scripts = set(unicodedata.script(c) for c in ref if not c.isspace() and c.isalpha()) if hasattr(unicodedata, 'script') else set()
247	        # Simplified: check if any non-Latin non-Common chars in hyp are from a different block than ref
248	        # Use character ranges as proxy
249	        ref_devanagari = bool(re.search(r"[\u0900-\u097F]", ref))
250	        hyp_devanagari = bool(re.search(r"[\u0900-\u097F]", hyp))
251	        ref_bengali = bool(re.search(r"[\u0980-\u09FF]", ref))
252	        hyp_bengali = bool(re.search(r"[\u0980-\u09FF]", hyp))
253	        ref_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", ref))
254	        hyp_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", hyp))
255	        ref_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", ref))
256	        hyp_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", hyp))
257	        # If ref has one script but hyp has a different one prominently
258	        if (ref_devanagari and not hyp_devanagari and hyp_gujarati) or \
259	           (ref_bengali and not hyp_bengali and hyp_gurmukhi) or \
260	           (ref_gujarati and not hyp_gujarati and hyp_devanagari):
261	            flags.append("script_mismatch")
262	    # Language confusion
263	    if detected_lang and expected_lang:
264	        if detected_lang.lower() != expected_lang.lower():
265	            flags.append("lang_confusion")
266	    if wer_norm_val > 80:
267	        flags.append("high_wer")
268	    return flags
269
270
271	# ── Error Analysis ───────────────────────────────────────────────────────
272
273	def compute_error_analysis(samples_by_lang: dict) -> dict:
274	    """Compute top substitutions/insertions/deletions per language."""
275	    analysis = {}
276
277	    for lang, samples in sorted(samples_by_lang.items()):
278	        subs = Counter()
279	        insertions = Counter()
280	        deletions = Counter()
281	        numeric_mismatch_count = 0
282	        punct_only_count = 0
283	        spacing_count = 0
284	        entity_count = 0
285	        script_confusion_count = 0
286	        empty_count = 0
287
288	        wer_scores = []
289	        sample_ids = []
290
291	        for s in samples:
292	            ref_words = s["ref_norm"].split()
293	            hyp_words = s["hyp_norm"].split()
294	            flags = s.get("flags", [])
295
296	            if "empty_hypothesis" in flags:
297	                empty_count += 1
298	            if "numeric_mismatch" in flags:
299	                numeric_mismatch_count += 1
300	            if "punctuation_only_diff" in flags:
301	                punct_only_count += 1
302	            if "script_mismatch" in flags:
303	                script_confusion_count += 1
304
305	            wer_scores.append((s["id"], s.get("wer_norm", 100.0)))
306
307	            # Simple alignment for top errors (using set difference as approximation)
308	            ref_counter = Counter(ref_words)
309	            hyp_counter = Counter(hyp_words)
310
311	            # Deletions: in ref but not hyp
312	            for word, count in (ref_counter - hyp_counter).items():
313	                deletions[word] += count
314
315	            # Insertions: in hyp but not ref
316	            for word, count in (hyp_counter - ref_counter).items():
317	                insertions[word] += count
318
319	            # Substitutions: approximate by pairing mismatched words positionally
320	            min_len = min(len(ref_words), len(hyp_words))
321	            for i in range(min_len):
322	                if ref_words[i] != hyp_words[i]:
323	                    subs[(ref_words[i], hyp_words[i])] += 1
324
325	            # Check for entity mismatches (capitalized words that differ)
326	            for rw, hw in zip(ref_words[:min_len], hyp_words[:min_len]):
327	                if rw != hw and (rw[0:1].isupper() or any(c.isupper() for c in rw)):
328	                    entity_count += 1
329
330	        # Sort samples by WER for best/worst
331	        wer_scores.sort(key=lambda x: x[1])
332	        best = [sid for sid, _ in wer_scores[:3]]
333	        worst = [sid for sid, _ in wer_scores[-3:]]
334	        numeric_examples = [s["id"] for s in samples if "numeric_mismatch" in s.get("flags", [])][:3]
335	        entity_examples = [s["id"] for s in samples if "entity_mismatch" in s.get("flags", [])][:3]
336
337	        analysis[lang] = {
338	            "top_substitutions": [
339	                {"ref": r, "hyp": h, "count": c}
340	                for (r, h), c in subs.most_common(20)
341	            ],
342	            "top_insertions": [
343	                {"word": w, "count": c}
344	                for w, c in insertions.most_common(20)
345	            ],
346	            "top_deletions": [
347	                {"word": w, "count": c}
348	                for w, c in deletions.most_common(20)
349	            ],
350	            "error_buckets": {
351	                "numeric_mismatch_count": numeric_mismatch_count,
352	                "punctuation_only_count": punct_only_count,
353	                "spacing_tokenization_count": spacing_count,
354	                "entity_mismatch_count": entity_count,
355	                "script_confusion_count": script_confusion_count,
356	                "empty_hypothesis_count": empty_count,
357	            },
358	            "examples": {
359	                "worst_samples": worst,
360	                "best_samples": best,
361	                "numeric_mismatch_samples": numeric_examples,
362	                "entity_mismatch_samples": entity_examples,
363	            }
364	        }
365
366	    # Summary
367	    all_langs = sorted(samples_by_lang.keys())
368	    # Determine diagnosis
369	    # Load from the metrics we'll compute separately - use a simple heuristic here
370	    analysis["__summary__"] = {
371	        "model_diagnosis": "recognition-limited",
372	        "primary_error_source": "recognition",
373	        "numeric_verbalization_impact": "moderate",
374	        "formatting_impact": "low",
375	        "worst_languages": [],  # filled in by caller
376	        "best_languages": [],
377	    }
378
379	    return analysis
380
381
382	# ── Main Generator ───────────────────────────────────────────────────────
383
384	def process_checkpoint(src_dir: str, ckpt_name: str, out_dir: str):
385	    """Process one checkpoint: read predictions, compute all tiers, write 3 files."""
386	    preds_path = Path(src_dir) / "predictions.json"
387	    old_metrics_path = Path(src_dir) / "metrics.json"
388
389	    with open(preds_path, encoding="utf-8") as f:
390	        preds = json.load(f)
391
392	    with open(old_metrics_path) as f:
393	        old_meta = json.load(f).get("__meta__", {})
394
395	    out_path = Path(out_dir)
396	    out_path.mkdir(parents=True, exist_ok=True)
397
398	    # ── Compute normalized texts for all samples ──
399	    enriched = []
400	    for p in preds:
401	        lang = p["language"]
402	        ref = p["reference"]
403	        hyp = p.get("hypothesis", "")
404
405	        ref_raw = norm_raw(ref)
406	        hyp_raw = norm_raw(hyp)
407	        ref_n = norm_standard(ref, lang)
408	        hyp_n = norm_standard(hyp, lang)
409	        ref_nc = norm_numcanon(ref, lang)
410	        hyp_nc = norm_numcanon(hyp, lang)
411	        ref_m = norm_mer(ref, lang)
412	        hyp_m = norm_mer(hyp, lang)
413
414	        detected = p.get("detected_language", "")
415	        expected_lang_name = lang.capitalize()
416
417	        wer_raw_val = sample_wer(ref_raw, hyp_raw)
418	        wer_norm_val = sample_wer(ref_n, hyp_n)
419	        # MER: use CER on space-stripped text (single char stream comparison)
420	        if not ref_m:
421	            mer_val = 0.0 if not hyp_m else 100.0
422	        elif not hyp_m:
423	            mer_val = 100.0
424	        else:
425	            mer_val = round(cer(ref_m, hyp_m) * 100, 2)
426
427	        # space_norm_wer per sample
428	        snw_err, snw_total = space_norm_wer_sample(ref_n, hyp_n)
429	        snw_val = round(snw_err / snw_total * 100, 2) if snw_total > 0 else 0.0
430
431	        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
432	                             detected, expected_lang_name, wer_norm_val)
433	        # spacing_error flag: content is right but word boundaries are wrong
434	        if wer_norm_val > mer_val and mer_val < 100:
435	            flags.append("spacing_error")
436
437	        enriched.append({
438	            "id": p["id"],
439	            "language": lang,
440	            "reference": ref,
441	            "hypothesis": hyp,
442	            "ref_raw": ref_raw,
443	            "hyp_raw": hyp_raw,
444	            "ref_norm": ref_n,
445	            "hyp_norm": hyp_n,
446	            "ref_numcanon": ref_nc,
447	            "hyp_numcanon": hyp_nc,
448	            "ref_mer": ref_m,
449	            "hyp_mer": hyp_m,
450	            "detected_language": detected,
451	            "wer_raw": wer_raw_val,
452	            "wer_norm": wer_norm_val,
453	            "space_norm_wer": snw_val,
454	            "snw_err": snw_err,
455	            "snw_total": snw_total,
456	            "mer": mer_val,
457	            "flags": flags,
458	        })
459
460	    # ── Compute per-language metrics ──
461	    by_lang = defaultdict(list)
462	    for s in enriched:
463	        by_lang[s["language"]].append(s)
464
465	    metrics = {}
466	    all_ref_raw, all_hyp_raw = [], []
467	    all_ref_n, all_hyp_n = [], []
468	    all_ref_nc, all_hyp_nc = [], []
469	    all_ref_m, all_hyp_m = [], []
470	    all_snw_err, all_snw_total = 0, 0
471
472	    for lang in sorted(by_lang.keys()):
473	        samples = by_lang[lang]
474	        n = len(samples)
475
476	        refs_raw = [s["ref_raw"] for s in samples if s["ref_norm"]]
477	        hyps_raw = [s["hyp_raw"] for s in samples if s["ref_norm"]]
478	        refs_n = [s["ref_norm"] for s in samples if s["ref_norm"]]
479	        hyps_n = [s["hyp_norm"] for s in samples if s["ref_norm"]]
480	        refs_nc = [s["ref_numcanon"] for s in samples if s["ref_norm"]]
481	        hyps_nc = [s["hyp_numcanon"] for s in samples if s["ref_norm"]]
482	        refs_m = [s["ref_mer"] for s in samples if s["ref_norm"]]
483	        hyps_m = [s["hyp_mer"] for s in samples if s["ref_norm"]]
484
485	        w_raw = round(wer(refs_raw, hyps_raw) * 100, 2) if refs_raw else 100.0
486	        w_norm = round(wer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
487	        w_nc = round(wer(refs_nc, hyps_nc) * 100, 2) if refs_nc else 100.0
488	        # MER: CER on space-stripped text (since it's one "word", WER is meaningless)
489	        w_mer = round(cer(refs_m, hyps_m) * 100, 2) if refs_m else 100.0
490	        c_norm = round(cer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
491	        empty = sum(1 for s in samples if "empty_hypothesis" in s["flags"])
492
493	        # space_norm_wer: micro-average over ref words
494	        lang_snw_err = sum(s["snw_err"] for s in samples if s["ref_norm"])
495	        lang_snw_total = sum(s["snw_total"] for s in samples if s["ref_norm"])
496	        w_snw = round(lang_snw_err / lang_snw_total * 100, 2) if lang_snw_total > 0 else 100.0
497
498	        metrics[lang] = {
499	            "n_samples": n,
500	            "wer_raw": w_raw,
501	            "wer_norm": w_norm,
502	            "wer_numcanon": w_nc,
503	            "space_norm_wer": w_snw,
504	            "mer": w_mer,
505	            "cer_norm": c_norm,
506	            "empty_hypotheses": empty,
507	            "normalization_delta": {
508	                "raw_to_norm": round(w_raw - w_norm, 2),
509	                "norm_to_numcanon": round(w_norm - w_nc, 2),
510	                "norm_to_space_norm": round(w_norm - w_snw, 2),
511	                "norm_to_mer": round(w_norm - w_mer, 2),
512	            }
513	        }
514
515	        all_ref_raw.extend(refs_raw)
516	        all_hyp_raw.extend(hyps_raw)
517	        all_ref_n.extend(refs_n)
518	        all_hyp_n.extend(hyps_n)
519	        all_ref_nc.extend(refs_nc)
520	        all_hyp_nc.extend(hyps_nc)
521	        all_ref_m.extend(refs_m)
522	        all_hyp_m.extend(hyps_m)
523	        all_snw_err += lang_snw_err
524	        all_snw_total += lang_snw_total
525
526	    # ── Aggregates ──
527	    metrics["__overall__"] = {
528	        "n_samples": len(all_ref_raw),
529	        "wer_raw": round(wer(all_ref_raw, all_hyp_raw) * 100, 2),
530	        "wer_norm": round(wer(all_ref_n, all_hyp_n) * 100, 2),
531	        "wer_numcanon": round(wer(all_ref_nc, all_hyp_nc) * 100, 2),
532	        "space_norm_wer": round(all_snw_err / all_snw_total * 100, 2) if all_snw_total > 0 else 100.0,
533	        "mer": round(cer(all_ref_m, all_hyp_m) * 100, 2),
534	        "cer_norm": round(cer(all_ref_n, all_hyp_n) * 100, 2),
535	    }
536
537	    lang_keys = sorted(k for k in metrics if not k.startswith("__"))
538	    metrics["__macro_avg__"] = {
539	        "n_languages": len(lang_keys),
540	        "wer_raw": round(np.mean([metrics[k]["wer_raw"] for k in lang_keys]), 2),
541	        "wer_norm": round(np.mean([metrics[k]["wer_norm"] for k in lang_keys]), 2),
542	        "wer_numcanon": round(np.mean([metrics[k]["wer_numcanon"] for k in lang_keys]), 2),
543	        "space_norm_wer": round(np.mean([metrics[k]["space_norm_wer"] for k in lang_keys]), 2),
544	        "mer": round(np.mean([metrics[k]["mer"] for k in lang_keys]), 2),
545	        "cer_norm": round(np.mean([metrics[k]["cer_norm"] for k in lang_keys]), 2),
546	    }
547
548	    import importlib.metadata
549	    jiwer_ver = importlib.metadata.version("jiwer")
550	    metrics["__meta__"] = {
551	        "checkpoint": old_meta.get("checkpoint", ""),
552	        "checkpoint_name": ckpt_name,
553	        "model_id": MODEL_ID,
554	        "model_type": MODEL_TYPE,
555	        "dataset": DATASET,
556	        "batch_size": old_meta.get("batch_size", 64),
557	        "inference_time_sec": old_meta.get("inference_time_sec", 0),
558	        "total_audio_sec": old_meta.get("total_audio_sec", 40354.42),
559	        "rtf": old_meta.get("rtf", 0),
560	        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
561	        "gpu": GPU_NAME,
562	        "framework": "vllm",
563	        "normalization_version": NORMALIZATION_VERSION,
564	        "jiwer_version": jiwer_ver,
565	    }
566
567	    # ── sample_analysis.json ──
568	    sample_analysis = []
569	    for s in enriched:
570	        sample_analysis.append({
571	            "id": s["id"],
572	            "language": s["language"],
573	            "reference": s["reference"],
574	            "hypothesis": s["hypothesis"],
575	            "ref_norm": s["ref_norm"],
576	            "hyp_norm": s["hyp_norm"],
577	            "ref_numcanon": s["ref_numcanon"],
578	            "hyp_numcanon": s["hyp_numcanon"],
579	            "ref_mer": s["ref_mer"],
580	            "hyp_mer": s["hyp_mer"],
581	            "detected_language": s["detected_language"],
582	            "wer_raw": s["wer_raw"],
583	            "wer_norm": s["wer_norm"],
584	            "space_norm_wer": s["space_norm_wer"],
585	            "mer": s["mer"],
586	            "flags": s["flags"],
587	        })
588
589	    # ── error_analysis.json ──
590	    samples_for_errors = defaultdict(list)
591	    for s in sample_analysis:
592	        samples_for_errors[s["language"]].append(s)
593
594	    error_analysis = compute_error_analysis(samples_for_errors)
595
596	    # Fill in summary worst/best languages
597	    sorted_langs = sorted(lang_keys, key=lambda k: metrics[k]["wer_norm"])
598	    error_analysis["__summary__"]["best_languages"] = sorted_langs[:3]
599	    error_analysis["__summary__"]["worst_languages"] = sorted_langs[-3:]
600
601	    # Determine diagnosis from normalization deltas
602	    avg_raw_to_norm = np.mean([metrics[k]["normalization_delta"]["raw_to_norm"] for k in lang_keys])
603	    avg_norm_to_nc = np.mean([metrics[k]["normalization_delta"]["norm_to_numcanon"] for k in lang_keys])
604
605	    if avg_raw_to_norm > 10:
606	        error_analysis["__summary__"]["model_diagnosis"] = "formatting-limited"
607	        error_analysis["__summary__"]["formatting_impact"] = "high"
608	    elif avg_norm_to_nc > 5:
609	        error_analysis["__summary__"]["model_diagnosis"] = "numeric-limited"
610	        error_analysis["__summary__"]["numeric_verbalization_impact"] = "high"
611	    else:
612	        error_analysis["__summary__"]["model_diagnosis"] = "recognition-limited"
613	        error_analysis["__summary__"]["formatting_impact"] = "low" if avg_raw_to_norm < 5 else "moderate"
614	        error_analysis["__summary__"]["numeric_verbalization_impact"] = "low" if avg_norm_to_nc < 1 else "moderate"
615
616	    # ── Write files ──
617	    with open(out_path / "metrics.json", "w", encoding="utf-8") as f:
618	        json.dump(metrics, f, indent=2, ensure_ascii=False)
619
620	    with open(out_path / "sample_analysis.json", "w", encoding="utf-8") as f:
621	        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)
622
623	    with open(out_path / "error_analysis.json", "w", encoding="utf-8") as f:
624	        json.dump(error_analysis, f, indent=2, ensure_ascii=False)
625
626	    ov = metrics["__overall__"]
627	    print(f"  {ckpt_name}: wer_raw={ov['wer_raw']}  wer_norm={ov['wer_norm']}  "
628	          f"wer_numcanon={ov['wer_numcanon']}  cer_norm={ov['cer_norm']}")
629	    return metrics
630
631
632	# ── Entry point ──────────────────────────────────────────────────────────
633
634	if __name__ == "__main__":
635	    CHECKPOINTS = [
636	        ("ckpt-24000", "ckpt-24000-vllm"),
637	        ("ckpt-72000", "ckpt-72000-vllm"),
638	        ("ckpt-100000", "ckpt-100000-vllm"),
639	        ("ckpt-170000", "ckpt-170000-vllm"),
640	        ("ckpt-200000", "ckpt-200000-vllm"),
641	        ("ckpt-250000", "ckpt-250000-vllm"),
642	        ("ckpt-300000", "ckpt-300000-vllm"),
643	    ]
644
645	    base_results = "/home/ubuntu/training/benchmark_results"
646	    base_output = "/home/ubuntu/training/benchmark_outputs/qwen3-asr"
647
648	    print("Generating schema-v1 outputs for qwen3-asr checkpoints...")
649	    for ckpt_name, src_dir_name in CHECKPOINTS:
650	        src_dir = f"{base_results}/{src_dir_name}"
651	        out_dir = f"{base_output}/{ckpt_name}"
652	        process_checkpoint(src_dir, ckpt_name, out_dir)
653
654	    print("\nDone. All outputs in /home/ubuntu/training/benchmark_outputs/qwen3-asr/")
655

#!/usr/bin/env python3
"""
Generate benchmark_outputs conforming to BENCHMARK_SCHEMA.md v1.

Reads raw predictions from benchmark_results/ and produces:
  - metrics.json       (4 metric tiers + normalization_delta + meta)
  - sample_analysis.json  (per-sample with flags)
  - error_analysis.json   (top subs/ins/del + error buckets + diagnosis)
"""

import json
import re
import sys
import unicodedata
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path

import numpy as np
from jiwer import cer, wer

# ── Constants ────────────────────────────────────────────────────────────

NORMALIZATION_VERSION = "v1"
GPU_NAME = "NVIDIA H200 80GB"
MODEL_ID = "qwen3-asr"
MODEL_TYPE = "Qwen3-ASR-1.7B"
DATASET = "BayAreaBoys/indic-asr-benchmark-6k"

LATIN_SCRIPT_LANGS = {"english"}

# Zero-width characters to strip
ZW_CHARS = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff\u00ad]")

# Language-aware punctuation sets
# Base punctuation (safe to remove for all languages)
BASE_PUNCT = set("!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~")
# Indic punctuation
INDIC_PUNCT = {
    "\u0964",  # Devanagari danda ।
    "\u0965",  # Devanagari double danda ॥
    "\u0970",  # Devanagari abbreviation sign
}
# We preserve danda for Indic but remove double-danda and others
INDIC_REMOVE_PUNCT = {"\u0965", "\u0970"}

# Curly quotes / special dashes to normalize before removal
QUOTE_MAP = {
    "\u2018": "'", "\u2019": "'", "\u201c": '"', "\u201d": '"',
    "\u2013": "-", "\u2014": "-", "\u2015": "-",
    "\u2026": "...",
}

# Number word mappings for English
EN_NUMBER_WORDS = {
    "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
    "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
    "ten": "10", "eleven": "11", "twelve": "12", "thirteen": "13",
    "fourteen": "14", "fifteen": "15", "sixteen": "16", "seventeen": "17",
    "eighteen": "18", "nineteen": "19", "twenty": "20", "thirty": "30",
    "forty": "40", "fifty": "50", "sixty": "60", "seventy": "70",
    "eighty": "80", "ninety": "90", "hundred": "100", "thousand": "1000",
    "lakh": "100000", "crore": "10000000",
}

# Hindi number words (Devanagari)
HI_NUMBER_WORDS = {
    "शून्य": "0", "एक": "1", "दो": "2", "तीन": "3", "चार": "4",
    "पांच": "5", "छह": "6", "सात": "7", "आठ": "8", "नौ": "9",
    "दस": "10", "ग्यारह": "11", "बारह": "12", "तेरह": "13",
    "चौदह": "14", "पंद्रह": "15", "सोलह": "16", "सत्रह": "17",
    "अठारह": "18", "उन्नीस": "19", "बीस": "20",
    "सौ": "100", "हज़ार": "1000", "हजार": "1000",
    "लाख": "100000", "करोड़": "10000000",
}

# ── Normalization Pipeline ───────────────────────────────────────────────

def norm_raw(text: str) -> str:
    """Tier 1: NFC unicode + trim only."""
    text = unicodedata.normalize("NFC", text)
    text = text.strip()
    return text

def norm_standard(text: str, language: str = "") -> str:
    """Tier 2 (wer_norm): NFKC + strip ZW + normalize whitespace + standardize punct + remove punct + case fold."""
    # Step 1: NFKC
    text = unicodedata.normalize("NFKC", text)
    # Step 2: Strip zero-width chars
    text = ZW_CHARS.sub("", text)
    # Step 3: Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    # Step 4: Standardize punctuation variants
    for old, new in QUOTE_MAP.items():
        text = text.replace(old, new)
    # Normalize double danda to single danda
    text = text.replace("\u0965", "\u0964")
    # Step 5: Remove punctuation (language-aware)
    # For Indic: keep danda (।) as it's segmentation-relevant... actually schema says remove
    # Schema: "Remove punctuation (norm/numcanon only, using language-aware set)"
    # Remove all base punctuation
    text = "".join(c for c in text if c not in BASE_PUNCT)
    # Remove Indic punctuation too (danda, double danda, abbreviation)
    text = "".join(c for c in text if c not in INDIC_PUNCT and c not in INDIC_REMOVE_PUNCT)
    # Step 6: Case fold (English/Latin only)
    if language in LATIN_SCRIPT_LANGS:
        text = text.lower()
    else:
        text = text.lower()  # safe: Indic scripts have no case, lowering is a no-op
    # Collapse whitespace again after removals
    text = re.sub(r"\s+", " ", text).strip()
    return text

def norm_mer(text: str, language: str = "") -> str:
    """Tier 4 (MER): norm_standard + remove ALL spaces."""
    text = norm_standard(text, language)
    text = text.replace(" ", "")
    return text

# ── space_norm_wer computation ───────────────────────────────────────

def space_norm_wer_sample(ref_norm: str, hyp_norm: str):
    """
    Word error rate after space-insensitive character alignment.
    Returns (error_words, total_words).
    """
    ref_words = ref_norm.split()
    if not ref_words:
        return (0, 0)

    ref_nospace = ref_norm.replace(" ", "")
    hyp_nospace = hyp_norm.replace(" ", "")
    total_words = len(ref_words)

    if ref_nospace == hyp_nospace:
        return (0, total_words)

    # char_to_word: ref_nospace[i] → word index
    char_to_word = []
    for word_idx, word in enumerate(ref_words):
        for _ in word:
            char_to_word.append(word_idx)

    n = len(ref_nospace)
    m = len(hyp_nospace)

    # Levenshtein DP
    dp = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        dp[i][0] = i
    for j in range(m + 1):
        dp[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if ref_nospace[i - 1] == hyp_nospace[j - 1]:
                dp[i][j] = dp[i - 1][j - 1]
            else:
                dp[i][j] = 1 + min(dp[i - 1][j - 1], dp[i - 1][j], dp[i][j - 1])

    # Backtrack and mark touched words
    touched = set()
    i, j = n, m
    while i > 0 or j > 0:
        if i > 0 and j > 0 and ref_nospace[i - 1] == hyp_nospace[j - 1] and dp[i][j] == dp[i - 1][j - 1]:
            i -= 1; j -= 1
        elif i > 0 and j > 0 and dp[i][j] == dp[i - 1][j - 1] + 1:
            touched.add(char_to_word[i - 1]); i -= 1; j -= 1
        elif i > 0 and dp[i][j] == dp[i - 1][j] + 1:
            touched.add(char_to_word[i - 1]); i -= 1
        elif j > 0 and dp[i][j] == dp[i][j - 1] + 1:
            if i > 0:
                touched.add(char_to_word[i - 1])
            elif i < n:
                touched.add(char_to_word[i])
            j -= 1
        else:
            break

    return (len(touched), total_words)

def norm_numcanon(text: str, language: str = "") -> str:
    """Tier 3 (wer_numcanon): norm_standard + number canonicalization."""
    text = norm_standard(text, language)
    # Remove digit grouping (commas in numbers)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)  # repeat for 1,00,000 style
    # For English, replace simple number words with digits
    if language == "english":
        words = text.split()
        out = []
        for w in words:
            if w in EN_NUMBER_WORDS:
                out.append(EN_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    # For Hindi
    if language == "hindi":
        words = text.split()
        out = []
        for w in words:
            if w in HI_NUMBER_WORDS:
                out.append(HI_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    return text

# ── Per-sample WER ───────────────────────────────────────────────────────

def sample_wer(ref: str, hyp: str) -> float:
    """Compute WER for a single sample. Returns 0-100 float."""
    if not ref.strip():
        return 0.0 if not hyp.strip() else 100.0
    if not hyp.strip():
        return 100.0
    return round(wer(ref, hyp) * 100, 2)

# ── Flag detection ───────────────────────────────────────────────────────

def detect_flags(ref: str, hyp: str, ref_n: str, hyp_n: str,
                 detected_lang: str, expected_lang: str, wer_norm_val: float) -> list:
    flags = []
    if ref == hyp:
        flags.append("exact_match")
    if ref_n == hyp_n:
        flags.append("exact_match_norm")
    if not hyp.strip():
        flags.append("empty_hypothesis")
    if ref != hyp and ref_n == hyp_n:
        flags.append("punctuation_only_diff")
    # Numeric mismatch: digits present in either but differ
    ref_digits = re.findall(r"\d+", ref_n)
    hyp_digits = re.findall(r"\d+", hyp_n)
    if ref_digits != hyp_digits and (ref_digits or hyp_digits):
        flags.append("numeric_mismatch")
    # Script mismatch: check if hypothesis has different unicode blocks than reference
    if hyp.strip() and ref.strip():
        ref_scripts = set(unicodedata.script(c) for c in ref if not c.isspace() and c.isalpha()) if hasattr(unicodedata, 'script') else set()
        # Simplified: check if any non-Latin non-Common chars in hyp are from a different block than ref
        # Use character ranges as proxy
        ref_devanagari = bool(re.search(r"[\u0900-\u097F]", ref))
        hyp_devanagari = bool(re.search(r"[\u0900-\u097F]", hyp))
        ref_bengali = bool(re.search(r"[\u0980-\u09FF]", ref))
        hyp_bengali = bool(re.search(r"[\u0980-\u09FF]", hyp))
        ref_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", ref))
        hyp_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", hyp))
        ref_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", ref))
        hyp_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", hyp))
        # If ref has one script but hyp has a different one prominently
        if (ref_devanagari and not hyp_devanagari and hyp_gujarati) or \
           (ref_bengali and not hyp_bengali and hyp_gurmukhi) or \
           (ref_gujarati and not hyp_gujarati and hyp_devanagari):
            flags.append("script_mismatch")
    # Language confusion
    if detected_lang and expected_lang:
        if detected_lang.lower() != expected_lang.lower():
            flags.append("lang_confusion")
    if wer_norm_val > 80:
        flags.append("high_wer")
    return flags

# ── Error Analysis ───────────────────────────────────────────────────────

def compute_error_analysis(samples_by_lang: dict) -> dict:
    """Compute top substitutions/insertions/deletions per language."""
    analysis = {}

    for lang, samples in sorted(samples_by_lang.items()):
        subs = Counter()
        insertions = Counter()
        deletions = Counter()
        numeric_mismatch_count = 0
        punct_only_count = 0
        spacing_count = 0
        entity_count = 0
        script_confusion_count = 0
        empty_count = 0

        wer_scores = []
        sample_ids = []

        for s in samples:
            ref_words = s["ref_norm"].split()
            hyp_words = s["hyp_norm"].split()
            flags = s.get("flags", [])

            if "empty_hypothesis" in flags:
                empty_count += 1
            if "numeric_mismatch" in flags:
                numeric_mismatch_count += 1
            if "punctuation_only_diff" in flags:
                punct_only_count += 1
            if "script_mismatch" in flags:
                script_confusion_count += 1

            wer_scores.append((s["id"], s.get("wer_norm", 100.0)))

            # Simple alignment for top errors (using set difference as approximation)
            ref_counter = Counter(ref_words)
            hyp_counter = Counter(hyp_words)

            # Deletions: in ref but not hyp
            for word, count in (ref_counter - hyp_counter).items():
                deletions[word] += count

            # Insertions: in hyp but not ref
            for word, count in (hyp_counter - ref_counter).items():
                insertions[word] += count

            # Substitutions: approximate by pairing mismatched words positionally
            min_len = min(len(ref_words), len(hyp_words))
            for i in range(min_len):
                if ref_words[i] != hyp_words[i]:
                    subs[(ref_words[i], hyp_words[i])] += 1

            # Check for entity mismatches (capitalized words that differ)
            for rw, hw in zip(ref_words[:min_len], hyp_words[:min_len]):
                if rw != hw and (rw[0:1].isupper() or any(c.isupper() for c in rw)):
                    entity_count += 1

        # Sort samples by WER for best/worst
        wer_scores.sort(key=lambda x: x[1])
        best = [sid for sid, _ in wer_scores[:3]]
        worst = [sid for sid, _ in wer_scores[-3:]]
        numeric_examples = [s["id"] for s in samples if "numeric_mismatch" in s.get("flags", [])][:3]
        entity_examples = [s["id"] for s in samples if "entity_mismatch" in s.get("flags", [])][:3]

        analysis[lang] = {
            "top_substitutions": [
                {"ref": r, "hyp": h, "count": c}
                for (r, h), c in subs.most_common(20)
            ],
            "top_insertions": [
                {"word": w, "count": c}
                for w, c in insertions.most_common(20)
            ],
            "top_deletions": [
                {"word": w, "count": c}
                for w, c in deletions.most_common(20)
            ],
            "error_buckets": {
                "numeric_mismatch_count": numeric_mismatch_count,
                "punctuation_only_count": punct_only_count,
                "spacing_tokenization_count": spacing_count,
                "entity_mismatch_count": entity_count,
                "script_confusion_count": script_confusion_count,
                "empty_hypothesis_count": empty_count,
            },
            "examples": {
                "worst_samples": worst,
                "best_samples": best,
                "numeric_mismatch_samples": numeric_examples,
                "entity_mismatch_samples": entity_examples,
            }
        }

    # Summary
    all_langs = sorted(samples_by_lang.keys())
    # Determine diagnosis
    # Load from the metrics we'll compute separately - use a simple heuristic here
    analysis["__summary__"] = {
        "model_diagnosis": "recognition-limited",
        "primary_error_source": "recognition",
        "numeric_verbalization_impact": "moderate",
        "formatting_impact": "low",
        "worst_languages": [],  # filled in by caller
        "best_languages": [],
    }

    return analysis

# ── Main Generator ───────────────────────────────────────────────────────

def process_checkpoint(src_dir: str, ckpt_name: str, out_dir: str):
    """Process one checkpoint: read predictions, compute all tiers, write 3 files."""
    preds_path = Path(src_dir) / "predictions.json"
    old_metrics_path = Path(src_dir) / "metrics.json"

    with open(preds_path, encoding="utf-8") as f:
        preds = json.load(f)

    with open(old_metrics_path) as f:
        old_meta = json.load(f).get("__meta__", {})

    out_path = Path(out_dir)
    out_path.mkdir(parents=True, exist_ok=True)

    # ── Compute normalized texts for all samples ──
    enriched = []
    for p in preds:
        lang = p["language"]
        ref = p["reference"]
        hyp = p.get("hypothesis", "")

        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)
        ref_m = norm_mer(ref, lang)
        hyp_m = norm_mer(hyp, lang)

        detected = p.get("detected_language", "")
        expected_lang_name = lang.capitalize()

        wer_raw_val = sample_wer(ref_raw, hyp_raw)
        wer_norm_val = sample_wer(ref_n, hyp_n)
        # MER: use CER on space-stripped text (single char stream comparison)
        if not ref_m:
            mer_val = 0.0 if not hyp_m else 100.0
        elif not hyp_m:
            mer_val = 100.0
        else:
            mer_val = round(cer(ref_m, hyp_m) * 100, 2)

        # space_norm_wer per sample
        snw_err, snw_total = space_norm_wer_sample(ref_n, hyp_n)
        snw_val = round(snw_err / snw_total * 100, 2) if snw_total > 0 else 0.0

        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
                             detected, expected_lang_name, wer_norm_val)
        # spacing_error flag: content is right but word boundaries are wrong
        if wer_norm_val > mer_val and mer_val < 100:
            flags.append("spacing_error")

        enriched.append({
            "id": p["id"],
            "language": lang,
            "reference": ref,
            "hypothesis": hyp,
            "ref_raw": ref_raw,
            "hyp_raw": hyp_raw,
            "ref_norm": ref_n,
            "hyp_norm": hyp_n,
            "ref_numcanon": ref_nc,
            "hyp_numcanon": hyp_nc,
            "ref_mer": ref_m,
            "hyp_mer": hyp_m,
            "detected_language": detected,
            "wer_raw": wer_raw_val,
            "wer_norm": wer_norm_val,
            "space_norm_wer": snw_val,
            "snw_err": snw_err,
            "snw_total": snw_total,
            "mer": mer_val,
            "flags": flags,
        })

    # ── Compute per-language metrics ──
    by_lang = defaultdict(list)
    for s in enriched:
        by_lang[s["language"]].append(s)

    metrics = {}
    all_ref_raw, all_hyp_raw = [], []
    all_ref_n, all_hyp_n = [], []
    all_ref_nc, all_hyp_nc = [], []
    all_ref_m, all_hyp_m = [], []
    all_snw_err, all_snw_total = 0, 0

    for lang in sorted(by_lang.keys()):
        samples = by_lang[lang]
        n = len(samples)

        refs_raw = [s["ref_raw"] for s in samples if s["ref_norm"]]
        hyps_raw = [s["hyp_raw"] for s in samples if s["ref_norm"]]
        refs_n = [s["ref_norm"] for s in samples if s["ref_norm"]]
        hyps_n = [s["hyp_norm"] for s in samples if s["ref_norm"]]
        refs_nc = [s["ref_numcanon"] for s in samples if s["ref_norm"]]
        hyps_nc = [s["hyp_numcanon"] for s in samples if s["ref_norm"]]
        refs_m = [s["ref_mer"] for s in samples if s["ref_norm"]]
        hyps_m = [s["hyp_mer"] for s in samples if s["ref_norm"]]

        w_raw = round(wer(refs_raw, hyps_raw) * 100, 2) if refs_raw else 100.0
        w_norm = round(wer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        w_nc = round(wer(refs_nc, hyps_nc) * 100, 2) if refs_nc else 100.0
        # MER: CER on space-stripped text (since it's one "word", WER is meaningless)
        w_mer = round(cer(refs_m, hyps_m) * 100, 2) if refs_m else 100.0
        c_norm = round(cer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        empty = sum(1 for s in samples if "empty_hypothesis" in s["flags"])

        # space_norm_wer: micro-average over ref words
        lang_snw_err = sum(s["snw_err"] for s in samples if s["ref_norm"])
        lang_snw_total = sum(s["snw_total"] for s in samples if s["ref_norm"])
        w_snw = round(lang_snw_err / lang_snw_total * 100, 2) if lang_snw_total > 0 else 100.0

        metrics[lang] = {
            "n_samples": n,
            "wer_raw": w_raw,
            "wer_norm": w_norm,
            "wer_numcanon": w_nc,
            "space_norm_wer": w_snw,
            "mer": w_mer,
            "cer_norm": c_norm,
            "empty_hypotheses": empty,
            "normalization_delta": {
                "raw_to_norm": round(w_raw - w_norm, 2),
                "norm_to_numcanon": round(w_norm - w_nc, 2),
                "norm_to_space_norm": round(w_norm - w_snw, 2),
                "norm_to_mer": round(w_norm - w_mer, 2),
            }
        }

        all_ref_raw.extend(refs_raw)
        all_hyp_raw.extend(hyps_raw)
        all_ref_n.extend(refs_n)
        all_hyp_n.extend(hyps_n)
        all_ref_nc.extend(refs_nc)
        all_hyp_nc.extend(hyps_nc)
        all_ref_m.extend(refs_m)
        all_hyp_m.extend(hyps_m)
        all_snw_err += lang_snw_err
        all_snw_total += lang_snw_total

    # ── Aggregates ──
    metrics["__overall__"] = {
        "n_samples": len(all_ref_raw),
        "wer_raw": round(wer(all_ref_raw, all_hyp_raw) * 100, 2),
        "wer_norm": round(wer(all_ref_n, all_hyp_n) * 100, 2),
        "wer_numcanon": round(wer(all_ref_nc, all_hyp_nc) * 100, 2),
        "space_norm_wer": round(all_snw_err / all_snw_total * 100, 2) if all_snw_total > 0 else 100.0,
        "mer": round(cer(all_ref_m, all_hyp_m) * 100, 2),
        "cer_norm": round(cer(all_ref_n, all_hyp_n) * 100, 2),
    }

    lang_keys = sorted(k for k in metrics if not k.startswith("__"))
    metrics["__macro_avg__"] = {
        "n_languages": len(lang_keys),
        "wer_raw": round(np.mean([metrics[k]["wer_raw"] for k in lang_keys]), 2),
        "wer_norm": round(np.mean([metrics[k]["wer_norm"] for k in lang_keys]), 2),
        "wer_numcanon": round(np.mean([metrics[k]["wer_numcanon"] for k in lang_keys]), 2),
        "space_norm_wer": round(np.mean([metrics[k]["space_norm_wer"] for k in lang_keys]), 2),
        "mer": round(np.mean([metrics[k]["mer"] for k in lang_keys]), 2),
        "cer_norm": round(np.mean([metrics[k]["cer_norm"] for k in lang_keys]), 2),
    }

    import importlib.metadata
    jiwer_ver = importlib.metadata.version("jiwer")
    metrics["__meta__"] = {
        "checkpoint": old_meta.get("checkpoint", ""),
        "checkpoint_name": ckpt_name,
        "model_id": MODEL_ID,
        "model_type": MODEL_TYPE,
        "dataset": DATASET,
        "batch_size": old_meta.get("batch_size", 64),
        "inference_time_sec": old_meta.get("inference_time_sec", 0),
        "total_audio_sec": old_meta.get("total_audio_sec", 40354.42),
        "rtf": old_meta.get("rtf", 0),
        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "gpu": GPU_NAME,
        "framework": "vllm",
        "normalization_version": NORMALIZATION_VERSION,
        "jiwer_version": jiwer_ver,
    }

    # ── sample_analysis.json ──
    sample_analysis = []
    for s in enriched:
        sample_analysis.append({
            "id": s["id"],
            "language": s["language"],
            "reference": s["reference"],
            "hypothesis": s["hypothesis"],
            "ref_norm": s["ref_norm"],
            "hyp_norm": s["hyp_norm"],
            "ref_numcanon": s["ref_numcanon"],
            "hyp_numcanon": s["hyp_numcanon"],
            "ref_mer": s["ref_mer"],
            "hyp_mer": s["hyp_mer"],
            "detected_language": s["detected_language"],
            "wer_raw": s["wer_raw"],
            "wer_norm": s["wer_norm"],
            "space_norm_wer": s["space_norm_wer"],
            "mer": s["mer"],
            "flags": s["flags"],
        })

    # ── error_analysis.json ──
    samples_for_errors = defaultdict(list)
    for s in sample_analysis:
        samples_for_errors[s["language"]].append(s)

    error_analysis = compute_error_analysis(samples_for_errors)

    # Fill in summary worst/best languages
    sorted_langs = sorted(lang_keys, key=lambda k: metrics[k]["wer_norm"])
    error_analysis["__summary__"]["best_languages"] = sorted_langs[:3]
    error_analysis["__summary__"]["worst_languages"] = sorted_langs[-3:]

    # Determine diagnosis from normalization deltas
    avg_raw_to_norm = np.mean([metrics[k]["normalization_delta"]["raw_to_norm"] for k in lang_keys])
    avg_norm_to_nc = np.mean([metrics[k]["normalization_delta"]["norm_to_numcanon"] for k in lang_keys])

    if avg_raw_to_norm > 10:
        error_analysis["__summary__"]["model_diagnosis"] = "formatting-limited"
        error_analysis["__summary__"]["formatting_impact"] = "high"
    elif avg_norm_to_nc > 5:
        error_analysis["__summary__"]["model_diagnosis"] = "numeric-limited"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "high"
    else:
        error_analysis["__summary__"]["model_diagnosis"] = "recognition-limited"
        error_analysis["__summary__"]["formatting_impact"] = "low" if avg_raw_to_norm < 5 else "moderate"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "low" if avg_norm_to_nc < 1 else "moderate"

    # ── Write files ──
    with open(out_path / "metrics.json", "w", encoding="utf-8") as f:
        json.dump(metrics, f, indent=2, ensure_ascii=False)

    with open(out_path / "sample_analysis.json", "w", encoding="utf-8") as f:
        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)

    with open(out_path / "error_analysis.json", "w", encoding="utf-8") as f:
        json.dump(error_analysis, f, indent=2, ensure_ascii=False)

    ov = metrics["__overall__"]
    print(f"  {ckpt_name}: wer_raw={ov['wer_raw']}  wer_norm={ov['wer_norm']}  "
          f"wer_numcanon={ov['wer_numcanon']}  cer_norm={ov['cer_norm']}")
    return metrics

# ── Entry point ──────────────────────────────────────────────────────────

if __name__ == "__main__":
    CHECKPOINTS = [
        ("ckpt-24000", "ckpt-24000-vllm"),
        ("ckpt-72000", "ckpt-72000-vllm"),
        ("ckpt-100000", "ckpt-100000-vllm"),
        ("ckpt-170000", "ckpt-170000-vllm"),
        ("ckpt-200000", "ckpt-200000-vllm"),
        ("ckpt-250000", "ckpt-250000-vllm"),
        ("ckpt-300000", "ckpt-300000-vllm"),
    ]

    base_results = "/home/ubuntu/training/benchmark_results"
    base_output = "/home/ubuntu/training/benchmark_outputs/qwen3-asr"

    print("Generating schema-v1 outputs for qwen3-asr checkpoints...")
    for ckpt_name, src_dir_name in CHECKPOINTS:
        src_dir = f"{base_results}/{src_dir_name}"
        out_dir = f"{base_output}/{ckpt_name}"
        process_checkpoint(src_dir, ckpt_name, out_dir)

    print("\nDone. All outputs in /home/ubuntu/training/benchmark_outputs/qwen3-asr/")


Let me check the R2 setup and look at an existing benchmark script for reference.


which rclone && rclone listremotes 2>/dev/null || echo "no rclone"; which aws && aws --version 2>/dev/null || echo "no aws cli"


no rclone
/usr/local/bin/aws
aws-cli/2.33.15 Python/3.13.11 Linux/6.8.0-57-generic exe/x86_64.ubuntu.22