# ASR Benchmark Schema

Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

---

## Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  <model_id>/
    <checkpoint_name>/
      metrics.json
      sample_analysis.json
      error_analysis.json
```

## Model Registry

| Model ID         | Display Name                    | Checkpoints              |
|------------------|---------------------------------|--------------------------|
| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

## Languages

| Language   | Code | Source                  | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Metric Tiers

Four metric tiers are computed. Each tier adds one normalization step on top of the previous.

| Metric         | Unicode | Whitespace     | Punctuation Removal | Case Fold | Number Canon | Purpose                        |
|----------------|---------|----------------|---------------------|-----------|--------------|--------------------------------|
| `wer_raw`      | NFC     | trim only      | No                  | No        | No           | Strict transcript fidelity     |
| `wer_norm`     | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**         |
| `wer_numcanon` | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness             |
| `mer`          | NFKC    | **all removed**| Yes (lang-aware)    | Yes*      | No           | Meaningful Error Rate          |
| `cer_norm`     | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic       |

*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

### MER — Meaningful Error Rate (Space-Normalized WER)

MER measures whether the model got the **content** right regardless of word boundary placement. It answers: "Did the model produce the right characters in the right order, even if it split or merged words incorrectly?"

**How it works**: Remove ALL spaces from both reference and hypothesis after standard normalization, then compute WER on the resulting single-token strings. This means:

```
"india"     == "indi a"      ✓ (split word — not penalized)
"good"      == "go od"       ✓ (split word — not penalized)
"hello world" == "helloworld" ✓ (merged words — not penalized)
"நிலையான"    == "நிலை யான"    ✓ (Indic word split — not penalized)
"india"     != "indonesia"   ✗ (different content — still an error)
```

**Why this matters for Indic ASR**: Many Indic scripts have ambiguous word boundaries. Models frequently produce correct character sequences but with incorrect spacing — especially for agglutinative languages (Tamil, Malayalam, Kannada) and compound words in Hindi/Marathi. Standard WER penalizes these as multiple errors (insertion + substitution), inflating the score. MER isolates true recognition errors from tokenization/segmentation errors.

**Computation**:
```python
def space_normalize(text: str) -> str:
    """Remove all spaces after standard normalization."""
    return norm_standard(text, language).replace(" ", "")

mer = wer(space_normalize(ref), space_normalize(hyp))
```

Note: Since space-normalized text is a single "word", MER is effectively identical to CER on the space-stripped text. But we report it as a WER-scale metric for direct comparison with other WER tiers.

**Interpretation**:
- If `wer_norm` is much worse than `mer`, the model has a **word segmentation problem** — it knows the content but not the boundaries.
- If `wer_norm` ≈ `mer`, spacing is not a significant error source.
- The gap `wer_norm - mer` quantifies exactly how much WER is inflated by spacing errors.

### Normalization Pipeline (applied in order)

```
Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon/mer)
Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
Step 3: Normalize whitespace (collapse runs, trim)
Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
Step 5: Remove punctuation (norm/numcanon/mer only, using language-aware set)
Step 6: Case fold (English only)
Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
Step 8: Space removal (mer only — remove ALL spaces to produce a single character stream)
```

---

## Schema: `metrics.json`

```jsonc
{
  "<language_name>": {
    "n_samples": 500,              // int
    "wer_raw": 42.10,             // float: % — minimal normalization
    "wer_norm": 34.07,            // float: % — primary metric
    "wer_numcanon": 32.50,        // float: % — with number canonicalization
    "mer": 28.40,                 // float: % — Meaningful Error Rate (space-normalized)
    "cer_norm": 12.30,            // float: % — character-level normalized
    "empty_hypotheses": 0,        // int: samples where model produced no output
    "normalization_delta": {
      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm (negative = norm helped)
      "norm_to_numcanon": -1.57,  // float: wer_norm - wer_numcanon
      "norm_to_mer": -5.67        // float: wer_norm - mer (quantifies spacing error inflation)
    }
  },
  // ... one key per language (12 total) ...

  "__overall__": {
    "n_samples": 6000,
    "wer_raw": 40.75,
    "wer_norm": 32.07,
    "wer_numcanon": 30.80,
    "mer": 26.50,
    "cer_norm": 11.34
  },

  "__macro_avg__": {
    "n_languages": 12,
    "wer_raw": 40.52,
    "wer_norm": 31.76,
    "wer_numcanon": 30.20,
    "mer": 26.10,
    "cer_norm": 11.03
  },

  "__meta__": {
    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
    "checkpoint_name": "ckpt-20000",
    "model_id": "gemma3n-e2b",
    "model_type": "gemma3n-E2B-asr",
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
    "batch_size": 128,
    "inference_time_sec": 723.7,
    "total_audio_sec": 40354.46,
    "rtf": 0.0179,
    "timestamp": "2026-03-26T14:30:00Z",
    "gpu": "NVIDIA A100 80GB",
    "framework": "transformers",
    "normalization_version": "v1",
    "jiwer_version": "3.1.0"
  }
}
```

### Required Fields

| Section          | Required                                                                                |
|------------------|--------------------------------------------------------------------------------------------------|
| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`                           |
| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`                         |
| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

---

## Schema: `sample_analysis.json`

Array of per-sample objects. One entry per sample (6000 total).

```jsonc
[
  {
    "id": "as_0000",
    "language": "assamese",
    "reference": "<ground truth>",
    "hypothesis": "<model output>",
    "ref_norm": "<after norm pipeline>",
    "hyp_norm": "<after norm pipeline>",
    "ref_numcanon": "<after numcanon pipeline>",
    "hyp_numcanon": "<after numcanon pipeline>",
    "ref_mer": "<after space removal>",
    "hyp_mer": "<after space removal>",
    "detected_language": "Assamese",
    "wer_raw": 45.0,
    "wer_norm": 30.0,
    "mer": 22.0,
    "flags": ["numeric_mismatch", "punctuation_only_diff", "spacing_error"]
  }
]
```

### Per-sample fields

| Field              | Type     | Required | Description                                              |
|--------------------|----------|----------|----------------------------------------------------------|
| `id`               | string   | Yes      | Unique sample ID                                         |
| `language`         | string   | Yes      | Language name, lowercase                                 |
| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
| `ref_mer`          | string   | No       | Reference after space removal (single char stream)       |
| `hyp_mer`          | string   | No       | Hypothesis after space removal (single char stream)      |
| `detected_language`| string   | No       | Language detected by model                               |
| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
| `mer`              | float    | No       | Per-sample MER — Meaningful Error Rate (%)               |
| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

### Flag Vocabulary

Agents SHOULD tag samples with applicable flags:

| Flag                    | Meaning                                                      |
|-------------------------|--------------------------------------------------------------|
| `exact_match`           | reference == hypothesis (raw)                                |
| `exact_match_norm`      | ref_norm == hyp_norm                                         |
| `numeric_mismatch`      | Difference involves digits or number words                   |
| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
| `empty_hypothesis`      | Model produced no output                                     |
| `script_mismatch`       | Hypothesis uses a different script than reference            |
| `lang_confusion`        | detected_language != expected language                        |
| `high_wer`              | wer_norm > 80%                                               |
| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |
| `spacing_error`         | wer_norm > mer — content is correct but word boundaries wrong|

---

## Schema: `error_analysis.json`

Per-language error breakdown. One file per checkpoint.

```jsonc
{
  "<language_name>": {
    "top_substitutions": [
      {"ref": "word_a", "hyp": "word_b", "count": 42},
      // top 20 by count
    ],
    "top_insertions": [
      {"word": "um", "count": 15},
      // top 20
    ],
    "top_deletions": [
      {"word": "the", "count": 30},
      // top 20
    ],
    "error_buckets": {
      "numeric_mismatch_count": 23,
      "punctuation_only_count": 45,
      "spacing_tokenization_count": 12,
      "entity_mismatch_count": 8,
      "script_confusion_count": 0,
      "empty_hypothesis_count": 0
    },
    "examples": {
      "worst_samples": ["as_0047", "as_0231", "as_0419"],
      "best_samples": ["as_0001", "as_0155", "as_0302"],
      "numeric_mismatch_samples": ["as_0089", "as_0201"],
      "entity_mismatch_samples": ["as_0134"]
    }
  },
  // ... one key per language ...

  "__summary__": {
    "model_diagnosis": "recognition-limited",
    "primary_error_source": "recognition",
    "numeric_verbalization_impact": "moderate",
    "formatting_impact": "low",
    "worst_languages": ["assamese", "malayalam", "kannada"],
    "best_languages": ["hindi", "english", "gujarati"]
  }
}
```

### `__summary__.model_diagnosis` values

| Value                 | Meaning                                           |
|-----------------------|---------------------------------------------------|
| `recognition-limited` | Core ASR errors dominate, normalization helps little |
| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
| `numeric-limited`     | Number verbalization is the primary error source   |
| `mixed`               | No single dominant error category                  |

### Required Fields

| Section        | Required                                                                    |
|----------------|-----------------------------------------------------------------------------|
| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

---

## Agent Instructions

### Output directory

```
/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
```

### Files to generate per checkpoint

| File                   | Required | Content                     |
|------------------------|----------|-----------------------------|
| `metrics.json`         | Yes      | All four metric tiers       |
| `sample_analysis.json` | Yes      | Per-sample predictions      |
| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

### Critical Rules

1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
4. All five metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`) are mandatory per language and in aggregates.
5. `normalization_delta` must be computed as `wer_raw - wer_norm`, `wer_norm - wer_numcanon`, and `wer_norm - mer`.
6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

### Model-Specific Notes

| Model            | model_id        | Checkpoint paths                                              |
|------------------|-----------------|---------------------------------------------------------------|
| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

### Metric Roles

| Role                        | Metric         | Use for                                              |
|-----------------------------|----------------|------------------------------------------------------|
| Primary research metric     | `wer_norm`     | Model comparison, paper reporting, dashboard default  |
| Strict production metric    | `wer_raw`      | Transcript fidelity, formatting quality               |
| Numeric robustness metric   | `wer_numcanon` | Isolating number verbalization errors                 |
| Content accuracy metric     | `mer`          | Meaningful Error Rate — ignores word boundary errors  |
| Script-sensitive metric     | `cer_norm`     | Indic script evaluation, character-level quality      |

---

## Dashboard Integration

### nginx endpoint (port 80 on 216.81.248.184)

```
GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
GET /api/benchmarks/<model_id>/                        → checkpoint listing
GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
```

### Next.js API routes (on Vercel)

```
GET /api/models                                → aggregated model + checkpoint list
GET /api/metrics?model=X&checkpoint=Y          → metrics.json
GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
GET /api/compare                               → cross-model WER comparison
```

Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

---

## Migration from Old Schema

Old schema fields map to new as follows:

| Old Field          | New Field        |
|--------------------|------------------|
| `wer`              | `wer_raw`        |
| `cer`              | (dropped — use `cer_norm`) |
| `wer_normalized`   | `wer_norm`       |
| `cer_normalized`   | `cer_norm`       |
| (did not exist)    | `wer_numcanon`   |
| (did not exist)    | `mer`            |
| (did not exist)    | `normalization_delta` |
| (did not exist)    | `error_analysis.json` |

Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.

-Four metric tiers are computed. Each tier adds one normalization step on top of the previous.

+Five metric tiers are computed. Each builds on standard normalization with one additional step.


Now update the files-to-generate count:


The file /home/ubuntu/training/benchmark_schema/BENCHMARK_SCHEMA.md has been updated successfully.

# ASR Benchmark Schema

Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

---

## Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  <model_id>/
    <checkpoint_name>/
      metrics.json
      sample_analysis.json
      error_analysis.json
```

## Model Registry

| Model ID         | Display Name                    | Checkpoints              |
|------------------|---------------------------------|--------------------------|
| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

## Languages

| Language   | Code | Source                  | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Metric Tiers

Five metric tiers are computed. Each builds on standard normalization with one additional step.

| Metric         | Unicode | Whitespace     | Punctuation Removal | Case Fold | Number Canon | Purpose                        |
|----------------|---------|----------------|---------------------|-----------|--------------|--------------------------------|
| `wer_raw`      | NFC     | trim only      | No                  | No        | No           | Strict transcript fidelity     |
| `wer_norm`     | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**         |
| `wer_numcanon` | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness             |
| `mer`          | NFKC    | **all removed**| Yes (lang-aware)    | Yes*      | No           | Meaningful Error Rate          |
| `cer_norm`     | NFKC    | normalize      | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic       |

*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

### MER — Meaningful Error Rate (Space-Normalized WER)

MER measures whether the model got the **content** right regardless of word boundary placement. It answers: "Did the model produce the right characters in the right order, even if it split or merged words incorrectly?"

**How it works**: Remove ALL spaces from both reference and hypothesis after standard normalization, then compute WER on the resulting single-token strings. This means:

```
"india"     == "indi a"      ✓ (split word — not penalized)
"good"      == "go od"       ✓ (split word — not penalized)
"hello world" == "helloworld" ✓ (merged words — not penalized)
"நிலையான"    == "நிலை யான"    ✓ (Indic word split — not penalized)
"india"     != "indonesia"   ✗ (different content — still an error)
```

**Why this matters for Indic ASR**: Many Indic scripts have ambiguous word boundaries. Models frequently produce correct character sequences but with incorrect spacing — especially for agglutinative languages (Tamil, Malayalam, Kannada) and compound words in Hindi/Marathi. Standard WER penalizes these as multiple errors (insertion + substitution), inflating the score. MER isolates true recognition errors from tokenization/segmentation errors.

**Computation**:
```python
def space_normalize(text: str) -> str:
    """Remove all spaces after standard normalization."""
    return norm_standard(text, language).replace(" ", "")

mer = wer(space_normalize(ref), space_normalize(hyp))
```

Note: Since space-normalized text is a single "word", MER is effectively identical to CER on the space-stripped text. But we report it as a WER-scale metric for direct comparison with other WER tiers.

**Interpretation**:
- If `wer_norm` is much worse than `mer`, the model has a **word segmentation problem** — it knows the content but not the boundaries.
- If `wer_norm` ≈ `mer`, spacing is not a significant error source.
- The gap `wer_norm - mer` quantifies exactly how much WER is inflated by spacing errors.

### Normalization Pipeline (applied in order)

```
Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon/mer)
Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
Step 3: Normalize whitespace (collapse runs, trim)
Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
Step 5: Remove punctuation (norm/numcanon/mer only, using language-aware set)
Step 6: Case fold (English only)
Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
Step 8: Space removal (mer only — remove ALL spaces to produce a single character stream)
```

---

## Schema: `metrics.json`

```jsonc
{
  "<language_name>": {
    "n_samples": 500,              // int
    "wer_raw": 42.10,             // float: % — minimal normalization
    "wer_norm": 34.07,            // float: % — primary metric
    "wer_numcanon": 32.50,        // float: % — with number canonicalization
    "mer": 28.40,                 // float: % — Meaningful Error Rate (space-normalized)
    "cer_norm": 12.30,            // float: % — character-level normalized
    "empty_hypotheses": 0,        // int: samples where model produced no output
    "normalization_delta": {
      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm (negative = norm helped)
      "norm_to_numcanon": -1.57,  // float: wer_norm - wer_numcanon
      "norm_to_mer": -5.67        // float: wer_norm - mer (quantifies spacing error inflation)
    }
  },
  // ... one key per language (12 total) ...

  "__overall__": {
    "n_samples": 6000,
    "wer_raw": 40.75,
    "wer_norm": 32.07,
    "wer_numcanon": 30.80,
    "mer": 26.50,
    "cer_norm": 11.34
  },

  "__macro_avg__": {
    "n_languages": 12,
    "wer_raw": 40.52,
    "wer_norm": 31.76,
    "wer_numcanon": 30.20,
    "mer": 26.10,
    "cer_norm": 11.03
  },

  "__meta__": {
    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
    "checkpoint_name": "ckpt-20000",
    "model_id": "gemma3n-e2b",
    "model_type": "gemma3n-E2B-asr",
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
    "batch_size": 128,
    "inference_time_sec": 723.7,
    "total_audio_sec": 40354.46,
    "rtf": 0.0179,
    "timestamp": "2026-03-26T14:30:00Z",
    "gpu": "NVIDIA A100 80GB",
    "framework": "transformers",
    "normalization_version": "v1",
    "jiwer_version": "3.1.0"
  }
}
```

### Required Fields

| Section          | Required                                                                                |
|------------------|--------------------------------------------------------------------------------------------------|
| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`                           |
| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`                         |
| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

---

## Schema: `sample_analysis.json`

Array of per-sample objects. One entry per sample (6000 total).

```jsonc
[
  {
    "id": "as_0000",
    "language": "assamese",
    "reference": "<ground truth>",
    "hypothesis": "<model output>",
    "ref_norm": "<after norm pipeline>",
    "hyp_norm": "<after norm pipeline>",
    "ref_numcanon": "<after numcanon pipeline>",
    "hyp_numcanon": "<after numcanon pipeline>",
    "ref_mer": "<after space removal>",
    "hyp_mer": "<after space removal>",
    "detected_language": "Assamese",
    "wer_raw": 45.0,
    "wer_norm": 30.0,
    "mer": 22.0,
    "flags": ["numeric_mismatch", "punctuation_only_diff", "spacing_error"]
  }
]
```

### Per-sample fields

| Field              | Type     | Required | Description                                              |
|--------------------|----------|----------|----------------------------------------------------------|
| `id`               | string   | Yes      | Unique sample ID                                         |
| `language`         | string   | Yes      | Language name, lowercase                                 |
| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
| `ref_mer`          | string   | No       | Reference after space removal (single char stream)       |
| `hyp_mer`          | string   | No       | Hypothesis after space removal (single char stream)      |
| `detected_language`| string   | No       | Language detected by model                               |
| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
| `mer`              | float    | No       | Per-sample MER — Meaningful Error Rate (%)               |
| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

### Flag Vocabulary

Agents SHOULD tag samples with applicable flags:

| Flag                    | Meaning                                                      |
|-------------------------|--------------------------------------------------------------|
| `exact_match`           | reference == hypothesis (raw)                                |
| `exact_match_norm`      | ref_norm == hyp_norm                                         |
| `numeric_mismatch`      | Difference involves digits or number words                   |
| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
| `empty_hypothesis`      | Model produced no output                                     |
| `script_mismatch`       | Hypothesis uses a different script than reference            |
| `lang_confusion`        | detected_language != expected language                        |
| `high_wer`              | wer_norm > 80%                                               |
| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |
| `spacing_error`         | wer_norm > mer — content is correct but word boundaries wrong|

---

## Schema: `error_analysis.json`

Per-language error breakdown. One file per checkpoint.

```jsonc
{
  "<language_name>": {
    "top_substitutions": [
      {"ref": "word_a", "hyp": "word_b", "count": 42},
      // top 20 by count
    ],
    "top_insertions": [
      {"word": "um", "count": 15},
      // top 20
    ],
    "top_deletions": [
      {"word": "the", "count": 30},
      // top 20
    ],
    "error_buckets": {
      "numeric_mismatch_count": 23,
      "punctuation_only_count": 45,
      "spacing_tokenization_count": 12,
      "entity_mismatch_count": 8,
      "script_confusion_count": 0,
      "empty_hypothesis_count": 0
    },
    "examples": {
      "worst_samples": ["as_0047", "as_0231", "as_0419"],
      "best_samples": ["as_0001", "as_0155", "as_0302"],
      "numeric_mismatch_samples": ["as_0089", "as_0201"],
      "entity_mismatch_samples": ["as_0134"]
    }
  },
  // ... one key per language ...

  "__summary__": {
    "model_diagnosis": "recognition-limited",
    "primary_error_source": "recognition",
    "numeric_verbalization_impact": "moderate",
    "formatting_impact": "low",
    "worst_languages": ["assamese", "malayalam", "kannada"],
    "best_languages": ["hindi", "english", "gujarati"]
  }
}
```

### `__summary__.model_diagnosis` values

| Value                 | Meaning                                           |
|-----------------------|---------------------------------------------------|
| `recognition-limited` | Core ASR errors dominate, normalization helps little |
| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
| `numeric-limited`     | Number verbalization is the primary error source   |
| `mixed`               | No single dominant error category                  |

### Required Fields

| Section        | Required                                                                    |
|----------------|-----------------------------------------------------------------------------|
| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

---

## Agent Instructions

### Output directory

```
/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
```

### Files to generate per checkpoint

| File                   | Required | Content                     |
|------------------------|----------|-----------------------------|
| `metrics.json`         | Yes      | All four metric tiers       |
| `sample_analysis.json` | Yes      | Per-sample predictions      |
| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

### Critical Rules

1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
4. All five metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `mer`, `cer_norm`) are mandatory per language and in aggregates.
5. `normalization_delta` must be computed as `wer_raw - wer_norm`, `wer_norm - wer_numcanon`, and `wer_norm - mer`.
6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

### Model-Specific Notes

| Model            | model_id        | Checkpoint paths                                              |
|------------------|-----------------|---------------------------------------------------------------|
| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

### Metric Roles

| Role                        | Metric         | Use for                                              |
|-----------------------------|----------------|------------------------------------------------------|
| Primary research metric     | `wer_norm`     | Model comparison, paper reporting, dashboard default  |
| Strict production metric    | `wer_raw`      | Transcript fidelity, formatting quality               |
| Numeric robustness metric   | `wer_numcanon` | Isolating number verbalization errors                 |
| Content accuracy metric     | `mer`          | Meaningful Error Rate — ignores word boundary errors  |
| Script-sensitive metric     | `cer_norm`     | Indic script evaluation, character-level quality      |

---

## Dashboard Integration

### nginx endpoint (port 80 on 216.81.248.184)

```
GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
GET /api/benchmarks/<model_id>/                        → checkpoint listing
GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
```

### Next.js API routes (on Vercel)

```
GET /api/models                                → aggregated model + checkpoint list
GET /api/metrics?model=X&checkpoint=Y          → metrics.json
GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
GET /api/compare                               → cross-model WER comparison
```

Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

---

## Migration from Old Schema

Old schema fields map to new as follows:

| Old Field          | New Field        |
|--------------------|------------------|
| `wer`              | `wer_raw`        |
| `cer`              | (dropped — use `cer_norm`) |
| `wer_normalized`   | `wer_norm`       |
| `cer_normalized`   | `cer_norm`       |
| (did not exist)    | `wer_numcanon`   |
| (did not exist)    | `mer`            |
| (did not exist)    | `normalization_delta` |
| (did not exist)    | `error_analysis.json` |

Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.

| File                   | Required | Content                     |

-| `metrics.json`         | Yes      | All four metric tiers       |

+| `metrics.json`         | Yes      | All five metric tiers       |


Schema is updated. Now let me regenerate all qwen3-asr outputs with MER included:


Edit

def norm_numcanon(text: str, language: str = "") -> str:

def norm_mer(text: str, language: str = "") -> str:
    """Tier 4 (MER): norm_standard + remove ALL spaces."""
    text = norm_standard(text, language)
    text = text.replace(" ", "")
    return text

def norm_numcanon(text: str, language: str = "") -> str:


The file /home/ubuntu/training/generate_schema_outputs.py has been updated successfully.

def norm_mer(text: str, language: str = "") -> str:
    """Tier 4 (MER): norm_standard + remove ALL spaces."""
    text = norm_standard(text, language)
    text = text.replace(" ", "")
    return text

def norm_numcanon(text: str, language: str = "") -> str:

#!/usr/bin/env python3
"""
Generate benchmark_outputs conforming to BENCHMARK_SCHEMA.md v1.

Reads raw predictions from benchmark_results/ and produces:
  - metrics.json       (4 metric tiers + normalization_delta + meta)
  - sample_analysis.json  (per-sample with flags)
  - error_analysis.json   (top subs/ins/del + error buckets + diagnosis)
"""

import json
import re
import sys
import unicodedata
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path

import numpy as np
from jiwer import cer, wer

# ── Constants ────────────────────────────────────────────────────────────

NORMALIZATION_VERSION = "v1"
GPU_NAME = "NVIDIA H200 80GB"
MODEL_ID = "qwen3-asr"
MODEL_TYPE = "Qwen3-ASR-1.7B"
DATASET = "BayAreaBoys/indic-asr-benchmark-6k"

LATIN_SCRIPT_LANGS = {"english"}

# Zero-width characters to strip
ZW_CHARS = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff\u00ad]")

# Language-aware punctuation sets
# Base punctuation (safe to remove for all languages)
BASE_PUNCT = set("!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~")
# Indic punctuation
INDIC_PUNCT = {
    "\u0964",  # Devanagari danda ।
    "\u0965",  # Devanagari double danda ॥
    "\u0970",  # Devanagari abbreviation sign
}
# We preserve danda for Indic but remove double-danda and others
INDIC_REMOVE_PUNCT = {"\u0965", "\u0970"}

# Curly quotes / special dashes to normalize before removal
QUOTE_MAP = {
    "\u2018": "'", "\u2019": "'", "\u201c": '"', "\u201d": '"',
    "\u2013": "-", "\u2014": "-", "\u2015": "-",
    "\u2026": "...",
}

# Number word mappings for English
EN_NUMBER_WORDS = {
    "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
    "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
    "ten": "10", "eleven": "11", "twelve": "12", "thirteen": "13",
    "fourteen": "14", "fifteen": "15", "sixteen": "16", "seventeen": "17",
    "eighteen": "18", "nineteen": "19", "twenty": "20", "thirty": "30",
    "forty": "40", "fifty": "50", "sixty": "60", "seventy": "70",
    "eighty": "80", "ninety": "90", "hundred": "100", "thousand": "1000",
    "lakh": "100000", "crore": "10000000",
}

# Hindi number words (Devanagari)
HI_NUMBER_WORDS = {
    "शून्य": "0", "एक": "1", "दो": "2", "तीन": "3", "चार": "4",
    "पांच": "5", "छह": "6", "सात": "7", "आठ": "8", "नौ": "9",
    "दस": "10", "ग्यारह": "11", "बारह": "12", "तेरह": "13",
    "चौदह": "14", "पंद्रह": "15", "सोलह": "16", "सत्रह": "17",
    "अठारह": "18", "उन्नीस": "19", "बीस": "20",
    "सौ": "100", "हज़ार": "1000", "हजार": "1000",
    "लाख": "100000", "करोड़": "10000000",
}

# ── Normalization Pipeline ───────────────────────────────────────────────

def norm_raw(text: str) -> str:
    """Tier 1: NFC unicode + trim only."""
    text = unicodedata.normalize("NFC", text)
    text = text.strip()
    return text

def norm_standard(text: str, language: str = "") -> str:
    """Tier 2 (wer_norm): NFKC + strip ZW + normalize whitespace + standardize punct + remove punct + case fold."""
    # Step 1: NFKC
    text = unicodedata.normalize("NFKC", text)
    # Step 2: Strip zero-width chars
    text = ZW_CHARS.sub("", text)
    # Step 3: Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    # Step 4: Standardize punctuation variants
    for old, new in QUOTE_MAP.items():
        text = text.replace(old, new)
    # Normalize double danda to single danda
    text = text.replace("\u0965", "\u0964")
    # Step 5: Remove punctuation (language-aware)
    # For Indic: keep danda (।) as it's segmentation-relevant... actually schema says remove
    # Schema: "Remove punctuation (norm/numcanon only, using language-aware set)"
    # Remove all base punctuation
    text = "".join(c for c in text if c not in BASE_PUNCT)
    # Remove Indic punctuation too (danda, double danda, abbreviation)
    text = "".join(c for c in text if c not in INDIC_PUNCT and c not in INDIC_REMOVE_PUNCT)
    # Step 6: Case fold (English/Latin only)
    if language in LATIN_SCRIPT_LANGS:
        text = text.lower()
    else:
        text = text.lower()  # safe: Indic scripts have no case, lowering is a no-op
    # Collapse whitespace again after removals
    text = re.sub(r"\s+", " ", text).strip()
    return text

def norm_numcanon(text: str, language: str = "") -> str:
    """Tier 3 (wer_numcanon): norm_standard + number canonicalization."""
    text = norm_standard(text, language)
    # Remove digit grouping (commas in numbers)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)  # repeat for 1,00,000 style
    # For English, replace simple number words with digits
    if language == "english":
        words = text.split()
        out = []
        for w in words:
            if w in EN_NUMBER_WORDS:
                out.append(EN_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    # For Hindi
    if language == "hindi":
        words = text.split()
        out = []
        for w in words:
            if w in HI_NUMBER_WORDS:
                out.append(HI_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    return text

# ── Per-sample WER ───────────────────────────────────────────────────────

def sample_wer(ref: str, hyp: str) -> float:
    """Compute WER for a single sample. Returns 0-100 float."""
    if not ref.strip():
        return 0.0 if not hyp.strip() else 100.0
    if not hyp.strip():
        return 100.0
    return round(wer(ref, hyp) * 100, 2)

# ── Flag detection ───────────────────────────────────────────────────────

def detect_flags(ref: str, hyp: str, ref_n: str, hyp_n: str,
                 detected_lang: str, expected_lang: str, wer_norm_val: float) -> list:
    flags = []
    if ref == hyp:
        flags.append("exact_match")
    if ref_n == hyp_n:
        flags.append("exact_match_norm")
    if not hyp.strip():
        flags.append("empty_hypothesis")
    if ref != hyp and ref_n == hyp_n:
        flags.append("punctuation_only_diff")
    # Numeric mismatch: digits present in either but differ
    ref_digits = re.findall(r"\d+", ref_n)
    hyp_digits = re.findall(r"\d+", hyp_n)
    if ref_digits != hyp_digits and (ref_digits or hyp_digits):
        flags.append("numeric_mismatch")
    # Script mismatch: check if hypothesis has different unicode blocks than reference
    if hyp.strip() and ref.strip():
        ref_scripts = set(unicodedata.script(c) for c in ref if not c.isspace() and c.isalpha()) if hasattr(unicodedata, 'script') else set()
        # Simplified: check if any non-Latin non-Common chars in hyp are from a different block than ref
        # Use character ranges as proxy
        ref_devanagari = bool(re.search(r"[\u0900-\u097F]", ref))
        hyp_devanagari = bool(re.search(r"[\u0900-\u097F]", hyp))
        ref_bengali = bool(re.search(r"[\u0980-\u09FF]", ref))
        hyp_bengali = bool(re.search(r"[\u0980-\u09FF]", hyp))
        ref_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", ref))
        hyp_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", hyp))
        ref_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", ref))
        hyp_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", hyp))
        # If ref has one script but hyp has a different one prominently
        if (ref_devanagari and not hyp_devanagari and hyp_gujarati) or \
           (ref_bengali and not hyp_bengali and hyp_gurmukhi) or \
           (ref_gujarati and not hyp_gujarati and hyp_devanagari):
            flags.append("script_mismatch")
    # Language confusion
    if detected_lang and expected_lang:
        if detected_lang.lower() != expected_lang.lower():
            flags.append("lang_confusion")
    if wer_norm_val > 80:
        flags.append("high_wer")
    return flags

# ── Error Analysis ───────────────────────────────────────────────────────

def compute_error_analysis(samples_by_lang: dict) -> dict:
    """Compute top substitutions/insertions/deletions per language."""
    analysis = {}

    for lang, samples in sorted(samples_by_lang.items()):
        subs = Counter()
        insertions = Counter()
        deletions = Counter()
        numeric_mismatch_count = 0
        punct_only_count = 0
        spacing_count = 0
        entity_count = 0
        script_confusion_count = 0
        empty_count = 0

        wer_scores = []
        sample_ids = []

        for s in samples:
            ref_words = s["ref_norm"].split()
            hyp_words = s["hyp_norm"].split()
            flags = s.get("flags", [])

            if "empty_hypothesis" in flags:
                empty_count += 1
            if "numeric_mismatch" in flags:
                numeric_mismatch_count += 1
            if "punctuation_only_diff" in flags:
                punct_only_count += 1
            if "script_mismatch" in flags:
                script_confusion_count += 1

            wer_scores.append((s["id"], s.get("wer_norm", 100.0)))

            # Simple alignment for top errors (using set difference as approximation)
            ref_counter = Counter(ref_words)
            hyp_counter = Counter(hyp_words)

            # Deletions: in ref but not hyp
            for word, count in (ref_counter - hyp_counter).items():
                deletions[word] += count

            # Insertions: in hyp but not ref
            for word, count in (hyp_counter - ref_counter).items():
                insertions[word] += count

            # Substitutions: approximate by pairing mismatched words positionally
            min_len = min(len(ref_words), len(hyp_words))
            for i in range(min_len):
                if ref_words[i] != hyp_words[i]:
                    subs[(ref_words[i], hyp_words[i])] += 1

            # Check for entity mismatches (capitalized words that differ)
            for rw, hw in zip(ref_words[:min_len], hyp_words[:min_len]):
                if rw != hw and (rw[0:1].isupper() or any(c.isupper() for c in rw)):
                    entity_count += 1

        # Sort samples by WER for best/worst
        wer_scores.sort(key=lambda x: x[1])
        best = [sid for sid, _ in wer_scores[:3]]
        worst = [sid for sid, _ in wer_scores[-3:]]
        numeric_examples = [s["id"] for s in samples if "numeric_mismatch" in s.get("flags", [])][:3]
        entity_examples = [s["id"] for s in samples if "entity_mismatch" in s.get("flags", [])][:3]

        analysis[lang] = {
            "top_substitutions": [
                {"ref": r, "hyp": h, "count": c}
                for (r, h), c in subs.most_common(20)
            ],
            "top_insertions": [
                {"word": w, "count": c}
                for w, c in insertions.most_common(20)
            ],
            "top_deletions": [
                {"word": w, "count": c}
                for w, c in deletions.most_common(20)
            ],
            "error_buckets": {
                "numeric_mismatch_count": numeric_mismatch_count,
                "punctuation_only_count": punct_only_count,
                "spacing_tokenization_count": spacing_count,
                "entity_mismatch_count": entity_count,
                "script_confusion_count": script_confusion_count,
                "empty_hypothesis_count": empty_count,
            },
            "examples": {
                "worst_samples": worst,
                "best_samples": best,
                "numeric_mismatch_samples": numeric_examples,
                "entity_mismatch_samples": entity_examples,
            }
        }

    # Summary
    all_langs = sorted(samples_by_lang.keys())
    # Determine diagnosis
    # Load from the metrics we'll compute separately - use a simple heuristic here
    analysis["__summary__"] = {
        "model_diagnosis": "recognition-limited",
        "primary_error_source": "recognition",
        "numeric_verbalization_impact": "moderate",
        "formatting_impact": "low",
        "worst_languages": [],  # filled in by caller
        "best_languages": [],
    }

    return analysis

# ── Main Generator ───────────────────────────────────────────────────────

def process_checkpoint(src_dir: str, ckpt_name: str, out_dir: str):
    """Process one checkpoint: read predictions, compute all tiers, write 3 files."""
    preds_path = Path(src_dir) / "predictions.json"
    old_metrics_path = Path(src_dir) / "metrics.json"

    with open(preds_path, encoding="utf-8") as f:
        preds = json.load(f)

    with open(old_metrics_path) as f:
        old_meta = json.load(f).get("__meta__", {})

    out_path = Path(out_dir)
    out_path.mkdir(parents=True, exist_ok=True)

    # ── Compute normalized texts for all samples ──
    enriched = []
    for p in preds:
        lang = p["language"]
        ref = p["reference"]
        hyp = p.get("hypothesis", "")

        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)

        detected = p.get("detected_language", "")
        expected_lang_name = lang.capitalize()

        wer_raw_val = sample_wer(ref_raw, hyp_raw)
        wer_norm_val = sample_wer(ref_n, hyp_n)

        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
                             detected, expected_lang_name, wer_norm_val)

        enriched.append({
            "id": p["id"],
            "language": lang,
            "reference": ref,
            "hypothesis": hyp,
            "ref_raw": ref_raw,
            "hyp_raw": hyp_raw,
            "ref_norm": ref_n,
            "hyp_norm": hyp_n,
            "ref_numcanon": ref_nc,
            "hyp_numcanon": hyp_nc,
            "detected_language": detected,
            "wer_raw": wer_raw_val,
            "wer_norm": wer_norm_val,
            "flags": flags,
        })

    # ── Compute per-language metrics ──
    by_lang = defaultdict(list)
    for s in enriched:
        by_lang[s["language"]].append(s)

    metrics = {}
    all_ref_raw, all_hyp_raw = [], []
    all_ref_n, all_hyp_n = [], []
    all_ref_nc, all_hyp_nc = [], []

    for lang in sorted(by_lang.keys()):
        samples = by_lang[lang]
        n = len(samples)

        refs_raw = [s["ref_raw"] for s in samples if s["ref_norm"]]
        hyps_raw = [s["hyp_raw"] for s in samples if s["ref_norm"]]
        refs_n = [s["ref_norm"] for s in samples if s["ref_norm"]]
        hyps_n = [s["hyp_norm"] for s in samples if s["ref_norm"]]
        refs_nc = [s["ref_numcanon"] for s in samples if s["ref_norm"]]
        hyps_nc = [s["hyp_numcanon"] for s in samples if s["ref_norm"]]

        w_raw = round(wer(refs_raw, hyps_raw) * 100, 2) if refs_raw else 100.0
        w_norm = round(wer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        w_nc = round(wer(refs_nc, hyps_nc) * 100, 2) if refs_nc else 100.0
        c_norm = round(cer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        empty = sum(1 for s in samples if "empty_hypothesis" in s["flags"])

        metrics[lang] = {
            "n_samples": n,
            "wer_raw": w_raw,
            "wer_norm": w_norm,
            "wer_numcanon": w_nc,
            "cer_norm": c_norm,
            "empty_hypotheses": empty,
            "normalization_delta": {
                "raw_to_norm": round(w_raw - w_norm, 2),
                "norm_to_numcanon": round(w_norm - w_nc, 2),
            }
        }

        all_ref_raw.extend(refs_raw)
        all_hyp_raw.extend(hyps_raw)
        all_ref_n.extend(refs_n)
        all_hyp_n.extend(hyps_n)
        all_ref_nc.extend(refs_nc)
        all_hyp_nc.extend(hyps_nc)

    # ── Aggregates ──
    metrics["__overall__"] = {
        "n_samples": len(all_ref_raw),
        "wer_raw": round(wer(all_ref_raw, all_hyp_raw) * 100, 2),
        "wer_norm": round(wer(all_ref_n, all_hyp_n) * 100, 2),
        "wer_numcanon": round(wer(all_ref_nc, all_hyp_nc) * 100, 2),
        "cer_norm": round(cer(all_ref_n, all_hyp_n) * 100, 2),
    }

    lang_keys = sorted(k for k in metrics if not k.startswith("__"))
    metrics["__macro_avg__"] = {
        "n_languages": len(lang_keys),
        "wer_raw": round(np.mean([metrics[k]["wer_raw"] for k in lang_keys]), 2),
        "wer_norm": round(np.mean([metrics[k]["wer_norm"] for k in lang_keys]), 2),
        "wer_numcanon": round(np.mean([metrics[k]["wer_numcanon"] for k in lang_keys]), 2),
        "cer_norm": round(np.mean([metrics[k]["cer_norm"] for k in lang_keys]), 2),
    }

    import importlib.metadata
    jiwer_ver = importlib.metadata.version("jiwer")
    metrics["__meta__"] = {
        "checkpoint": old_meta.get("checkpoint", ""),
        "checkpoint_name": ckpt_name,
        "model_id": MODEL_ID,
        "model_type": MODEL_TYPE,
        "dataset": DATASET,
        "batch_size": old_meta.get("batch_size", 64),
        "inference_time_sec": old_meta.get("inference_time_sec", 0),
        "total_audio_sec": old_meta.get("total_audio_sec", 40354.42),
        "rtf": old_meta.get("rtf", 0),
        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "gpu": GPU_NAME,
        "framework": "vllm",
        "normalization_version": NORMALIZATION_VERSION,
        "jiwer_version": jiwer_ver,
    }

    # ── sample_analysis.json ──
    sample_analysis = []
    for s in enriched:
        sample_analysis.append({
            "id": s["id"],
            "language": s["language"],
            "reference": s["reference"],
            "hypothesis": s["hypothesis"],
            "ref_norm": s["ref_norm"],
            "hyp_norm": s["hyp_norm"],
            "ref_numcanon": s["ref_numcanon"],
            "hyp_numcanon": s["hyp_numcanon"],
            "detected_language": s["detected_language"],
            "wer_raw": s["wer_raw"],
            "wer_norm": s["wer_norm"],
            "flags": s["flags"],
        })

    # ── error_analysis.json ──
    samples_for_errors = defaultdict(list)
    for s in sample_analysis:
        samples_for_errors[s["language"]].append(s)

    error_analysis = compute_error_analysis(samples_for_errors)

    # Fill in summary worst/best languages
    sorted_langs = sorted(lang_keys, key=lambda k: metrics[k]["wer_norm"])
    error_analysis["__summary__"]["best_languages"] = sorted_langs[:3]
    error_analysis["__summary__"]["worst_languages"] = sorted_langs[-3:]

    # Determine diagnosis from normalization deltas
    avg_raw_to_norm = np.mean([metrics[k]["normalization_delta"]["raw_to_norm"] for k in lang_keys])
    avg_norm_to_nc = np.mean([metrics[k]["normalization_delta"]["norm_to_numcanon"] for k in lang_keys])

    if avg_raw_to_norm > 10:
        error_analysis["__summary__"]["model_diagnosis"] = "formatting-limited"
        error_analysis["__summary__"]["formatting_impact"] = "high"
    elif avg_norm_to_nc > 5:
        error_analysis["__summary__"]["model_diagnosis"] = "numeric-limited"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "high"
    else:
        error_analysis["__summary__"]["model_diagnosis"] = "recognition-limited"
        error_analysis["__summary__"]["formatting_impact"] = "low" if avg_raw_to_norm < 5 else "moderate"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "low" if avg_norm_to_nc < 1 else "moderate"

    # ── Write files ──
    with open(out_path / "metrics.json", "w", encoding="utf-8") as f:
        json.dump(metrics, f, indent=2, ensure_ascii=False)

    with open(out_path / "sample_analysis.json", "w", encoding="utf-8") as f:
        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)

    with open(out_path / "error_analysis.json", "w", encoding="utf-8") as f:
        json.dump(error_analysis, f, indent=2, ensure_ascii=False)

    ov = metrics["__overall__"]
    print(f"  {ckpt_name}: wer_raw={ov['wer_raw']}  wer_norm={ov['wer_norm']}  "
          f"wer_numcanon={ov['wer_numcanon']}  cer_norm={ov['cer_norm']}")
    return metrics

# ── Entry point ──────────────────────────────────────────────────────────

if __name__ == "__main__":
    CHECKPOINTS = [
        ("ckpt-24000", "ckpt-24000-vllm"),
        ("ckpt-72000", "ckpt-72000-vllm"),
        ("ckpt-100000", "ckpt-100000-vllm"),
        ("ckpt-170000", "ckpt-170000-vllm"),
    ]

    base_results = "/home/ubuntu/training/benchmark_results"
    base_output = "/home/ubuntu/training/benchmark_outputs/qwen3-asr"

    print("Generating schema-v1 outputs for qwen3-asr checkpoints...")
    for ckpt_name, src_dir_name in CHECKPOINTS:
        src_dir = f"{base_results}/{src_dir_name}"
        out_dir = f"{base_output}/{ckpt_name}"
        process_checkpoint(src_dir, ckpt_name, out_dir)

    print("\nDone. All outputs in /home/ubuntu/training/benchmark_outputs/qwen3-asr/")

+def norm_mer(text: str, language: str = "") -> str:

+    """Tier 4 (MER): norm_standard + remove ALL spaces."""

+    text = norm_standard(text, language)


Now add MER computation to the enrichment loop:


        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)

        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)
        ref_m = norm_mer(ref, lang)
        hyp_m = norm_mer(hyp, lang)


        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)
        ref_m = norm_mer(ref, lang)
        hyp_m = norm_mer(hyp, lang)

#!/usr/bin/env python3
"""
Generate benchmark_outputs conforming to BENCHMARK_SCHEMA.md v1.

Reads raw predictions from benchmark_results/ and produces:
  - metrics.json       (4 metric tiers + normalization_delta + meta)
  - sample_analysis.json  (per-sample with flags)
  - error_analysis.json   (top subs/ins/del + error buckets + diagnosis)
"""

import json
import re
import sys
import unicodedata
from collections import Counter, defaultdict
from datetime import datetime, timezone
from pathlib import Path

import numpy as np
from jiwer import cer, wer

# ── Constants ────────────────────────────────────────────────────────────

NORMALIZATION_VERSION = "v1"
GPU_NAME = "NVIDIA H200 80GB"
MODEL_ID = "qwen3-asr"
MODEL_TYPE = "Qwen3-ASR-1.7B"
DATASET = "BayAreaBoys/indic-asr-benchmark-6k"

LATIN_SCRIPT_LANGS = {"english"}

# Zero-width characters to strip
ZW_CHARS = re.compile(r"[\u200b-\u200f\u202a-\u202e\ufeff\u00ad]")

# Language-aware punctuation sets
# Base punctuation (safe to remove for all languages)
BASE_PUNCT = set("!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~")
# Indic punctuation
INDIC_PUNCT = {
    "\u0964",  # Devanagari danda ।
    "\u0965",  # Devanagari double danda ॥
    "\u0970",  # Devanagari abbreviation sign
}
# We preserve danda for Indic but remove double-danda and others
INDIC_REMOVE_PUNCT = {"\u0965", "\u0970"}

# Curly quotes / special dashes to normalize before removal
QUOTE_MAP = {
    "\u2018": "'", "\u2019": "'", "\u201c": '"', "\u201d": '"',
    "\u2013": "-", "\u2014": "-", "\u2015": "-",
    "\u2026": "...",
}

# Number word mappings for English
EN_NUMBER_WORDS = {
    "zero": "0", "one": "1", "two": "2", "three": "3", "four": "4",
    "five": "5", "six": "6", "seven": "7", "eight": "8", "nine": "9",
    "ten": "10", "eleven": "11", "twelve": "12", "thirteen": "13",
    "fourteen": "14", "fifteen": "15", "sixteen": "16", "seventeen": "17",
    "eighteen": "18", "nineteen": "19", "twenty": "20", "thirty": "30",
    "forty": "40", "fifty": "50", "sixty": "60", "seventy": "70",
    "eighty": "80", "ninety": "90", "hundred": "100", "thousand": "1000",
    "lakh": "100000", "crore": "10000000",
}

# Hindi number words (Devanagari)
HI_NUMBER_WORDS = {
    "शून्य": "0", "एक": "1", "दो": "2", "तीन": "3", "चार": "4",
    "पांच": "5", "छह": "6", "सात": "7", "आठ": "8", "नौ": "9",
    "दस": "10", "ग्यारह": "11", "बारह": "12", "तेरह": "13",
    "चौदह": "14", "पंद्रह": "15", "सोलह": "16", "सत्रह": "17",
    "अठारह": "18", "उन्नीस": "19", "बीस": "20",
    "सौ": "100", "हज़ार": "1000", "हजार": "1000",
    "लाख": "100000", "करोड़": "10000000",
}

# ── Normalization Pipeline ───────────────────────────────────────────────

def norm_raw(text: str) -> str:
    """Tier 1: NFC unicode + trim only."""
    text = unicodedata.normalize("NFC", text)
    text = text.strip()
    return text

def norm_standard(text: str, language: str = "") -> str:
    """Tier 2 (wer_norm): NFKC + strip ZW + normalize whitespace + standardize punct + remove punct + case fold."""
    # Step 1: NFKC
    text = unicodedata.normalize("NFKC", text)
    # Step 2: Strip zero-width chars
    text = ZW_CHARS.sub("", text)
    # Step 3: Normalize whitespace
    text = re.sub(r"\s+", " ", text).strip()
    # Step 4: Standardize punctuation variants
    for old, new in QUOTE_MAP.items():
        text = text.replace(old, new)
    # Normalize double danda to single danda
    text = text.replace("\u0965", "\u0964")
    # Step 5: Remove punctuation (language-aware)
    # For Indic: keep danda (।) as it's segmentation-relevant... actually schema says remove
    # Schema: "Remove punctuation (norm/numcanon only, using language-aware set)"
    # Remove all base punctuation
    text = "".join(c for c in text if c not in BASE_PUNCT)
    # Remove Indic punctuation too (danda, double danda, abbreviation)
    text = "".join(c for c in text if c not in INDIC_PUNCT and c not in INDIC_REMOVE_PUNCT)
    # Step 6: Case fold (English/Latin only)
    if language in LATIN_SCRIPT_LANGS:
        text = text.lower()
    else:
        text = text.lower()  # safe: Indic scripts have no case, lowering is a no-op
    # Collapse whitespace again after removals
    text = re.sub(r"\s+", " ", text).strip()
    return text

def norm_mer(text: str, language: str = "") -> str:
    """Tier 4 (MER): norm_standard + remove ALL spaces."""
    text = norm_standard(text, language)
    text = text.replace(" ", "")
    return text

def norm_numcanon(text: str, language: str = "") -> str:
    """Tier 3 (wer_numcanon): norm_standard + number canonicalization."""
    text = norm_standard(text, language)
    # Remove digit grouping (commas in numbers)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)
    text = re.sub(r"(\d),(\d)", r"\1\2", text)  # repeat for 1,00,000 style
    # For English, replace simple number words with digits
    if language == "english":
        words = text.split()
        out = []
        for w in words:
            if w in EN_NUMBER_WORDS:
                out.append(EN_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    # For Hindi
    if language == "hindi":
        words = text.split()
        out = []
        for w in words:
            if w in HI_NUMBER_WORDS:
                out.append(HI_NUMBER_WORDS[w])
            else:
                out.append(w)
        text = " ".join(out)
    return text

# ── Per-sample WER ───────────────────────────────────────────────────────

def sample_wer(ref: str, hyp: str) -> float:
    """Compute WER for a single sample. Returns 0-100 float."""
    if not ref.strip():
        return 0.0 if not hyp.strip() else 100.0
    if not hyp.strip():
        return 100.0
    return round(wer(ref, hyp) * 100, 2)

# ── Flag detection ───────────────────────────────────────────────────────

def detect_flags(ref: str, hyp: str, ref_n: str, hyp_n: str,
                 detected_lang: str, expected_lang: str, wer_norm_val: float) -> list:
    flags = []
    if ref == hyp:
        flags.append("exact_match")
    if ref_n == hyp_n:
        flags.append("exact_match_norm")
    if not hyp.strip():
        flags.append("empty_hypothesis")
    if ref != hyp and ref_n == hyp_n:
        flags.append("punctuation_only_diff")
    # Numeric mismatch: digits present in either but differ
    ref_digits = re.findall(r"\d+", ref_n)
    hyp_digits = re.findall(r"\d+", hyp_n)
    if ref_digits != hyp_digits and (ref_digits or hyp_digits):
        flags.append("numeric_mismatch")
    # Script mismatch: check if hypothesis has different unicode blocks than reference
    if hyp.strip() and ref.strip():
        ref_scripts = set(unicodedata.script(c) for c in ref if not c.isspace() and c.isalpha()) if hasattr(unicodedata, 'script') else set()
        # Simplified: check if any non-Latin non-Common chars in hyp are from a different block than ref
        # Use character ranges as proxy
        ref_devanagari = bool(re.search(r"[\u0900-\u097F]", ref))
        hyp_devanagari = bool(re.search(r"[\u0900-\u097F]", hyp))
        ref_bengali = bool(re.search(r"[\u0980-\u09FF]", ref))
        hyp_bengali = bool(re.search(r"[\u0980-\u09FF]", hyp))
        ref_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", ref))
        hyp_gujarati = bool(re.search(r"[\u0A80-\u0AFF]", hyp))
        ref_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", ref))
        hyp_gurmukhi = bool(re.search(r"[\u0A00-\u0A7F]", hyp))
        # If ref has one script but hyp has a different one prominently
        if (ref_devanagari and not hyp_devanagari and hyp_gujarati) or \
           (ref_bengali and not hyp_bengali and hyp_gurmukhi) or \
           (ref_gujarati and not hyp_gujarati and hyp_devanagari):
            flags.append("script_mismatch")
    # Language confusion
    if detected_lang and expected_lang:
        if detected_lang.lower() != expected_lang.lower():
            flags.append("lang_confusion")
    if wer_norm_val > 80:
        flags.append("high_wer")
    return flags

# ── Error Analysis ───────────────────────────────────────────────────────

def compute_error_analysis(samples_by_lang: dict) -> dict:
    """Compute top substitutions/insertions/deletions per language."""
    analysis = {}

    for lang, samples in sorted(samples_by_lang.items()):
        subs = Counter()
        insertions = Counter()
        deletions = Counter()
        numeric_mismatch_count = 0
        punct_only_count = 0
        spacing_count = 0
        entity_count = 0
        script_confusion_count = 0
        empty_count = 0

        wer_scores = []
        sample_ids = []

        for s in samples:
            ref_words = s["ref_norm"].split()
            hyp_words = s["hyp_norm"].split()
            flags = s.get("flags", [])

            if "empty_hypothesis" in flags:
                empty_count += 1
            if "numeric_mismatch" in flags:
                numeric_mismatch_count += 1
            if "punctuation_only_diff" in flags:
                punct_only_count += 1
            if "script_mismatch" in flags:
                script_confusion_count += 1

            wer_scores.append((s["id"], s.get("wer_norm", 100.0)))

            # Simple alignment for top errors (using set difference as approximation)
            ref_counter = Counter(ref_words)
            hyp_counter = Counter(hyp_words)

            # Deletions: in ref but not hyp
            for word, count in (ref_counter - hyp_counter).items():
                deletions[word] += count

            # Insertions: in hyp but not ref
            for word, count in (hyp_counter - ref_counter).items():
                insertions[word] += count

            # Substitutions: approximate by pairing mismatched words positionally
            min_len = min(len(ref_words), len(hyp_words))
            for i in range(min_len):
                if ref_words[i] != hyp_words[i]:
                    subs[(ref_words[i], hyp_words[i])] += 1

            # Check for entity mismatches (capitalized words that differ)
            for rw, hw in zip(ref_words[:min_len], hyp_words[:min_len]):
                if rw != hw and (rw[0:1].isupper() or any(c.isupper() for c in rw)):
                    entity_count += 1

        # Sort samples by WER for best/worst
        wer_scores.sort(key=lambda x: x[1])
        best = [sid for sid, _ in wer_scores[:3]]
        worst = [sid for sid, _ in wer_scores[-3:]]
        numeric_examples = [s["id"] for s in samples if "numeric_mismatch" in s.get("flags", [])][:3]
        entity_examples = [s["id"] for s in samples if "entity_mismatch" in s.get("flags", [])][:3]

        analysis[lang] = {
            "top_substitutions": [
                {"ref": r, "hyp": h, "count": c}
                for (r, h), c in subs.most_common(20)
            ],
            "top_insertions": [
                {"word": w, "count": c}
                for w, c in insertions.most_common(20)
            ],
            "top_deletions": [
                {"word": w, "count": c}
                for w, c in deletions.most_common(20)
            ],
            "error_buckets": {
                "numeric_mismatch_count": numeric_mismatch_count,
                "punctuation_only_count": punct_only_count,
                "spacing_tokenization_count": spacing_count,
                "entity_mismatch_count": entity_count,
                "script_confusion_count": script_confusion_count,
                "empty_hypothesis_count": empty_count,
            },
            "examples": {
                "worst_samples": worst,
                "best_samples": best,
                "numeric_mismatch_samples": numeric_examples,
                "entity_mismatch_samples": entity_examples,
            }
        }

    # Summary
    all_langs = sorted(samples_by_lang.keys())
    # Determine diagnosis
    # Load from the metrics we'll compute separately - use a simple heuristic here
    analysis["__summary__"] = {
        "model_diagnosis": "recognition-limited",
        "primary_error_source": "recognition",
        "numeric_verbalization_impact": "moderate",
        "formatting_impact": "low",
        "worst_languages": [],  # filled in by caller
        "best_languages": [],
    }

    return analysis

# ── Main Generator ───────────────────────────────────────────────────────

def process_checkpoint(src_dir: str, ckpt_name: str, out_dir: str):
    """Process one checkpoint: read predictions, compute all tiers, write 3 files."""
    preds_path = Path(src_dir) / "predictions.json"
    old_metrics_path = Path(src_dir) / "metrics.json"

    with open(preds_path, encoding="utf-8") as f:
        preds = json.load(f)

    with open(old_metrics_path) as f:
        old_meta = json.load(f).get("__meta__", {})

    out_path = Path(out_dir)
    out_path.mkdir(parents=True, exist_ok=True)

    # ── Compute normalized texts for all samples ──
    enriched = []
    for p in preds:
        lang = p["language"]
        ref = p["reference"]
        hyp = p.get("hypothesis", "")

        ref_raw = norm_raw(ref)
        hyp_raw = norm_raw(hyp)
        ref_n = norm_standard(ref, lang)
        hyp_n = norm_standard(hyp, lang)
        ref_nc = norm_numcanon(ref, lang)
        hyp_nc = norm_numcanon(hyp, lang)

        detected = p.get("detected_language", "")
        expected_lang_name = lang.capitalize()

        wer_raw_val = sample_wer(ref_raw, hyp_raw)
        wer_norm_val = sample_wer(ref_n, hyp_n)

        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
                             detected, expected_lang_name, wer_norm_val)

        enriched.append({
            "id": p["id"],
            "language": lang,
            "reference": ref,
            "hypothesis": hyp,
            "ref_raw": ref_raw,
            "hyp_raw": hyp_raw,
            "ref_norm": ref_n,
            "hyp_norm": hyp_n,
            "ref_numcanon": ref_nc,
            "hyp_numcanon": hyp_nc,
            "detected_language": detected,
            "wer_raw": wer_raw_val,
            "wer_norm": wer_norm_val,
            "flags": flags,
        })

    # ── Compute per-language metrics ──
    by_lang = defaultdict(list)
    for s in enriched:
        by_lang[s["language"]].append(s)

    metrics = {}
    all_ref_raw, all_hyp_raw = [], []
    all_ref_n, all_hyp_n = [], []
    all_ref_nc, all_hyp_nc = [], []

    for lang in sorted(by_lang.keys()):
        samples = by_lang[lang]
        n = len(samples)

        refs_raw = [s["ref_raw"] for s in samples if s["ref_norm"]]
        hyps_raw = [s["hyp_raw"] for s in samples if s["ref_norm"]]
        refs_n = [s["ref_norm"] for s in samples if s["ref_norm"]]
        hyps_n = [s["hyp_norm"] for s in samples if s["ref_norm"]]
        refs_nc = [s["ref_numcanon"] for s in samples if s["ref_norm"]]
        hyps_nc = [s["hyp_numcanon"] for s in samples if s["ref_norm"]]

        w_raw = round(wer(refs_raw, hyps_raw) * 100, 2) if refs_raw else 100.0
        w_norm = round(wer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        w_nc = round(wer(refs_nc, hyps_nc) * 100, 2) if refs_nc else 100.0
        c_norm = round(cer(refs_n, hyps_n) * 100, 2) if refs_n else 100.0
        empty = sum(1 for s in samples if "empty_hypothesis" in s["flags"])

        metrics[lang] = {
            "n_samples": n,
            "wer_raw": w_raw,
            "wer_norm": w_norm,
            "wer_numcanon": w_nc,
            "cer_norm": c_norm,
            "empty_hypotheses": empty,
            "normalization_delta": {
                "raw_to_norm": round(w_raw - w_norm, 2),
                "norm_to_numcanon": round(w_norm - w_nc, 2),
            }
        }

        all_ref_raw.extend(refs_raw)
        all_hyp_raw.extend(hyps_raw)
        all_ref_n.extend(refs_n)
        all_hyp_n.extend(hyps_n)
        all_ref_nc.extend(refs_nc)
        all_hyp_nc.extend(hyps_nc)

    # ── Aggregates ──
    metrics["__overall__"] = {
        "n_samples": len(all_ref_raw),
        "wer_raw": round(wer(all_ref_raw, all_hyp_raw) * 100, 2),
        "wer_norm": round(wer(all_ref_n, all_hyp_n) * 100, 2),
        "wer_numcanon": round(wer(all_ref_nc, all_hyp_nc) * 100, 2),
        "cer_norm": round(cer(all_ref_n, all_hyp_n) * 100, 2),
    }

    lang_keys = sorted(k for k in metrics if not k.startswith("__"))
    metrics["__macro_avg__"] = {
        "n_languages": len(lang_keys),
        "wer_raw": round(np.mean([metrics[k]["wer_raw"] for k in lang_keys]), 2),
        "wer_norm": round(np.mean([metrics[k]["wer_norm"] for k in lang_keys]), 2),
        "wer_numcanon": round(np.mean([metrics[k]["wer_numcanon"] for k in lang_keys]), 2),
        "cer_norm": round(np.mean([metrics[k]["cer_norm"] for k in lang_keys]), 2),
    }

    import importlib.metadata
    jiwer_ver = importlib.metadata.version("jiwer")
    metrics["__meta__"] = {
        "checkpoint": old_meta.get("checkpoint", ""),
        "checkpoint_name": ckpt_name,
        "model_id": MODEL_ID,
        "model_type": MODEL_TYPE,
        "dataset": DATASET,
        "batch_size": old_meta.get("batch_size", 64),
        "inference_time_sec": old_meta.get("inference_time_sec", 0),
        "total_audio_sec": old_meta.get("total_audio_sec", 40354.42),
        "rtf": old_meta.get("rtf", 0),
        "timestamp": datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),
        "gpu": GPU_NAME,
        "framework": "vllm",
        "normalization_version": NORMALIZATION_VERSION,
        "jiwer_version": jiwer_ver,
    }

    # ── sample_analysis.json ──
    sample_analysis = []
    for s in enriched:
        sample_analysis.append({
            "id": s["id"],
            "language": s["language"],
            "reference": s["reference"],
            "hypothesis": s["hypothesis"],
            "ref_norm": s["ref_norm"],
            "hyp_norm": s["hyp_norm"],
            "ref_numcanon": s["ref_numcanon"],
            "hyp_numcanon": s["hyp_numcanon"],
            "detected_language": s["detected_language"],
            "wer_raw": s["wer_raw"],
            "wer_norm": s["wer_norm"],
            "flags": s["flags"],
        })

    # ── error_analysis.json ──
    samples_for_errors = defaultdict(list)
    for s in sample_analysis:
        samples_for_errors[s["language"]].append(s)

    error_analysis = compute_error_analysis(samples_for_errors)

    # Fill in summary worst/best languages
    sorted_langs = sorted(lang_keys, key=lambda k: metrics[k]["wer_norm"])
    error_analysis["__summary__"]["best_languages"] = sorted_langs[:3]
    error_analysis["__summary__"]["worst_languages"] = sorted_langs[-3:]

    # Determine diagnosis from normalization deltas
    avg_raw_to_norm = np.mean([metrics[k]["normalization_delta"]["raw_to_norm"] for k in lang_keys])
    avg_norm_to_nc = np.mean([metrics[k]["normalization_delta"]["norm_to_numcanon"] for k in lang_keys])

    if avg_raw_to_norm > 10:
        error_analysis["__summary__"]["model_diagnosis"] = "formatting-limited"
        error_analysis["__summary__"]["formatting_impact"] = "high"
    elif avg_norm_to_nc > 5:
        error_analysis["__summary__"]["model_diagnosis"] = "numeric-limited"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "high"
    else:
        error_analysis["__summary__"]["model_diagnosis"] = "recognition-limited"
        error_analysis["__summary__"]["formatting_impact"] = "low" if avg_raw_to_norm < 5 else "moderate"
        error_analysis["__summary__"]["numeric_verbalization_impact"] = "low" if avg_norm_to_nc < 1 else "moderate"

    # ── Write files ──
    with open(out_path / "metrics.json", "w", encoding="utf-8") as f:
        json.dump(metrics, f, indent=2, ensure_ascii=False)

    with open(out_path / "sample_analysis.json", "w", encoding="utf-8") as f:
        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)

    with open(out_path / "error_analysis.json", "w", encoding="utf-8") as f:
        json.dump(error_analysis, f, indent=2, ensure_ascii=False)

    ov = metrics["__overall__"]
    print(f"  {ckpt_name}: wer_raw={ov['wer_raw']}  wer_norm={ov['wer_norm']}  "
          f"wer_numcanon={ov['wer_numcanon']}  cer_norm={ov['cer_norm']}")
    return metrics

# ── Entry point ──────────────────────────────────────────────────────────

if __name__ == "__main__":
    CHECKPOINTS = [
        ("ckpt-24000", "ckpt-24000-vllm"),
        ("ckpt-72000", "ckpt-72000-vllm"),
        ("ckpt-100000", "ckpt-100000-vllm"),
        ("ckpt-170000", "ckpt-170000-vllm"),
    ]

    base_results = "/home/ubuntu/training/benchmark_results"
    base_output = "/home/ubuntu/training/benchmark_outputs/qwen3-asr"

    print("Generating schema-v1 outputs for qwen3-asr checkpoints...")
    for ckpt_name, src_dir_name in CHECKPOINTS:
        src_dir = f"{base_results}/{src_dir_name}"
        out_dir = f"{base_output}/{ckpt_name}"
        process_checkpoint(src_dir, ckpt_name, out_dir)

    print("\nDone. All outputs in /home/ubuntu/training/benchmark_outputs/qwen3-asr/")


        wer_raw_val = sample_wer(ref_raw, hyp_raw)
        wer_norm_val = sample_wer(ref_n, hyp_n)

        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
                             detected, expected_lang_name, wer_norm_val)

        enriched.append({
            "id": p["id"],
            "language": lang,
            "reference": ref,
            "hypothesis": hyp,
            "ref_raw": ref_raw,
            "hyp_raw": hyp_raw,
            "ref_norm": ref_n,
            "hyp_norm": hyp_n,
            "ref_numcanon": ref_nc,
            "hyp_numcanon": hyp_nc,
            "detected_language": detected,
            "wer_raw": wer_raw_val,
            "wer_norm": wer_norm_val,
            "flags": flags,
        })

        wer_raw_val = sample_wer(ref_raw, hyp_raw)
        wer_norm_val = sample_wer(ref_n, hyp_n)
        mer_val = sample_wer(ref_m, hyp_m)

        flags = detect_flags(ref_raw, hyp_raw, ref_n, hyp_n,
                             detected, expected_lang_name, wer_norm_val)
        # spacing_error flag: content is right but word boundaries are wrong
        if wer_norm_val > mer_val and mer_val < 100:
            flags.append("spacing_error")

        enriched.append({
            "id": p["id"],
            "language": lang,
            "reference": ref,
            "hypothesis": hyp,
            "ref_raw": ref_raw,
            "hyp_raw": hyp_raw,
            "ref_norm": ref_n,
            "hyp_norm": hyp_n,
            "ref_numcanon": ref_nc,
            "hyp_numcanon": hyp_nc,
            "ref_mer": ref_m,
            "hyp_mer": hyp_m,
            "detected_language": detected,
            "wer_raw": wer_raw_val,
            "wer_norm": wer_norm_val,
            "mer": mer_val,
            "flags": flags,
        })