Write

# ASR Benchmark Schema

Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

---

## Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  <model_id>/
    <checkpoint_name>/
      metrics.json
      sample_analysis.json
      error_analysis.json
```

## Model Registry

| Model ID         | Display Name                    | Checkpoints              |
|------------------|---------------------------------|--------------------------|
| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

## Languages

| Language   | Code | Source                  | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Metric Tiers

Four metric tiers are computed. Each tier adds one normalization step on top of the previous.

| Metric         | Unicode | Whitespace | Punctuation Removal | Case Fold | Number Canon | Purpose                        |
|----------------|---------|------------|---------------------|-----------|--------------|--------------------------------|
| `wer_raw`      | NFC     | trim only  | No                  | No        | No           | Strict transcript fidelity     |
| `wer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**         |
| `wer_numcanon` | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness             |
| `cer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic       |

*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

### Normalization Pipeline (applied in order)

```
Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon)
Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
Step 3: Normalize whitespace (collapse runs, trim)
Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
Step 5: Remove punctuation (norm/numcanon only, using language-aware set — preserve Devanagari danda for segmentation)
Step 6: Case fold (English only)
Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
```

---

## Schema: `metrics.json`

```jsonc
{
  "<language_name>": {
    "n_samples": 500,              // int
    "wer_raw": 42.10,             // float: % — minimal normalization
    "wer_norm": 34.07,            // float: % — primary metric
    "wer_numcanon": 32.50,        // float: % — with number canonicalization
    "cer_norm": 12.30,            // float: % — character-level normalized
    "empty_hypotheses": 0,        // int: samples where model produced no output
    "normalization_delta": {
      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm (negative = norm helped)
      "norm_to_numcanon": -1.57   // float: wer_norm - wer_numcanon
    }
  },
  // ... one key per language (12 total) ...

  "__overall__": {
    "n_samples": 6000,
    "wer_raw": 40.75,
    "wer_norm": 32.07,
    "wer_numcanon": 30.80,
    "cer_norm": 11.34
  },

  "__macro_avg__": {
    "n_languages": 12,
    "wer_raw": 40.52,
    "wer_norm": 31.76,
    "wer_numcanon": 30.20,
    "cer_norm": 11.03
  },

  "__meta__": {
    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
    "checkpoint_name": "ckpt-20000",
    "model_id": "gemma3n-e2b",
    "model_type": "gemma3n-E2B-asr",
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
    "batch_size": 128,
    "inference_time_sec": 723.7,
    "total_audio_sec": 40354.46,
    "rtf": 0.0179,
    "timestamp": "2026-03-26T14:30:00Z",
    "gpu": "NVIDIA A100 80GB",
    "framework": "transformers",
    "normalization_version": "v1",
    "jiwer_version": "3.1.0"
  }
}
```

### Required Fields

| Section          | Required                                                                                |
|------------------|--------------------------------------------------------------------------------------------------|
| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                  |
| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                |
| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

---

## Schema: `sample_analysis.json`

Array of per-sample objects. One entry per sample (6000 total).

```jsonc
[
  {
    "id": "as_0000",
    "language": "assamese",
    "reference": "<ground truth>",
    "hypothesis": "<model output>",
    "ref_norm": "<after norm pipeline>",
    "hyp_norm": "<after norm pipeline>",
    "ref_numcanon": "<after numcanon pipeline>",
    "hyp_numcanon": "<after numcanon pipeline>",
    "detected_language": "Assamese",
    "wer_raw": 45.0,
    "wer_norm": 30.0,
    "flags": ["numeric_mismatch", "punctuation_only_diff"]
  }
]
```

### Per-sample fields

| Field              | Type     | Required | Description                                              |
|--------------------|----------|----------|----------------------------------------------------------|
| `id`               | string   | Yes      | Unique sample ID                                         |
| `language`         | string   | Yes      | Language name, lowercase                                 |
| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
| `detected_language`| string   | No       | Language detected by model                               |
| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

### Flag Vocabulary

Agents SHOULD tag samples with applicable flags:

| Flag                    | Meaning                                                      |
|-------------------------|--------------------------------------------------------------|
| `exact_match`           | reference == hypothesis (raw)                                |
| `exact_match_norm`      | ref_norm == hyp_norm                                         |
| `numeric_mismatch`      | Difference involves digits or number words                   |
| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
| `empty_hypothesis`      | Model produced no output                                     |
| `script_mismatch`       | Hypothesis uses a different script than reference            |
| `lang_confusion`        | detected_language != expected language                        |
| `high_wer`              | wer_norm > 80%                                               |
| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |

---

## Schema: `error_analysis.json`

Per-language error breakdown. One file per checkpoint.

```jsonc
{
  "<language_name>": {
    "top_substitutions": [
      {"ref": "word_a", "hyp": "word_b", "count": 42},
      // top 20 by count
    ],
    "top_insertions": [
      {"word": "um", "count": 15},
      // top 20
    ],
    "top_deletions": [
      {"word": "the", "count": 30},
      // top 20
    ],
    "error_buckets": {
      "numeric_mismatch_count": 23,
      "punctuation_only_count": 45,
      "spacing_tokenization_count": 12,
      "entity_mismatch_count": 8,
      "script_confusion_count": 0,
      "empty_hypothesis_count": 0
    },
    "examples": {
      "worst_samples": ["as_0047", "as_0231", "as_0419"],
      "best_samples": ["as_0001", "as_0155", "as_0302"],
      "numeric_mismatch_samples": ["as_0089", "as_0201"],
      "entity_mismatch_samples": ["as_0134"]
    }
  },
  // ... one key per language ...

  "__summary__": {
    "model_diagnosis": "recognition-limited",
    "primary_error_source": "recognition",
    "numeric_verbalization_impact": "moderate",
    "formatting_impact": "low",
    "worst_languages": ["assamese", "malayalam", "kannada"],
    "best_languages": ["hindi", "english", "gujarati"]
  }
}
```

### `__summary__.model_diagnosis` values

| Value                 | Meaning                                           |
|-----------------------|---------------------------------------------------|
| `recognition-limited` | Core ASR errors dominate, normalization helps little |
| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
| `numeric-limited`     | Number verbalization is the primary error source   |
| `mixed`               | No single dominant error category                  |

### Required Fields

| Section        | Required                                                                    |
|----------------|-----------------------------------------------------------------------------|
| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

---

## Agent Instructions

### Output directory

```
/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
```

### Files to generate per checkpoint

| File                   | Required | Content                     |
|------------------------|----------|-----------------------------|
| `metrics.json`         | Yes      | All four metric tiers       |
| `sample_analysis.json` | Yes      | Per-sample predictions      |
| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

### Critical Rules

1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
4. All four metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`) are mandatory per language and in aggregates.
5. `normalization_delta` must be computed as `wer_raw - wer_norm` and `wer_norm - wer_numcanon`.
6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

### Model-Specific Notes

| Model            | model_id        | Checkpoint paths                                              |
|------------------|-----------------|---------------------------------------------------------------|
| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

### Metric Roles

| Role                       | Metric         | Use for                                              |
|----------------------------|----------------|------------------------------------------------------|
| Primary research metric    | `wer_norm`     | Model comparison, paper reporting, dashboard default  |
| Strict production metric   | `wer_raw`      | Transcript fidelity, formatting quality               |
| Numeric robustness metric  | `wer_numcanon` | Isolating number verbalization errors                 |
| Script-sensitive metric    | `cer_norm`     | Indic script evaluation, character-level quality      |

---

## Dashboard Integration

### nginx endpoint (port 80 on 216.81.248.184)

```
GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
GET /api/benchmarks/<model_id>/                        → checkpoint listing
GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
```

### Next.js API routes (on Vercel)

```
GET /api/models                                → aggregated model + checkpoint list
GET /api/metrics?model=X&checkpoint=Y          → metrics.json
GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
GET /api/compare                               → cross-model WER comparison
```

Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

---

## Migration from Old Schema

Old schema fields map to new as follows:

| Old Field          | New Field        |
|--------------------|------------------|
| `wer`              | `wer_raw`        |
| `cer`              | (dropped — use `cer_norm`) |
| `wer_normalized`   | `wer_norm`       |
| `cer_normalized`   | `cer_norm`       |
| (did not exist)    | `wer_numcanon`   |
| (did not exist)    | `normalization_delta` |
| (did not exist)    | `error_analysis.json` |

Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.


The file /home/ubuntu/training/benchmark_schema/BENCHMARK_SCHEMA.md has been updated successfully.

# ASR Benchmark Schema

Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

---

## Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  <model_id>/
    <checkpoint_name>/
      metrics.json
      sample_analysis.json
      error_analysis.json
```

## Model Registry

| Model ID         | Display Name                    | Checkpoints              |
|------------------|---------------------------------|--------------------------|
| `qwen3-asr`     | Qwen3-ASR-1.7B                 | ckpt-24000, 72000, 100000|
| `gemma3n-e2b`   | Gemma-3n-E2B-ASR               | ckpt-10000, 20000        |
| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

## Languages

| Language   | Code | Source                  | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Metric Tiers

Four metric tiers are computed. Each tier adds one normalization step on top of the previous.

| Metric         | Unicode | Whitespace | Punctuation Removal | Case Fold | Number Canon | Purpose                        |
|----------------|---------|------------|---------------------|-----------|--------------|--------------------------------|
| `wer_raw`      | NFC     | trim only  | No                  | No        | No           | Strict transcript fidelity     |
| `wer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**         |
| `wer_numcanon` | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness             |
| `cer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic       |

*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

### Normalization Pipeline (applied in order)

```
Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon)
Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)
Step 3: Normalize whitespace (collapse runs, trim)
Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)
Step 5: Remove punctuation (norm/numcanon only, using language-aware set — preserve Devanagari danda for segmentation)
Step 6: Case fold (English only)
Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)
```

---

## Schema: `metrics.json`

```jsonc
{
  "<language_name>": {
    "n_samples": 500,              // int
    "wer_raw": 42.10,             // float: % — minimal normalization
    "wer_norm": 34.07,            // float: % — primary metric
    "wer_numcanon": 32.50,        // float: % — with number canonicalization
    "cer_norm": 12.30,            // float: % — character-level normalized
    "empty_hypotheses": 0,        // int: samples where model produced no output
    "normalization_delta": {
      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm (negative = norm helped)
      "norm_to_numcanon": -1.57   // float: wer_norm - wer_numcanon
    }
  },
  // ... one key per language (12 total) ...

  "__overall__": {
    "n_samples": 6000,
    "wer_raw": 40.75,
    "wer_norm": 32.07,
    "wer_numcanon": 30.80,
    "cer_norm": 11.34
  },

  "__macro_avg__": {
    "n_languages": 12,
    "wer_raw": 40.52,
    "wer_norm": 31.76,
    "wer_numcanon": 30.20,
    "cer_norm": 11.03
  },

  "__meta__": {
    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",
    "checkpoint_name": "ckpt-20000",
    "model_id": "gemma3n-e2b",
    "model_type": "gemma3n-E2B-asr",
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",
    "batch_size": 128,
    "inference_time_sec": 723.7,
    "total_audio_sec": 40354.46,
    "rtf": 0.0179,
    "timestamp": "2026-03-26T14:30:00Z",
    "gpu": "NVIDIA A100 80GB",
    "framework": "transformers",
    "normalization_version": "v1",
    "jiwer_version": "3.1.0"
  }
}
```

### Required Fields

| Section          | Required                                                                                |
|------------------|--------------------------------------------------------------------------------------------------|
| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |
| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                  |
| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                |
| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

---

## Schema: `sample_analysis.json`

Array of per-sample objects. One entry per sample (6000 total).

```jsonc
[
  {
    "id": "as_0000",
    "language": "assamese",
    "reference": "<ground truth>",
    "hypothesis": "<model output>",
    "ref_norm": "<after norm pipeline>",
    "hyp_norm": "<after norm pipeline>",
    "ref_numcanon": "<after numcanon pipeline>",
    "hyp_numcanon": "<after numcanon pipeline>",
    "detected_language": "Assamese",
    "wer_raw": 45.0,
    "wer_norm": 30.0,
    "flags": ["numeric_mismatch", "punctuation_only_diff"]
  }
]
```

### Per-sample fields

| Field              | Type     | Required | Description                                              |
|--------------------|----------|----------|----------------------------------------------------------|
| `id`               | string   | Yes      | Unique sample ID                                         |
| `language`         | string   | Yes      | Language name, lowercase                                 |
| `reference`        | string   | Yes      | Ground truth transcription (original)                    |
| `hypothesis`       | string   | Yes      | Model prediction (original)                              |
| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |
| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |
| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |
| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |
| `detected_language`| string   | No       | Language detected by model                               |
| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |
| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |
| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

### Flag Vocabulary

Agents SHOULD tag samples with applicable flags:

| Flag                    | Meaning                                                      |
|-------------------------|--------------------------------------------------------------|
| `exact_match`           | reference == hypothesis (raw)                                |
| `exact_match_norm`      | ref_norm == hyp_norm                                         |
| `numeric_mismatch`      | Difference involves digits or number words                   |
| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |
| `empty_hypothesis`      | Model produced no output                                     |
| `script_mismatch`       | Hypothesis uses a different script than reference            |
| `lang_confusion`        | detected_language != expected language                        |
| `high_wer`              | wer_norm > 80%                                               |
| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |

---

## Schema: `error_analysis.json`

Per-language error breakdown. One file per checkpoint.

```jsonc
{
  "<language_name>": {
    "top_substitutions": [
      {"ref": "word_a", "hyp": "word_b", "count": 42},
      // top 20 by count
    ],
    "top_insertions": [
      {"word": "um", "count": 15},
      // top 20
    ],
    "top_deletions": [
      {"word": "the", "count": 30},
      // top 20
    ],
    "error_buckets": {
      "numeric_mismatch_count": 23,
      "punctuation_only_count": 45,
      "spacing_tokenization_count": 12,
      "entity_mismatch_count": 8,
      "script_confusion_count": 0,
      "empty_hypothesis_count": 0
    },
    "examples": {
      "worst_samples": ["as_0047", "as_0231", "as_0419"],
      "best_samples": ["as_0001", "as_0155", "as_0302"],
      "numeric_mismatch_samples": ["as_0089", "as_0201"],
      "entity_mismatch_samples": ["as_0134"]
    }
  },
  // ... one key per language ...

  "__summary__": {
    "model_diagnosis": "recognition-limited",
    "primary_error_source": "recognition",
    "numeric_verbalization_impact": "moderate",
    "formatting_impact": "low",
    "worst_languages": ["assamese", "malayalam", "kannada"],
    "best_languages": ["hindi", "english", "gujarati"]
  }
}
```

### `__summary__.model_diagnosis` values

| Value                 | Meaning                                           |
|-----------------------|---------------------------------------------------|
| `recognition-limited` | Core ASR errors dominate, normalization helps little |
| `formatting-limited`  | Most errors are punctuation/casing/spacing         |
| `numeric-limited`     | Number verbalization is the primary error source   |
| `mixed`               | No single dominant error category                  |

### Required Fields

| Section        | Required                                                                    |
|----------------|-----------------------------------------------------------------------------|
| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |
| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

---

## Agent Instructions

### Output directory

```
/home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
```

### Files to generate per checkpoint

| File                   | Required | Content                     |
|------------------------|----------|-----------------------------|
| `metrics.json`         | Yes      | All four metric tiers       |
| `sample_analysis.json` | Yes      | Per-sample predictions      |
| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

### Critical Rules

1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.
2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.
3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.
4. All four metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`) are mandatory per language and in aggregates.
5. `normalization_delta` must be computed as `wer_raw - wer_norm` and `wer_norm - wer_numcanon`.
6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.
7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

### Model-Specific Notes

| Model            | model_id        | Checkpoint paths                                              |
|------------------|-----------------|---------------------------------------------------------------|
| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |
| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |
| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

### Metric Roles

| Role                       | Metric         | Use for                                              |
|----------------------------|----------------|------------------------------------------------------|
| Primary research metric    | `wer_norm`     | Model comparison, paper reporting, dashboard default  |
| Strict production metric   | `wer_raw`      | Transcript fidelity, formatting quality               |
| Numeric robustness metric  | `wer_numcanon` | Isolating number verbalization errors                 |
| Script-sensitive metric    | `cer_norm`     | Indic script evaluation, character-level quality      |

---

## Dashboard Integration

### nginx endpoint (port 80 on 216.81.248.184)

```
GET /api/benchmarks/                                   → model directory listing (JSON autoindex)
GET /api/benchmarks/<model_id>/                        → checkpoint listing
GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file
GET /api/benchmarks/<model_id>/<ckpt>/sample_analysis.json
GET /api/benchmarks/<model_id>/<ckpt>/error_analysis.json
```

### Next.js API routes (on Vercel)

```
GET /api/models                                → aggregated model + checkpoint list
GET /api/metrics?model=X&checkpoint=Y          → metrics.json
GET /api/samples?model=X&checkpoint=Y&language=Z&page=N&pageSize=N
GET /api/errors?model=X&checkpoint=Y           → error_analysis.json
GET /api/compare                               → cross-model WER comparison
```

Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

---

## Migration from Old Schema

Old schema fields map to new as follows:

| Old Field          | New Field        |
|--------------------|------------------|
| `wer`              | `wer_raw`        |
| `cer`              | (dropped — use `cer_norm`) |
| `wer_normalized`   | `wer_norm`       |
| `cer_normalized`   | `cer_norm`       |
| (did not exist)    | `wer_numcanon`   |
| (did not exist)    | `normalization_delta` |
| (did not exist)    | `error_analysis.json` |

Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.

-# ASR Benchmark Schema & Agent Instructions

+Canonical JSON schema for all ASR benchmark outputs. Every agent MUST conform to this schema.

-This document defines the **canonical JSON schema** for all ASR benchmark outputs across all models. Every agent producing benchmark results MUST conform to this schema so the dashboard can ingest results uniformly.

-| Model ID              | Display Name                       | Has Checkpoints | Notes                     |

-| `parakeet-1.1b`      | Parakeet-1.1B-Language-Guided     | No (yet)        | Use `baseline` as folder  |

+| Model ID         | Display Name                    | Checkpoints              |

+| `parakeet-1.1b` | Parakeet-1.1B-Language-Guided  | baseline (placeholder)   |

-| Language   | Code | Source Dataset          | Samples |

+| Language   | Code | Source                  | Samples |

| assamese   | as   | IndicVoices (valid)     | 500     |

| bengali    | bn   | Kathbath (test_known)   | 500     |

+Four metric tiers are computed. Each tier adds one normalization step on top of the previous.

+| Metric         | Unicode | Whitespace | Punctuation Removal | Case Fold | Number Canon | Purpose                        |

+| `wer_raw`      | NFC     | trim only  | No                  | No        | No           | Strict transcript fidelity     |

+| `wer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | **Primary ASR metric**         |

+| `wer_numcanon` | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | Yes          | Numeric robustness             |

+| `cer_norm`     | NFKC    | normalize  | Yes (lang-aware)    | Yes*      | No           | Script-sensitive / Indic       |

+*Case fold only for scripts with case (Latin). Indic scripts are unaffected.

+### Normalization Pipeline (applied in order)

+Step 1: Unicode normalize (NFC for raw, NFKC for norm/numcanon)

+Step 2: Strip zero-width joiners/non-joiners (U+200B..U+200F, U+FEFF)

+Step 3: Normalize whitespace (collapse runs, trim)

+Step 4: Standardize punctuation variants (curly quotes → straight, em-dash → hyphen, double-danda → single)

+Step 5: Remove punctuation (norm/numcanon only, using language-aware set — preserve Devanagari danda for segmentation)

+Step 7: Number canonicalization (numcanon only — digit grouping removal, Arabic numeral unification)

-  // Per-language metrics (one key per language, lowercase English name)

-    "n_samples": 500,            // int: number of samples evaluated

-    "wer": 38.01,                // float: Word Error Rate (%)

-    "cer": 19.68,                // float: Character Error Rate (%)

-    "wer_normalized": 34.07,     // float: WER after text normalization (%)

-    "cer_normalized": 19.49,     // float: CER after text normalization (%)

-    "empty_hypotheses": 0        // int: count of empty model outputs

+    "n_samples": 500,              // int

+    "wer_raw": 42.10,             // float: % — minimal normalization

+    "wer_norm": 34.07,            // float: % — primary metric

+    "wer_numcanon": 32.50,        // float: % — with number canonicalization

+    "cer_norm": 12.30,            // float: % — character-level normalized

+    "empty_hypotheses": 0,        // int: samples where model produced no output

+      "raw_to_norm": -8.03,       // float: wer_raw - wer_norm (negative = norm helped)

+      "norm_to_numcanon": -1.57   // float: wer_norm - wer_numcanon

-  // ... repeat for all 12 languages ...

+  // ... one key per language (12 total) ...

-    "n_samples": 6000,           // int: total samples (should be 6000)

-    "wer": 35.16,                // float: micro-averaged WER

-    "cer": 15.53,                // float: micro-averaged CER

-    "wer_normalized": 29.97,     // float: micro-averaged normalized WER

-    "cer_normalized": 14.48      // float: micro-averaged normalized CER

-    "n_languages": 12,           // int: number of languages

-    "wer": 35.32,                // float: macro-averaged WER

-    "cer": 15.49,                // float: macro-averaged CER

-    "wer_normalized": 30.08,     // float: macro-averaged normalized WER

-    "cer_normalized": 14.37      // float: macro-averaged normalized CER

-    "checkpoint": "<full_path>",         // str: full path to checkpoint

-    "checkpoint_name": "ckpt-20000",     // str: short checkpoint name

-    "model_id": "gemma3n-e2b",           // str: model ID from registry above

-    "model_type": "gemma3n-E2B-asr",     // str: model architecture type

-    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",  // str: HF dataset name

-    "batch_size": 128,                   // int: inference batch size

-    "inference_time_sec": 723.7,         // float: total inference wall time

-    "total_audio_sec": 40354.46,         // float: total audio duration

-    "rtf": 0.0179,                       // float: Real-Time Factor

-    "gpu": "NVIDIA A100 80GB",           // str: GPU used (optional)

-    "framework": "transformers"          // str: inference framework (optional)

+    "checkpoint": "/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-20000",

+    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",

-**Per-language**: `n_samples`, `wer`, `cer`, `wer_normalized`, `cer_normalized`, `empty_hypotheses`

-**__overall__**: `n_samples`, `wer`, `cer`, `wer_normalized`, `cer_normalized`

-**__macro_avg__**: `n_languages`, `wer`, `cer`, `wer_normalized`, `cer_normalized`

-**__meta__**: `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`

+| Section          | Required                                                                                |

+| Per-language     | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`, `empty_hypotheses`, `normalization_delta` |

+| `__overall__`    | `n_samples`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                  |

+| `__macro_avg__`  | `n_languages`, `wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`                                |

+| `__meta__`       | `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`, `normalization_version` |

-An array of per-sample prediction objects:

+Array of per-sample objects. One entry per sample (6000 total).

-    "id": "as_0000",                    // str: unique sample ID

-    "language": "assamese",             // str: language name (lowercase)

-    "reference": "<ground truth text>", // str: reference transcription

-    "hypothesis": "<model output>",     // str: model prediction

-    "detected_language": "Assamese",    // str: language detected by model (optional)

-    "ref_normalized": "<normalized>",   // str: normalized reference (optional)

-    "hyp_normalized": "<normalized>"    // str: normalized hypothesis (optional)

+    "ref_norm": "<after norm pipeline>",

+    "hyp_norm": "<after norm pipeline>",

+    "ref_numcanon": "<after numcanon pipeline>",

+    "hyp_numcanon": "<after numcanon pipeline>",

+    "flags": ["numeric_mismatch", "punctuation_only_diff"]

-  // ... one entry per sample (6000 total) ...

+| Field              | Type     | Required | Description                                              |

+| `id`               | string   | Yes      | Unique sample ID                                         |

+| `language`         | string   | Yes      | Language name, lowercase                                 |

+| `reference`        | string   | Yes      | Ground truth transcription (original)                    |

+| `hypothesis`       | string   | Yes      | Model prediction (original)                              |

+| `ref_norm`         | string   | Yes      | Reference after norm pipeline                            |

+| `hyp_norm`         | string   | Yes      | Hypothesis after norm pipeline                           |

+| `ref_numcanon`     | string   | No       | Reference after numcanon pipeline                        |

+| `hyp_numcanon`     | string   | No       | Hypothesis after numcanon pipeline                       |

+| `detected_language`| string   | No       | Language detected by model                               |

+| `wer_raw`          | float    | No       | Per-sample WER raw (%)                                   |

+| `wer_norm`         | float    | No       | Per-sample WER normalized (%)                            |

+| `flags`            | string[] | No       | Diagnostic tags (see flag vocabulary below)              |

+Agents SHOULD tag samples with applicable flags:

+| Flag                    | Meaning                                                      |

+| `exact_match`           | reference == hypothesis (raw)                                |

+| `exact_match_norm`      | ref_norm == hyp_norm                                         |

+| `numeric_mismatch`      | Difference involves digits or number words                   |

+| `punctuation_only_diff` | Raw differs but norm matches — punctuation was the only gap  |

+| `empty_hypothesis`      | Model produced no output                                     |

+| `script_mismatch`       | Hypothesis uses a different script than reference            |

+| `lang_confusion`        | detected_language != expected language                        |

+| `high_wer`              | wer_norm > 80%                                               |

+| `entity_mismatch`       | Named entity (proper noun) was misrecognized                 |

-### For ALL agents (Qwen, Gemma, Parakeet)

+Per-language error breakdown. One file per checkpoint.

-1. **Run benchmarks** on the full `BayAreaBoys/indic-asr-benchmark-6k` dataset (6000 samples, 12 languages, 500 each).

+      "spacing_tokenization_count": 12,

+      "numeric_mismatch_samples": ["as_0089", "as_0201"],

+      "entity_mismatch_samples": ["as_0134"]

-2. **Output directory**: Place results in:

-   /home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/

-   - `model_id`: One of `qwen3-asr`, `gemma3n-e2b`, `parakeet-1.1b`

-   - `checkpoint_name`: e.g. `ckpt-24000`, `ckpt-72000`, `baseline`

+    "model_diagnosis": "recognition-limited",

+    "primary_error_source": "recognition",

+    "numeric_verbalization_impact": "moderate",

+    "worst_languages": ["assamese", "malayalam", "kannada"],

+    "best_languages": ["hindi", "english", "gujarati"]

-3. **Generate two files** per checkpoint:

-   - `metrics.json` — conforming to the metrics schema above

-   - `sample_analysis.json` — conforming to the sample analysis schema above

-4. **CRITICAL**: The `__meta__.model_id` field MUST match the directory model_id exactly.

+| Value                 | Meaning                                           |

+| `recognition-limited` | Core ASR errors dominate, normalization helps little |

+| `formatting-limited`  | Most errors are punctuation/casing/spacing         |

+| `numeric-limited`     | Number verbalization is the primary error source   |

+| `mixed`               | No single dominant error category                  |

-5. **CRITICAL**: The `__meta__.timestamp` MUST be set to the ISO 8601 UTC time when the benchmark run completed.

-6. **Text normalization**: Apply Unicode NFKD normalization, lowercase, strip punctuation for `*_normalized` fields. Keep raw WER/CER computed on original text.

+| Section        | Required                                                                    |

+| Per-language   | `top_substitutions` (>=10), `top_insertions` (>=10), `top_deletions` (>=10), `error_buckets` |

+| `__summary__`  | `model_diagnosis`, `primary_error_source`, `worst_languages`, `best_languages` |

-   - WER/CER: Use `jiwer` library or equivalent

-   - `__overall__`: Micro-average (concatenate all samples, compute once)

-   - `__macro_avg__`: Average per-language metrics

+| File                   | Required | Content                     |

+| `metrics.json`         | Yes      | All four metric tiers       |

+| `sample_analysis.json` | Yes      | Per-sample predictions      |

+| `error_analysis.json`  | Yes      | Error breakdown + diagnosis |

+1. `__meta__.model_id` MUST match the directory `<model_id>` exactly.

+2. `__meta__.timestamp` MUST be ISO 8601 UTC of when the run completed.

+3. `__meta__.normalization_version` MUST be `"v1"` for this schema version.

+4. All four metric tiers (`wer_raw`, `wer_norm`, `wer_numcanon`, `cer_norm`) are mandatory per language and in aggregates.

+5. `normalization_delta` must be computed as `wer_raw - wer_norm` and `wer_norm - wer_numcanon`.

+6. Use `jiwer` for WER/CER. `__overall__` = micro-average. `__macro_avg__` = mean of per-language values.

+7. Save intermediate normalized texts in `sample_analysis.json` (`ref_norm`, `hyp_norm`) so results are auditable.

-- Checkpoint paths: `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`

-- Set `model_id: "qwen3-asr"` in `__meta__`

+| Model            | model_id        | Checkpoint paths                                              |

+| Qwen3-ASR        | `qwen3-asr`     | `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`       |

+| Gemma3n-E2B      | `gemma3n-e2b`   | `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`     |

+| Parakeet-1.1B    | `parakeet-1.1b` | TBD — use `baseline` as checkpoint_name                       |

-- Checkpoint paths: `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`

-- Set `model_id: "gemma3n-e2b"` in `__meta__`

-- No checkpoints yet — use `baseline` as checkpoint_name

-- Set `model_id: "parakeet-1.1b"` in `__meta__`

-- When checkpoints become available, follow same naming convention

+| Role                       | Metric         | Use for                                              |

+| Primary research metric    | `wer_norm`     | Model comparison, paper reporting, dashboard default  |

+| Strict production metric   | `wer_raw`      | Transcript fidelity, formatting quality               |

+| Numeric robustness metric  | `wer_numcanon` | Isolating number verbalization errors                 |

+| Script-sensitive metric    | `cer_norm`     | Indic script evaluation, character-level quality      |

-[Benchmark Scripts] --> metrics.json + sample_analysis.json

-              /home/ubuntu/training/benchmark_outputs/

-              [REST API / Static file serve]

-                    [Next.js Dashboard]

+GET /api/benchmarks/                                   → model directory listing (JSON autoindex)

+GET /api/benchmarks/<model_id>/                        → checkpoint listing

+GET /api/benchmarks/<model_id>/<ckpt>/metrics.json     → metrics file

-The dashboard reads from `/home/ubuntu/training/benchmark_outputs/` via a local API route. Agents place files there; the dashboard picks them up automatically.

-### API Endpoints (served by Next.js API routes)

-GET /api/models                           → list of models with checkpoint info

-GET /api/models/:modelId/checkpoints      → list checkpoints for a model

-GET /api/metrics/:modelId/:checkpoint     → metrics.json content

-GET /api/samples/:modelId/:checkpoint     → sample_analysis.json content

-GET /api/compare?models=...&checkpoints=... → cross-model/checkpoint comparison

+GET /api/models                                → aggregated model + checkpoint list

+GET /api/metrics?model=X&checkpoint=Y          → metrics.json

+GET /api/errors?model=X&checkpoint=Y           → error_analysis.json

+GET /api/compare                               → cross-model WER comparison

+Agents place files in `benchmark_outputs/`. Dashboard picks them up on next request. No restart or redeploy needed.

-Existing results in `benchmark_results/` and `benchmark_results_gemma3n/` should be copied to the new canonical location. The dashboard migration script handles this automatically.

+Old schema fields map to new as follows:

+| Old Field          | New Field        |

+| `wer`              | `wer_raw`        |

+| `cer`              | (dropped — use `cer_norm`) |

+| `wer_normalized`   | `wer_norm`       |

+| `cer_normalized`   | `cer_norm`       |

+| (did not exist)    | `wer_numcanon`   |

+| (did not exist)    | `normalization_delta` |

+| (did not exist)    | `error_analysis.json` |

+Agents must regenerate results using the new schema. Old `benchmark_results/` and `benchmark_results_gemma3n/` directories are retained for reference but are NOT consumed by the dashboard.

# ASR Benchmark Schema & Agent Instructions

## Overview

This document defines the **canonical JSON schema** for all ASR benchmark outputs across all models. Every agent producing benchmark results MUST conform to this schema so the dashboard can ingest results uniformly.

---

## Models & Directory Layout

```
/home/ubuntu/training/benchmark_outputs/
  qwen3-asr/
    ckpt-24000/
      metrics.json
      sample_analysis.json
    ckpt-72000/
      metrics.json
      sample_analysis.json
    ckpt-100000/
      metrics.json
      sample_analysis.json
  gemma3n-e2b/
    ckpt-10000/
      metrics.json
      sample_analysis.json
    ckpt-20000/
      metrics.json
      sample_analysis.json
  parakeet-1.1b/
    baseline/
      metrics.json
      sample_analysis.json
```

### Model Registry

| Model ID              | Display Name                       | Has Checkpoints | Notes                     |
|-----------------------|------------------------------------|-----------------|---------------------------|
| `qwen3-asr`          | Qwen3-ASR-1.7B                    | Yes             | ckpt-24000, 72000, 100000 |
| `gemma3n-e2b`        | Gemma-3n-E2B-ASR                  | Yes             | ckpt-10000, 20000         |
| `parakeet-1.1b`      | Parakeet-1.1B-Language-Guided     | No (yet)        | Use `baseline` as folder  |

### Languages (12 Indic + English)

| Language   | Code | Source Dataset          | Samples |
|------------|------|-------------------------|---------|
| assamese   | as   | IndicVoices (valid)     | 500     |
| bengali    | bn   | Kathbath (test_known)   | 500     |
| english    | en   | Svarah                  | 500     |
| gujarati   | gu   | Kathbath (test_known)   | 500     |
| hindi      | hi   | Kathbath (test_known)   | 500     |
| kannada    | kn   | Kathbath (test_known)   | 500     |
| malayalam  | ml   | Kathbath (test_known)   | 500     |
| marathi    | mr   | Kathbath (test_known)   | 500     |
| odia       | or   | Kathbath (test_known)   | 500     |
| punjabi    | pa   | Kathbath (test_known)   | 500     |
| tamil      | ta   | Kathbath (test_known)   | 500     |
| telugu     | te   | Kathbath (test_known)   | 500     |

---

## Schema: `metrics.json`

```jsonc
{
  // Per-language metrics (one key per language, lowercase English name)
  "<language_name>": {
    "n_samples": 500,            // int: number of samples evaluated
    "wer": 38.01,                // float: Word Error Rate (%)
    "cer": 19.68,                // float: Character Error Rate (%)
    "wer_normalized": 34.07,     // float: WER after text normalization (%)
    "cer_normalized": 19.49,     // float: CER after text normalization (%)
    "empty_hypotheses": 0        // int: count of empty model outputs
  },
  // ... repeat for all 12 languages ...

  // Aggregate metrics
  "__overall__": {
    "n_samples": 6000,           // int: total samples (should be 6000)
    "wer": 35.16,                // float: micro-averaged WER
    "cer": 15.53,                // float: micro-averaged CER
    "wer_normalized": 29.97,     // float: micro-averaged normalized WER
    "cer_normalized": 14.48      // float: micro-averaged normalized CER
  },

  "__macro_avg__": {
    "n_languages": 12,           // int: number of languages
    "wer": 35.32,                // float: macro-averaged WER
    "cer": 15.49,                // float: macro-averaged CER
    "wer_normalized": 30.08,     // float: macro-averaged normalized WER
    "cer_normalized": 14.37      // float: macro-averaged normalized CER
  },

  // Run metadata
  "__meta__": {
    "checkpoint": "<full_path>",         // str: full path to checkpoint
    "checkpoint_name": "ckpt-20000",     // str: short checkpoint name
    "model_id": "gemma3n-e2b",           // str: model ID from registry above
    "model_type": "gemma3n-E2B-asr",     // str: model architecture type
    "dataset": "BayAreaBoys/indic-asr-benchmark-6k",  // str: HF dataset name
    "batch_size": 128,                   // int: inference batch size
    "inference_time_sec": 723.7,         // float: total inference wall time
    "total_audio_sec": 40354.46,         // float: total audio duration
    "rtf": 0.0179,                       // float: Real-Time Factor
    "timestamp": "2026-03-26T14:30:00Z", // str: ISO 8601 timestamp of run
    "gpu": "NVIDIA A100 80GB",           // str: GPU used (optional)
    "framework": "transformers"          // str: inference framework (optional)
  }
}
```

### Required Fields

**Per-language**: `n_samples`, `wer`, `cer`, `wer_normalized`, `cer_normalized`, `empty_hypotheses`
**__overall__**: `n_samples`, `wer`, `cer`, `wer_normalized`, `cer_normalized`
**__macro_avg__**: `n_languages`, `wer`, `cer`, `wer_normalized`, `cer_normalized`
**__meta__**: `checkpoint_name`, `model_id`, `dataset`, `inference_time_sec`, `total_audio_sec`, `rtf`, `timestamp`

---

## Schema: `sample_analysis.json`

An array of per-sample prediction objects:

```jsonc
[
  {
    "id": "as_0000",                    // str: unique sample ID
    "language": "assamese",             // str: language name (lowercase)
    "reference": "<ground truth text>", // str: reference transcription
    "hypothesis": "<model output>",     // str: model prediction
    "detected_language": "Assamese",    // str: language detected by model (optional)
    "ref_normalized": "<normalized>",   // str: normalized reference (optional)
    "hyp_normalized": "<normalized>"    // str: normalized hypothesis (optional)
  }
  // ... one entry per sample (6000 total) ...
]
```

---

## Agent Instructions

### For ALL agents (Qwen, Gemma, Parakeet)

1. **Run benchmarks** on the full `BayAreaBoys/indic-asr-benchmark-6k` dataset (6000 samples, 12 languages, 500 each).

2. **Output directory**: Place results in:
   ```
   /home/ubuntu/training/benchmark_outputs/<model_id>/<checkpoint_name>/
   ```
   - `model_id`: One of `qwen3-asr`, `gemma3n-e2b`, `parakeet-1.1b`
   - `checkpoint_name`: e.g. `ckpt-24000`, `ckpt-72000`, `baseline`

3. **Generate two files** per checkpoint:
   - `metrics.json` — conforming to the metrics schema above
   - `sample_analysis.json` — conforming to the sample analysis schema above

4. **CRITICAL**: The `__meta__.model_id` field MUST match the directory model_id exactly.

5. **CRITICAL**: The `__meta__.timestamp` MUST be set to the ISO 8601 UTC time when the benchmark run completed.

6. **Text normalization**: Apply Unicode NFKD normalization, lowercase, strip punctuation for `*_normalized` fields. Keep raw WER/CER computed on original text.

7. **Metrics computation**:
   - WER/CER: Use `jiwer` library or equivalent
   - `__overall__`: Micro-average (concatenate all samples, compute once)
   - `__macro_avg__`: Average per-language metrics

### Model-Specific Notes

#### Qwen3-ASR Agent
- Checkpoints: `ckpt-24000`, `ckpt-72000`, `ckpt-100000`
- Checkpoint paths: `/home/ubuntu/training/checkpoints/qwen3-asr-ckpt-{N}`
- Supports batch_size=32 with vLLM
- Set `model_id: "qwen3-asr"` in `__meta__`

#### Gemma3n-E2B Agent
- Checkpoints: `ckpt-10000`, `ckpt-20000`
- Checkpoint paths: `/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-{N}`
- Set `model_id: "gemma3n-e2b"` in `__meta__`

#### Parakeet-1.1B Agent
- No checkpoints yet — use `baseline` as checkpoint_name
- Set `model_id: "parakeet-1.1b"` in `__meta__`
- When checkpoints become available, follow same naming convention

---

## Dashboard Data Flow

```
[Benchmark Scripts] --> metrics.json + sample_analysis.json
                            |
                            v
              /home/ubuntu/training/benchmark_outputs/
                            |
                            v
              [REST API / Static file serve]
                            |
                            v
                    [Next.js Dashboard]
```

The dashboard reads from `/home/ubuntu/training/benchmark_outputs/` via a local API route. Agents place files there; the dashboard picks them up automatically.

### API Endpoints (served by Next.js API routes)

```
GET /api/models                           → list of models with checkpoint info
GET /api/models/:modelId/checkpoints      → list checkpoints for a model
GET /api/metrics/:modelId/:checkpoint     → metrics.json content
GET /api/samples/:modelId/:checkpoint     → sample_analysis.json content
GET /api/compare?models=...&checkpoints=... → cross-model/checkpoint comparison
```

---

## Migrating Existing Results

Existing results in `benchmark_results/` and `benchmark_results_gemma3n/` should be copied to the new canonical location. The dashboard migration script handles this automatically.


Now update the dashboard types and API to support the new schema while remaining backward-compatible with existing data until agents regenerate.


// === Metric tiers ===
// wer_raw:      minimal normalization, strict fidelity
// wer_norm:     PRIMARY metric — NFKC + whitespace + punctuation + case fold
// wer_numcanon: wer_norm + number canonicalization
// cer_norm:     character-level normalized — critical for Indic scripts

export interface NormalizationDelta {
  raw_to_norm: number;       // wer_raw - wer_norm (negative = norm helped)
  norm_to_numcanon: number;  // wer_norm - wer_numcanon
}

export interface LanguageMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
  empty_hypotheses: number;
  normalization_delta: NormalizationDelta;
}

export interface AggregateMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MacroAvgMetrics {
  n_languages: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MetaMeta {
  checkpoint: string;
  checkpoint_name: string;
  model_id: string;
  model_type?: string;
  dataset: string;
  batch_size?: number;
  force_language?: boolean;
  inference_time_sec: number;
  total_audio_sec: number;
  rtf: number;
  timestamp?: string;
  gpu?: string;
  framework?: string;
  normalization_version?: string;
  jiwer_version?: string;
}

export type MetricsFile = {
  [language: string]: LanguageMetrics | AggregateMetrics | MacroAvgMetrics | MetaMeta;
} & {
  __overall__: AggregateMetrics;
  __macro_avg__: MacroAvgMetrics;
  __meta__: MetaMeta;
};

export interface SampleAnalysis {
  id: string;
  language: string;
  reference: string;
  hypothesis: string;
  ref_norm?: string;
  hyp_norm?: string;
  ref_numcanon?: string;
  hyp_numcanon?: string;
  detected_language?: string;
  wer_raw?: number;
  wer_norm?: number;
  flags?: string[];
}

export interface ErrorBuckets {
  numeric_mismatch_count: number;
  punctuation_only_count: number;
  spacing_tokenization_count: number;
  entity_mismatch_count: number;
  script_confusion_count: number;
  empty_hypothesis_count: number;
}

export interface SubstitutionEntry {
  ref: string;
  hyp: string;
  count: number;
}

export interface WordCountEntry {
  word: string;
  count: number;
}

export interface LanguageErrorAnalysis {
  top_substitutions: SubstitutionEntry[];
  top_insertions: WordCountEntry[];
  top_deletions: WordCountEntry[];
  error_buckets: ErrorBuckets;
  examples: {
    worst_samples: string[];
    best_samples: string[];
    numeric_mismatch_samples: string[];
    entity_mismatch_samples: string[];
  };
}

export interface ErrorAnalysisSummary {
  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
  primary_error_source: string;
  numeric_verbalization_impact: string;
  formatting_impact: string;
  worst_languages: string[];
  best_languages: string[];
}

export type ErrorAnalysisFile = {
  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
} & {
  __summary__: ErrorAnalysisSummary;
};

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string;
}

// === Constants ===

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
};

export const LANGUAGES = [
  "assamese", "bengali", "english", "gujarati", "hindi", "kannada",
  "malayalam", "marathi", "odia", "punjabi", "tamil", "telugu",
] as const;

export const LANGUAGE_CODES: Record<string, string> = {
  assamese: "as", bengali: "bn", english: "en", gujarati: "gu",
  hindi: "hi", kannada: "kn", malayalam: "ml", marathi: "mr",
  odia: "or", punjabi: "pa", tamil: "ta", telugu: "te",
};

export const METRIC_TIERS = ["wer_raw", "wer_norm", "wer_numcanon", "cer_norm"] as const;
export type MetricTier = typeof METRIC_TIERS[number];

export const METRIC_LABELS: Record<MetricTier, string> = {
  wer_raw: "WER Raw",
  wer_norm: "WER Norm",
  wer_numcanon: "WER NumCanon",
  cer_norm: "CER Norm",
};

export const METRIC_DESCRIPTIONS: Record<MetricTier, string> = {
  wer_raw: "Strict transcript fidelity",
  wer_norm: "Primary ASR metric",
  wer_numcanon: "Numeric robustness",
  cer_norm: "Script-sensitive (Indic)",
};

export const CHART_COLORS = [
  "#6366f1", "#8b5cf6", "#a78bfa", "#c084fc",
  "#e879f9", "#f472b6", "#fb7185", "#f87171",
  "#fb923c", "#fbbf24", "#a3e635", "#34d399",
];

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};

// === Backward compatibility ===
// Converts old-schema metrics into new-schema shape so dashboard works
// with both old data (pre-migration) and new data.

export interface OldLanguageMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
  empty_hypotheses: number;
}

export function isOldSchema(obj: Record<string, unknown>): boolean {
  return "wer" in obj && !("wer_raw" in obj);
}

export function migrateLanguageMetrics(old: OldLanguageMetrics): LanguageMetrics {
  return {
    n_samples: old.n_samples,
    wer_raw: old.wer,
    wer_norm: old.wer_normalized,
    wer_numcanon: old.wer_normalized, // no numcanon data in old schema, use norm as fallback
    cer_norm: old.cer_normalized,
    empty_hypotheses: old.empty_hypotheses,
    normalization_delta: {
      raw_to_norm: old.wer - old.wer_normalized,
      norm_to_numcanon: 0,
    },
  };
}


The file /home/ubuntu/training/asr-benchmark-dashboard/src/lib/types.ts has been updated successfully.

// === Metric tiers ===
// wer_raw:      minimal normalization, strict fidelity
// wer_norm:     PRIMARY metric — NFKC + whitespace + punctuation + case fold
// wer_numcanon: wer_norm + number canonicalization
// cer_norm:     character-level normalized — critical for Indic scripts

export interface NormalizationDelta {
  raw_to_norm: number;       // wer_raw - wer_norm (negative = norm helped)
  norm_to_numcanon: number;  // wer_norm - wer_numcanon
}

export interface LanguageMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
  empty_hypotheses: number;
  normalization_delta: NormalizationDelta;
}

export interface AggregateMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MacroAvgMetrics {
  n_languages: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MetaMeta {
  checkpoint: string;
  checkpoint_name: string;
  model_id: string;
  model_type?: string;
  dataset: string;
  batch_size?: number;
  force_language?: boolean;
  inference_time_sec: number;
  total_audio_sec: number;
  rtf: number;
  timestamp?: string;
  gpu?: string;
  framework?: string;
  normalization_version?: string;
  jiwer_version?: string;
}

export type MetricsFile = {
  [language: string]: LanguageMetrics | AggregateMetrics | MacroAvgMetrics | MetaMeta;
} & {
  __overall__: AggregateMetrics;
  __macro_avg__: MacroAvgMetrics;
  __meta__: MetaMeta;
};

export interface SampleAnalysis {
  id: string;
  language: string;
  reference: string;
  hypothesis: string;
  ref_norm?: string;
  hyp_norm?: string;
  ref_numcanon?: string;
  hyp_numcanon?: string;
  detected_language?: string;
  wer_raw?: number;
  wer_norm?: number;
  flags?: string[];
}

export interface ErrorBuckets {
  numeric_mismatch_count: number;
  punctuation_only_count: number;
  spacing_tokenization_count: number;
  entity_mismatch_count: number;
  script_confusion_count: number;
  empty_hypothesis_count: number;
}

export interface SubstitutionEntry {
  ref: string;
  hyp: string;
  count: number;
}

export interface WordCountEntry {
  word: string;
  count: number;
}

export interface LanguageErrorAnalysis {
  top_substitutions: SubstitutionEntry[];
  top_insertions: WordCountEntry[];
  top_deletions: WordCountEntry[];
  error_buckets: ErrorBuckets;
  examples: {
    worst_samples: string[];
    best_samples: string[];
    numeric_mismatch_samples: string[];
    entity_mismatch_samples: string[];
  };
}

export interface ErrorAnalysisSummary {
  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
  primary_error_source: string;
  numeric_verbalization_impact: string;
  formatting_impact: string;
  worst_languages: string[];
  best_languages: string[];
}

export type ErrorAnalysisFile = {
  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
} & {
  __summary__: ErrorAnalysisSummary;
};

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string;
}

// === Constants ===

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
};

export const LANGUAGES = [
  "assamese", "bengali", "english", "gujarati", "hindi", "kannada",
  "malayalam", "marathi", "odia", "punjabi", "tamil", "telugu",
] as const;

export const LANGUAGE_CODES: Record<string, string> = {
  assamese: "as", bengali: "bn", english: "en", gujarati: "gu",
  hindi: "hi", kannada: "kn", malayalam: "ml", marathi: "mr",
  odia: "or", punjabi: "pa", tamil: "ta", telugu: "te",
};

export const METRIC_TIERS = ["wer_raw", "wer_norm", "wer_numcanon", "cer_norm"] as const;
export type MetricTier = typeof METRIC_TIERS[number];

export const METRIC_LABELS: Record<MetricTier, string> = {
  wer_raw: "WER Raw",
  wer_norm: "WER Norm",
  wer_numcanon: "WER NumCanon",
  cer_norm: "CER Norm",
};

export const METRIC_DESCRIPTIONS: Record<MetricTier, string> = {
  wer_raw: "Strict transcript fidelity",
  wer_norm: "Primary ASR metric",
  wer_numcanon: "Numeric robustness",
  cer_norm: "Script-sensitive (Indic)",
};

export const CHART_COLORS = [
  "#6366f1", "#8b5cf6", "#a78bfa", "#c084fc",
  "#e879f9", "#f472b6", "#fb7185", "#f87171",
  "#fb923c", "#fbbf24", "#a3e635", "#34d399",
];

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};

// === Backward compatibility ===
// Converts old-schema metrics into new-schema shape so dashboard works
// with both old data (pre-migration) and new data.

export interface OldLanguageMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
  empty_hypotheses: number;
}

export function isOldSchema(obj: Record<string, unknown>): boolean {
  return "wer" in obj && !("wer_raw" in obj);
}

export function migrateLanguageMetrics(old: OldLanguageMetrics): LanguageMetrics {
  return {
    n_samples: old.n_samples,
    wer_raw: old.wer,
    wer_norm: old.wer_normalized,
    wer_numcanon: old.wer_normalized, // no numcanon data in old schema, use norm as fallback
    cer_norm: old.cer_normalized,
    empty_hypotheses: old.empty_hypotheses,
    normalization_delta: {
      raw_to_norm: old.wer - old.wer_normalized,
      norm_to_numcanon: 0,
    },
  };
}

+// wer_raw:      minimal normalization, strict fidelity

+// wer_norm:     PRIMARY metric — NFKC + whitespace + punctuation + case fold

+// wer_numcanon: wer_norm + number canonicalization

+// cer_norm:     character-level normalized — critical for Indic scripts

+  raw_to_norm: number;       // wer_raw - wer_norm (negative = norm helped)

+  norm_to_numcanon: number;  // wer_norm - wer_numcanon

+  normalization_delta: NormalizationDelta;

-  [language: string]: LanguageMetrics | OverallMetrics | MacroAvgMetrics | MetaMeta;

+  [language: string]: LanguageMetrics | AggregateMetrics | MacroAvgMetrics | MetaMeta;

+export interface LanguageErrorAnalysis {

+  top_substitutions: SubstitutionEntry[];

+    numeric_mismatch_samples: string[];

+export interface ErrorAnalysisSummary {

+  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";

+  numeric_verbalization_impact: string;

+  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;

-  [key: string]: number | string; // checkpoint_name: wer value

+export const METRIC_TIERS = ["wer_raw", "wer_norm", "wer_numcanon", "cer_norm"] as const;

+export type MetricTier = typeof METRIC_TIERS[number];

+export const METRIC_LABELS: Record<MetricTier, string> = {

+export const METRIC_DESCRIPTIONS: Record<MetricTier, string> = {

+  wer_raw: "Strict transcript fidelity",

+  cer_norm: "Script-sensitive (Indic)",

+// Converts old-schema metrics into new-schema shape so dashboard works

+// with both old data (pre-migration) and new data.

+export function isOldSchema(obj: Record<string, unknown>): boolean {

+  return "wer" in obj && !("wer_raw" in obj);

+export function migrateLanguageMetrics(old: OldLanguageMetrics): LanguageMetrics {

+    wer_numcanon: old.wer_normalized, // no numcanon data in old schema, use norm as fallback

+    empty_hypotheses: old.empty_hypotheses,

+      raw_to_norm: old.wer - old.wer_normalized,

export interface LanguageMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
  empty_hypotheses: number;
}

export interface OverallMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
}

export interface MacroAvgMetrics {
  n_languages: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
}

export interface MetaMeta {
  checkpoint: string;
  checkpoint_name: string;
  model_id: string;
  model_type?: string;
  dataset: string;
  batch_size?: number;
  force_language?: boolean;
  inference_time_sec: number;
  total_audio_sec: number;
  rtf: number;
  timestamp?: string;
  gpu?: string;
  framework?: string;
}

export type MetricsFile = {
  [language: string]: LanguageMetrics | OverallMetrics | MacroAvgMetrics | MetaMeta;
} & {
  __overall__: OverallMetrics;
  __macro_avg__: MacroAvgMetrics;
  __meta__: MetaMeta;
};

export interface SampleAnalysis {
  id: string;
  language: string;
  reference: string;
  hypothesis: string;
  detected_language?: string;
  ref_normalized?: string;
  hyp_normalized?: string;
}

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string; // checkpoint_name: wer value
}

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
};

export const LANGUAGES = [
  "assamese", "bengali", "english", "gujarati", "hindi", "kannada",
  "malayalam", "marathi", "odia", "punjabi", "tamil", "telugu",
] as const;

export const LANGUAGE_CODES: Record<string, string> = {
  assamese: "as", bengali: "bn", english: "en", gujarati: "gu",
  hindi: "hi", kannada: "kn", malayalam: "ml", marathi: "mr",
  odia: "or", punjabi: "pa", tamil: "ta", telugu: "te",
};

export const CHART_COLORS = [
  "#6366f1", "#8b5cf6", "#a78bfa", "#c084fc",
  "#e879f9", "#f472b6", "#fb7185", "#f87171",
  "#fb923c", "#fbbf24", "#a3e635", "#34d399",
];

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};


Now update the data layer to handle both old and new schemas:


import {
  MetricsFile, SampleAnalysis, ErrorAnalysisFile,
  ModelInfo, MODEL_REGISTRY, LANGUAGES,
  isOldSchema, migrateLanguageMetrics, OldLanguageMetrics,
  AggregateMetrics, MacroAvgMetrics,
} from "./types";

const BENCHMARK_API = process.env.BENCHMARK_API_URL || "http://localhost:8787";

interface NginxDirEntry {
  name: string;
  type: "directory" | "file";
  mtime: string;
}

async function fetchJson<T>(url: string): Promise<T | null> {
  try {
    const res = await fetch(url, { cache: "no-store" });
    if (!res.ok) return null;
    return (await res.json()) as T;
  } catch {
    return null;
  }
}

// Migrate old-schema metrics.json into new shape
function migrateMetricsFile(raw: Record<string, unknown>): MetricsFile {
  // Check if this is old schema by inspecting a language key
  const firstLang = LANGUAGES.find((l) => l in raw);
  if (!firstLang || !isOldSchema(raw[firstLang] as Record<string, unknown>)) {
    return raw as unknown as MetricsFile;
  }

  const migrated: Record<string, unknown> = {};

  for (const lang of LANGUAGES) {
    if (raw[lang]) {
      migrated[lang] = migrateLanguageMetrics(raw[lang] as OldLanguageMetrics);
    }
  }

  // Migrate __overall__
  const oldOverall = raw.__overall__ as { n_samples: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;
  if (oldOverall) {
    migrated.__overall__ = {
      n_samples: oldOverall.n_samples,
      wer_raw: oldOverall.wer,
      wer_norm: oldOverall.wer_normalized,
      wer_numcanon: oldOverall.wer_normalized,
      cer_norm: oldOverall.cer_normalized,
    } as AggregateMetrics;
  }

  // Migrate __macro_avg__
  const oldMacro = raw.__macro_avg__ as { n_languages: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;
  if (oldMacro) {
    migrated.__macro_avg__ = {
      n_languages: oldMacro.n_languages,
      wer_raw: oldMacro.wer,
      wer_norm: oldMacro.wer_normalized,
      wer_numcanon: oldMacro.wer_normalized,
      cer_norm: oldMacro.cer_normalized,
    } as MacroAvgMetrics;
  }

  migrated.__meta__ = raw.__meta__;

  return migrated as unknown as MetricsFile;
}

// Migrate old sample_analysis entries
function migrateSamples(raw: Record<string, unknown>[]): SampleAnalysis[] {
  return raw.map((s) => {
    const sample = s as Record<string, string>;
    return {
      id: sample.id,
      language: sample.language,
      reference: sample.reference,
      hypothesis: sample.hypothesis,
      ref_norm: sample.ref_norm || sample.ref_normalized,
      hyp_norm: sample.hyp_norm || sample.hyp_normalized,
      detected_language: sample.detected_language,
    } as SampleAnalysis;
  });
}

export async function getModels(): Promise<ModelInfo[]> {
  const models: ModelInfo[] = [];

  for (const [modelId, displayName] of Object.entries(MODEL_REGISTRY)) {
    const entries = await fetchJson<NginxDirEntry[]>(
      `${BENCHMARK_API}/api/benchmarks/${modelId}/`
    );

    if (!entries) {
      models.push({ id: modelId, displayName, checkpoints: [] });
      continue;
    }

    const dirs = entries
      .filter((e) => e.type === "directory")
      .map((e) => e.name);

    const checkpoints: string[] = [];
    for (const dir of dirs) {
      const metrics = await fetchJson<Record<string, unknown>>(
        `${BENCHMARK_API}/api/benchmarks/${modelId}/${dir}/metrics.json`
      );
      if (metrics) checkpoints.push(dir);
    }

    checkpoints.sort((a, b) => {
      const numA = parseInt(a.replace(/\D/g, "")) || 0;
      const numB = parseInt(b.replace(/\D/g, "")) || 0;
      return numA - numB;
    });

    models.push({ id: modelId, displayName, checkpoints });
  }

  return models;
}

export async function getMetrics(modelId: string, checkpoint: string): Promise<MetricsFile | null> {
  const raw = await fetchJson<Record<string, unknown>>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/metrics.json`
  );
  if (!raw) return null;
  return migrateMetricsFile(raw);
}

export async function getSamples(modelId: string, checkpoint: string): Promise<SampleAnalysis[] | null> {
  const raw = await fetchJson<Record<string, unknown>[]>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/sample_analysis.json`
  );
  if (!raw) return null;
  return migrateSamples(raw);
}

export async function getErrorAnalysis(modelId: string, checkpoint: string): Promise<ErrorAnalysisFile | null> {
  return fetchJson<ErrorAnalysisFile>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/error_analysis.json`
  );
}


The file /home/ubuntu/training/asr-benchmark-dashboard/src/lib/data.ts has been updated successfully.

import {
  MetricsFile, SampleAnalysis, ErrorAnalysisFile,
  ModelInfo, MODEL_REGISTRY, LANGUAGES,
  isOldSchema, migrateLanguageMetrics, OldLanguageMetrics,
  AggregateMetrics, MacroAvgMetrics,
} from "./types";

const BENCHMARK_API = process.env.BENCHMARK_API_URL || "http://localhost:8787";

interface NginxDirEntry {
  name: string;
  type: "directory" | "file";
  mtime: string;
}

async function fetchJson<T>(url: string): Promise<T | null> {
  try {
    const res = await fetch(url, { cache: "no-store" });
    if (!res.ok) return null;
    return (await res.json()) as T;
  } catch {
    return null;
  }
}

// Migrate old-schema metrics.json into new shape
function migrateMetricsFile(raw: Record<string, unknown>): MetricsFile {
  // Check if this is old schema by inspecting a language key
  const firstLang = LANGUAGES.find((l) => l in raw);
  if (!firstLang || !isOldSchema(raw[firstLang] as Record<string, unknown>)) {
    return raw as unknown as MetricsFile;
  }

  const migrated: Record<string, unknown> = {};

  for (const lang of LANGUAGES) {
    if (raw[lang]) {
      migrated[lang] = migrateLanguageMetrics(raw[lang] as OldLanguageMetrics);
    }
  }

  // Migrate __overall__
  const oldOverall = raw.__overall__ as { n_samples: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;
  if (oldOverall) {
    migrated.__overall__ = {
      n_samples: oldOverall.n_samples,
      wer_raw: oldOverall.wer,
      wer_norm: oldOverall.wer_normalized,
      wer_numcanon: oldOverall.wer_normalized,
      cer_norm: oldOverall.cer_normalized,
    } as AggregateMetrics;
  }

  // Migrate __macro_avg__
  const oldMacro = raw.__macro_avg__ as { n_languages: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;
  if (oldMacro) {
    migrated.__macro_avg__ = {
      n_languages: oldMacro.n_languages,
      wer_raw: oldMacro.wer,
      wer_norm: oldMacro.wer_normalized,
      wer_numcanon: oldMacro.wer_normalized,
      cer_norm: oldMacro.cer_normalized,
    } as MacroAvgMetrics;
  }

  migrated.__meta__ = raw.__meta__;

  return migrated as unknown as MetricsFile;
}

// Migrate old sample_analysis entries
function migrateSamples(raw: Record<string, unknown>[]): SampleAnalysis[] {
  return raw.map((s) => {
    const sample = s as Record<string, string>;
    return {
      id: sample.id,
      language: sample.language,
      reference: sample.reference,
      hypothesis: sample.hypothesis,
      ref_norm: sample.ref_norm || sample.ref_normalized,
      hyp_norm: sample.hyp_norm || sample.hyp_normalized,
      detected_language: sample.detected_language,
    } as SampleAnalysis;
  });
}

export async function getModels(): Promise<ModelInfo[]> {
  const models: ModelInfo[] = [];

  for (const [modelId, displayName] of Object.entries(MODEL_REGISTRY)) {
    const entries = await fetchJson<NginxDirEntry[]>(
      `${BENCHMARK_API}/api/benchmarks/${modelId}/`
    );

    if (!entries) {
      models.push({ id: modelId, displayName, checkpoints: [] });
      continue;
    }

    const dirs = entries
      .filter((e) => e.type === "directory")
      .map((e) => e.name);

    const checkpoints: string[] = [];
    for (const dir of dirs) {
      const metrics = await fetchJson<Record<string, unknown>>(
        `${BENCHMARK_API}/api/benchmarks/${modelId}/${dir}/metrics.json`
      );
      if (metrics) checkpoints.push(dir);
    }

    checkpoints.sort((a, b) => {
      const numA = parseInt(a.replace(/\D/g, "")) || 0;
      const numB = parseInt(b.replace(/\D/g, "")) || 0;
      return numA - numB;
    });

    models.push({ id: modelId, displayName, checkpoints });
  }

  return models;
}

export async function getMetrics(modelId: string, checkpoint: string): Promise<MetricsFile | null> {
  const raw = await fetchJson<Record<string, unknown>>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/metrics.json`
  );
  if (!raw) return null;
  return migrateMetricsFile(raw);
}

export async function getSamples(modelId: string, checkpoint: string): Promise<SampleAnalysis[] | null> {
  const raw = await fetchJson<Record<string, unknown>[]>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/sample_analysis.json`
  );
  if (!raw) return null;
  return migrateSamples(raw);
}

export async function getErrorAnalysis(modelId: string, checkpoint: string): Promise<ErrorAnalysisFile | null> {
  return fetchJson<ErrorAnalysisFile>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/error_analysis.json`
  );
}

-import { MetricsFile, SampleAnalysis, ModelInfo, MODEL_REGISTRY } from "./types";

+  MetricsFile, SampleAnalysis, ErrorAnalysisFile,

+  ModelInfo, MODEL_REGISTRY, LANGUAGES,

+  isOldSchema, migrateLanguageMetrics, OldLanguageMetrics,

+// Migrate old-schema metrics.json into new shape

+function migrateMetricsFile(raw: Record<string, unknown>): MetricsFile {

+  // Check if this is old schema by inspecting a language key

+  const firstLang = LANGUAGES.find((l) => l in raw);

+  if (!firstLang || !isOldSchema(raw[firstLang] as Record<string, unknown>)) {

+    return raw as unknown as MetricsFile;

+  const migrated: Record<string, unknown> = {};

+      migrated[lang] = migrateLanguageMetrics(raw[lang] as OldLanguageMetrics);

+  const oldOverall = raw.__overall__ as { n_samples: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;

+      wer_norm: oldOverall.wer_normalized,

+      wer_numcanon: oldOverall.wer_normalized,

+      cer_norm: oldOverall.cer_normalized,

+  const oldMacro = raw.__macro_avg__ as { n_languages: number; wer: number; cer: number; wer_normalized: number; cer_normalized: number } | undefined;

+      n_languages: oldMacro.n_languages,

+      wer_norm: oldMacro.wer_normalized,

+      wer_numcanon: oldMacro.wer_normalized,

+      cer_norm: oldMacro.cer_normalized,

+  return migrated as unknown as MetricsFile;

+function migrateSamples(raw: Record<string, unknown>[]): SampleAnalysis[] {

+    const sample = s as Record<string, string>;

+      ref_norm: sample.ref_norm || sample.ref_normalized,

+      hyp_norm: sample.hyp_norm || sample.hyp_normalized,

+      detected_language: sample.detected_language,

-    // Check which directories have metrics.json

-      const metrics = await fetchJson<MetricsFile>(

+      const metrics = await fetchJson<Record<string, unknown>>(

+  const raw = await fetchJson<Record<string, unknown>>(

+  const raw = await fetchJson<Record<string, unknown>[]>(

+export async function getErrorAnalysis(modelId: string, checkpoint: string): Promise<ErrorAnalysisFile | null> {

+    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/error_analysis.json`

import { MetricsFile, SampleAnalysis, ModelInfo, MODEL_REGISTRY } from "./types";

const BENCHMARK_API = process.env.BENCHMARK_API_URL || "http://localhost:8787";

interface NginxDirEntry {
  name: string;
  type: "directory" | "file";
  mtime: string;
}

async function fetchJson<T>(url: string): Promise<T | null> {
  try {
    const res = await fetch(url, { cache: "no-store" });
    if (!res.ok) return null;
    return (await res.json()) as T;
  } catch {
    return null;
  }
}

export async function getModels(): Promise<ModelInfo[]> {
  const models: ModelInfo[] = [];

  for (const [modelId, displayName] of Object.entries(MODEL_REGISTRY)) {
    const entries = await fetchJson<NginxDirEntry[]>(
      `${BENCHMARK_API}/api/benchmarks/${modelId}/`
    );

    if (!entries) {
      models.push({ id: modelId, displayName, checkpoints: [] });
      continue;
    }

    const dirs = entries
      .filter((e) => e.type === "directory")
      .map((e) => e.name);

    // Check which directories have metrics.json
    const checkpoints: string[] = [];
    for (const dir of dirs) {
      const metrics = await fetchJson<MetricsFile>(
        `${BENCHMARK_API}/api/benchmarks/${modelId}/${dir}/metrics.json`
      );
      if (metrics) checkpoints.push(dir);
    }

    checkpoints.sort((a, b) => {
      const numA = parseInt(a.replace(/\D/g, "")) || 0;
      const numB = parseInt(b.replace(/\D/g, "")) || 0;
      return numA - numB;
    });

    models.push({ id: modelId, displayName, checkpoints });
  }

  return models;
}

export async function getMetrics(modelId: string, checkpoint: string): Promise<MetricsFile | null> {
  return fetchJson<MetricsFile>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/metrics.json`
  );
}

export async function getSamples(modelId: string, checkpoint: string): Promise<SampleAnalysis[] | null> {
  return fetchJson<SampleAnalysis[]>(
    `${BENCHMARK_API}/api/benchmarks/${modelId}/${checkpoint}/sample_analysis.json`
  );
}


Now update the API routes — the compare route needs the new field names, and add the errors route:


import { getMetrics, getModels } from "@/lib/data";
import { LANGUAGES, LanguageMetrics } from "@/lib/types";

export async function GET() {
  const models = await getModels();
  // { modelId: { ckpt: { lang: { wer_raw, wer_norm, wer_numcanon, cer_norm } } } }
  const result: Record<string, Record<string, Record<string, Record<string, number>>>> = {};

  for (const model of models) {
    result[model.id] = {};
    for (const ckpt of model.checkpoints) {
      const metrics = await getMetrics(model.id, ckpt);
      if (!metrics) continue;
      const langMetrics: Record<string, Record<string, number>> = {};
      for (const lang of LANGUAGES) {
        const lm = metrics[lang] as LanguageMetrics | undefined;
        if (lm && typeof lm === "object" && "wer_norm" in lm) {
          langMetrics[lang] = {
            wer_raw: lm.wer_raw,
            wer_norm: lm.wer_norm,
            wer_numcanon: lm.wer_numcanon,
            cer_norm: lm.cer_norm,
          };
        }
      }
      langMetrics["__overall__"] = {
        wer_raw: metrics.__overall__?.wer_raw ?? 0,
        wer_norm: metrics.__overall__?.wer_norm ?? 0,
        wer_numcanon: metrics.__overall__?.wer_numcanon ?? 0,
        cer_norm: metrics.__overall__?.cer_norm ?? 0,
      };
      result[model.id][ckpt] = langMetrics;
    }
  }

  return Response.json(result);
}


The file /home/ubuntu/training/asr-benchmark-dashboard/src/app/api/compare/route.ts has been updated successfully.

import { getMetrics, getModels } from "@/lib/data";
import { LANGUAGES, LanguageMetrics } from "@/lib/types";

export async function GET() {
  const models = await getModels();
  // { modelId: { ckpt: { lang: { wer_raw, wer_norm, wer_numcanon, cer_norm } } } }
  const result: Record<string, Record<string, Record<string, Record<string, number>>>> = {};

  for (const model of models) {
    result[model.id] = {};
    for (const ckpt of model.checkpoints) {
      const metrics = await getMetrics(model.id, ckpt);
      if (!metrics) continue;
      const langMetrics: Record<string, Record<string, number>> = {};
      for (const lang of LANGUAGES) {
        const lm = metrics[lang] as LanguageMetrics | undefined;
        if (lm && typeof lm === "object" && "wer_norm" in lm) {
          langMetrics[lang] = {
            wer_raw: lm.wer_raw,
            wer_norm: lm.wer_norm,
            wer_numcanon: lm.wer_numcanon,
            cer_norm: lm.cer_norm,
          };
        }
      }
      langMetrics["__overall__"] = {
        wer_raw: metrics.__overall__?.wer_raw ?? 0,
        wer_norm: metrics.__overall__?.wer_norm ?? 0,
        wer_numcanon: metrics.__overall__?.wer_numcanon ?? 0,
        cer_norm: metrics.__overall__?.cer_norm ?? 0,
      };
      result[model.id][ckpt] = langMetrics;
    }
  }

  return Response.json(result);
}

-import { LANGUAGES } from "@/lib/types";

+import { LANGUAGES, LanguageMetrics } from "@/lib/types";

-  const result: Record<string, Record<string, Record<string, number>>> = {};

+  // { modelId: { ckpt: { lang: { wer_raw, wer_norm, wer_numcanon, cer_norm } } } }

+  const result: Record<string, Record<string, Record<string, Record<string, number>>>> = {};

-      const langMetrics: Record<string, number> = {};

+      const langMetrics: Record<string, Record<string, number>> = {};

-        if (lm && typeof lm === "object" && "wer" in lm) {

-          langMetrics[lang] = (lm as { wer: number }).wer;

+        const lm = metrics[lang] as LanguageMetrics | undefined;

+        if (lm && typeof lm === "object" && "wer_norm" in lm) {

+            wer_numcanon: lm.wer_numcanon,

-      langMetrics["__overall__"] = metrics.__overall__?.wer ?? 0;

+        wer_raw: metrics.__overall__?.wer_raw ?? 0,

+        wer_norm: metrics.__overall__?.wer_norm ?? 0,

+        wer_numcanon: metrics.__overall__?.wer_numcanon ?? 0,

+        cer_norm: metrics.__overall__?.cer_norm ?? 0,

import { getMetrics, getModels } from "@/lib/data";
import { LANGUAGES } from "@/lib/types";

export async function GET() {
  const models = await getModels();
  const result: Record<string, Record<string, Record<string, number>>> = {};

  for (const model of models) {
    result[model.id] = {};
    for (const ckpt of model.checkpoints) {
      const metrics = await getMetrics(model.id, ckpt);
      if (!metrics) continue;
      const langMetrics: Record<string, number> = {};
      for (const lang of LANGUAGES) {
        const lm = metrics[lang];
        if (lm && typeof lm === "object" && "wer" in lm) {
          langMetrics[lang] = (lm as { wer: number }).wer;
        }
      }
      langMetrics["__overall__"] = metrics.__overall__?.wer ?? 0;
      result[model.id][ckpt] = langMetrics;
    }
  }

  return Response.json(result);
}


Bash

Create errors API directory


(Bash completed with no output)


Write

import { NextRequest } from "next/server";
import { getErrorAnalysis } from "@/lib/data";

export async function GET(request: NextRequest) {
  const modelId = request.nextUrl.searchParams.get("model");
  const checkpoint = request.nextUrl.searchParams.get("checkpoint");

  if (!modelId || !checkpoint) {
    return Response.json({ error: "model and checkpoint params required" }, { status: 400 });
  }

  const errors = await getErrorAnalysis(modelId, checkpoint);
  if (!errors) {
    return Response.json({ error: "Not found" }, { status: 404 });
  }

  return Response.json(errors);
}


File created successfully at: /home/ubuntu/training/asr-benchmark-dashboard/src/app/api/errors/route.ts

import { NextRequest } from "next/server";
import { getErrorAnalysis } from "@/lib/data";

export async function GET(request: NextRequest) {
  const modelId = request.nextUrl.searchParams.get("model");
  const checkpoint = request.nextUrl.searchParams.get("checkpoint");

  if (!modelId || !checkpoint) {
    return Response.json({ error: "model and checkpoint params required" }, { status: 400 });
  }

  const errors = await getErrorAnalysis(modelId, checkpoint);
  if (!errors) {
    return Response.json({ error: "Not found" }, { status: 404 });
  }

  return Response.json(errors);
}


Now update the frontend components to use the new metric tiers. The main page, metric cards, bar charts, heatmap, and compare chart all need to reference `wer_raw`/`wer_norm`/`wer_numcanon`/`cer_norm` instead of the old field names:


1→"use client";
     2→
     3→import {
     4→  BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip,
     5→  ResponsiveContainer, Cell, ReferenceLine,
     6→} from "recharts";
     7→import { CHART_COLORS, LANGUAGES } from "@/lib/types";
     8→
     9→interface LanguageBarChartProps {
    10→  data: { language: string; wer: number; cer: number; wer_normalized: number }[];
    11→  metric: "wer" | "cer" | "wer_normalized";
    12→  title: string;
    13→  height?: number;
    14→}
    15→
    16→export default function LanguageBarChart({ data, metric, title, height = 320 }: LanguageBarChartProps) {
    17→  const sorted = [...data].sort((a, b) => a[metric] - b[metric]);
    18→  const avg = sorted.reduce((s, d) => s + d[metric], 0) / sorted.length;
    19→
    20→  return (
    21→

    22→

    23→
{title}

    24→        avg: {avg.toFixed(2)}%
    25→

    26→      <ResponsiveContainer width="100%" height={height}>
    27→        <BarChart data={sorted} margin={{ top: 5, right: 10, left: 0, bottom: 5 }}>
    28→          <CartesianGrid strokeDasharray="3 3" vertical={false} />
    29→          <XAxis
    30→            dataKey="language"
    31→            tick={{ fontSize: 10 }}
    32→            tickFormatter={(v: string) => v.slice(0, 3).toUpperCase()}
    33→            axisLine={false}
    34→            tickLine={false}
    35→          />
    36→          <YAxis
    37→            tick={{ fontSize: 10 }}
    38→            axisLine={false}
    39→            tickLine={false}
    40→            tickFormatter={(v: number) => `${v}%`}
    41→          />
    42→          <Tooltip
    43→            formatter={(value: unknown) => [`${Number(value).toFixed(2)}%`, metric.toUpperCase()]}
    44→            contentStyle={{
    45→              background: "#18181b",
    46→              border: "1px solid #3f3f46",
    47→              borderRadius: "8px",
    48→              fontSize: "12px",
    49→            }}
    50→          />
    51→          <ReferenceLine y={avg} stroke="#71717a" strokeDasharray="5 5" />
    52→          <Bar dataKey={metric} radius={[4, 4, 0, 0]} maxBarSize={40}>
    53→            {sorted.map((_, i) => (
    54→              <Cell key={i} fill={CHART_COLORS[i % CHART_COLORS.length]} />
    55→            ))}
    56→          </Bar>
    57→        </BarChart>
    58→      </ResponsiveContainer>
    59→

    60→  );
    61→}
    62→

"use client";

import {
  BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip,
  ResponsiveContainer, Cell, ReferenceLine,
} from "recharts";
import { CHART_COLORS, LANGUAGES } from "@/lib/types";

interface LanguageBarChartProps {
  data: { language: string; wer: number; cer: number; wer_normalized: number }[];
  metric: "wer" | "cer" | "wer_normalized";
  title: string;
  height?: number;
}

export default function LanguageBarChart({ data, metric, title, height = 320 }: LanguageBarChartProps) {
  const sorted = [...data].sort((a, b) => a[metric] - b[metric]);
  const avg = sorted.reduce((s, d) => s + d[metric], 0) / sorted.length;

  return (

{title}

        avg: {avg.toFixed(2)}%

      <ResponsiveContainer width="100%" height={height}>
        <BarChart data={sorted} margin={{ top: 5, right: 10, left: 0, bottom: 5 }}>
          <CartesianGrid strokeDasharray="3 3" vertical={false} />
          <XAxis
            dataKey="language"
            tick={{ fontSize: 10 }}
            tickFormatter={(v: string) => v.slice(0, 3).toUpperCase()}
            axisLine={false}
            tickLine={false}
          />
          <YAxis
            tick={{ fontSize: 10 }}
            axisLine={false}
            tickLine={false}
            tickFormatter={(v: number) => `${v}%`}
          />
          <Tooltip
            formatter={(value: unknown) => [`${Number(value).toFixed(2)}%`, metric.toUpperCase()]}
            contentStyle={{
              background: "#18181b",
              border: "1px solid #3f3f46",
              borderRadius: "8px",
              fontSize: "12px",
            }}
          />
          <ReferenceLine y={avg} stroke="#71717a" strokeDasharray="5 5" />
          <Bar dataKey={metric} radius={[4, 4, 0, 0]} maxBarSize={40}>
            {sorted.map((_, i) => (
              <Cell key={i} fill={CHART_COLORS[i % CHART_COLORS.length]} />
            ))}
          </Bar>
        </BarChart>
      </ResponsiveContainer>

  );
}


"use client";

import {
  BarChart, Bar, XAxis, YAxis, CartesianGrid, Tooltip,
  ResponsiveContainer, Cell, ReferenceLine,
} from "recharts";
import { CHART_COLORS, MetricTier, METRIC_LABELS } from "@/lib/types";

interface LanguageBarChartProps {
  data: { language: string; [key: string]: number | string }[];
  metric: MetricTier;
  title: string;
  height?: number;
}

export default function LanguageBarChart({ data, metric, title, height = 320 }: LanguageBarChartProps) {
  const sorted = [...data].sort((a, b) => (a[metric] as number) - (b[metric] as number));
  const avg = sorted.reduce((s, d) => s + (d[metric] as number), 0) / sorted.length;

  return (

{title}

        avg: {avg.toFixed(2)}%

      <ResponsiveContainer width="100%" height={height}>
        <BarChart data={sorted} margin={{ top: 5, right: 10, left: 0, bottom: 5 }}>
          <CartesianGrid strokeDasharray="3 3" vertical={false} />
          <XAxis
            dataKey="language"
            tick={{ fontSize: 10 }}
            tickFormatter={(v: string) => v.slice(0, 3).toUpperCase()}
            axisLine={false}
            tickLine={false}
          />
          <YAxis
            tick={{ fontSize: 10 }}
            axisLine={false}
            tickLine={false}
            tickFormatter={(v: number) => `${v}%`}
          />
          <Tooltip
            formatter={(value: unknown) => [`${Number(value).toFixed(2)}%`, METRIC_LABELS[metric]]}
            contentStyle={{
              background: "#18181b",
              border: "1px solid #3f3f46",
              borderRadius: "8px",
              fontSize: "12px",
            }}
          />
          <ReferenceLine y={avg} stroke="#71717a" strokeDasharray="5 5" />
          <Bar dataKey={metric} radius={[4, 4, 0, 0]} maxBarSize={40}>
            {sorted.map((_, i) => (
              <Cell key={i} fill={CHART_COLORS[i % CHART_COLORS.length]} />
            ))}
          </Bar>
        </BarChart>
      </ResponsiveContainer>

  );
}