So, i've got a new task for you. so i've got these benchmarks scores from these new superior models that i want to compare as a baseline and compare on the dashboard. can you now go through these /home/ubuntu/training/api_results.zip and understand the structure, and process our WER scores as i mentioned above, and once processed manually by you, i want you to create a new section in dashboard which compares these superior models and one of our or all of our checkpoints(should be able to select ckpt-wise and deselect if clumsy) and ensure everything is computed by you, if any scores exist, igfnore them and calculate as per our norms, as we would need apples-apples.


Bash

cd /home/ubuntu/training && ls -la api_results.zip && unzip -l api_results.zip | head -60

Check zip contents


-rw-rw-r-- 1 ubuntu ubuntu 3847885 Mar 26 20:36 api_results.zip
Archive:  api_results.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2026-03-26 20:30   api_results/
  4544063  2026-03-26 20:30   api_results/elevenlabs_full_6k.jsonl
     2888  2026-03-26 20:30   api_results/gemini_flash_v2_run.log
  5160180  2026-03-26 20:30   api_results/gemini_gemini_3_flash_preview_full_6k.jsonl
    42459  2026-03-26 20:30   api_results/sarvam_test_5per.jsonl
  4376984  2026-03-26 20:30   api_results/sarvam_full_6k.jsonl
    12184  2026-03-26 20:30   api_results/gemini_fast_run.log
     7261  2026-03-26 20:30   api_results/elevenlabs_v1_run.log
        0  2026-03-26 20:30   api_results/gemini_pro_run.log
  4446190  2026-03-26 20:30   api_results/gemini_gemini_3_1_pro_preview_full_6k.jsonl
    10611  2026-03-26 20:30   api_results/benchmark_run.log
  4460297  2026-03-26 20:30   api_results/gemini_3_flash_strict_full_6k.jsonl
  5137465  2026-03-26 20:30   api_results/gemini_flash_v2_full_6k.jsonl
     2882  2026-03-26 20:30   api_results/gemini_3flash_strict_run.log
        0  2026-03-26 20:30   api_results/gemini_run.log
  4559589  2026-03-26 20:30   api_results/elevenlabs_v1_full_6k.jsonl
    44158  2026-03-26 20:30   api_results/elevenlabs_test_5per.jsonl
    18007  2026-03-26 20:30   api_results/sample.txt
---------                     -------
 32825218                     18 files


unzip -o api_results.zip -d . 2>&1 | tail -5

Extract zip


  inflating: ./api_results/gemini_3flash_strict_run.log
 extracting: ./api_results/gemini_run.log
  inflating: ./api_results/elevenlabs_v1_full_6k.jsonl
  inflating: ./api_results/elevenlabs_test_5per.jsonl
  inflating: ./api_results/sample.txt


cd /home/ubuntu/training/api_results && head -5 elevenlabs_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 elevenlabs_full_6k.jsonl

Check elevenlabs format


{"id": "as_0000", "language": "assamese", "lang_code": "as", "filename": "as_0000.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0000.wav", "reference": "অঁ দুদিনীয়া মই দুদিনীয়া বুলিয়ে শুনিছিলোঁ এনেকৈ ভালে লাগিব", "duration": 3.292, "provider": "elevenlabs", "model": "scribe_v2", "transcript": "অ দুদিনীয়া মইও দুদিনীয়া বুলি ভাবিছিলোঁ। ভালে লাগিব", "status": "ok", "detected_lang": "asm"}
{"id": "as_0001", "language": "assamese", "lang_code": "as", "filename": "as_0001.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0001.wav", "reference": "পাকিস্তান", "duration": 1.151, "provider": "elevenlabs", "model": "scribe_v2", "transcript": "পাকিস্তান", "status": "ok", "detected_lang": "asm"}
{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "elevenlabs", "model": "scribe_v2", "transcript": "অহ বাৰ, বিশ, তেইশ, পঁচিশ, ত্ৰিশ", "status": "ok", "detected_lang": "asm"}


head -3 gemini_gemini_3_flash_preview_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 gemini_gemini_3_flash_preview_full_6k.jsonl

Check gemini flash format


{"id": "as_0001", "language": "assamese", "lang_code": "as", "filename": "as_0001.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0001.wav", "reference": "পাকিস্তান", "duration": 1.151, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "পাকিস্তান", "status": "ok"}
{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "বাৰ বিশ তেইশ পঁচিশ ত্ৰিশ", "status": "ok"}
{"id": "as_0003", "language": "assamese", "lang_code": "as", "filename": "as_0003.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0003.wav", "reference": "ফুটবল সঁচাকৈয়ে প্ৰিয় যিহে যিহেতু মই ফুটবল নিজে খেলোঁ ধৰি লওক স্কুলতো খেলোঁ স্কুলত খেলিছিলোঁ ছৰী স্কুলতো খেলিছিলোঁ", "duration": 9.759, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "ফুটবল সঁচাকৈয়ে প্ৰিয় যিহেতু মই ফুটবল নিজে খেলো ধৰিলক স্কুলতো খেলো স্কুলত খেলিছিলো চৰী স্কুলতো খেলিছিলো", "status": "ok"}


head -3 sarvam_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 sarvam_full_6k.jsonl

Check sarvam format


{"id": "as_0000", "language": "assamese", "lang_code": "as", "filename": "as_0000.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0000.wav", "reference": "অঁ দুদিনীয়া মই দুদিনীয়া বুলিয়ে শুনিছিলোঁ এনেকৈ ভালে লাগিব", "duration": 3.292, "provider": "sarvam", "model": "saaras:v3", "transcript": "অঁ, 2-দিনীয়া ময়ো 2-দিনীয়া বুলিয়ে শুনিছিলোঁ, তেনেকৈ ভালেই লাগিব।", "status": "ok"}
{"id": "as_0001", "language": "assamese", "lang_code": "as", "filename": "as_0001.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0001.wav", "reference": "পাকিস্তান", "duration": 1.151, "provider": "sarvam", "model": "saaras:v3", "transcript": "পাকিস্তান।", "status": "ok"}
{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "sarvam", "model": "saaras:v3", "transcript": "অঁ 12 20 23 25 30।", "status": "ok"}


head -3 gemini_gemini_3_1_pro_preview_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 gemini_gemini_3_1_pro_preview_full_6k.jsonl

Check gemini pro format


{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "gemini", "model": "gemini-3.1-pro-preview", "transcript": "বাৰ বিশ তেইশ পঁচিশ ত্ৰিশ", "status": "ok"}
{"id": "as_0004", "language": "assamese", "lang_code": "as", "filename": "as_0004.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0004.wav", "reference": "তেওঁলোকক দমন কৰিবলৈ চৰকাৰে ভাৰতৰ যিসকল সীমাৰক্ষী বাহিনী আছে বি এছ এফ এ হওক বা এছ এছ বি য়ে হওক তেওঁলোকক বিশেষভাবে সক্ষম কৰি ৰাখে", "duration": 6.591, "provider": "gemini", "model": "gemini-3.1-pro-preview", "transcript": "তেওঁলোকক দমন কৰিবলৈ চৰকাৰে ভাৰতৰ যিসকল সীমাৰক্ষী বাহিনী আছে বি এছ এফ এ হওক বা এছ এছ বি এ হওক তেওঁলোকক বিশেষভাৱে সক্ষম কৰি ৰাখে", "status": "ok"}
{"id": "as_0001", "language": "assamese", "lang_code": "as", "filename": "as_0001.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0001.wav", "reference": "পাকিস্তান", "duration": 1.151, "provider": "gemini", "model": "gemini-3.1-pro-preview", "transcript": "পাকিস্তান", "status": "ok"}


head -3 gemini_flash_v2_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 gemini_flash_v2_full_6k.jsonl

Check gemini flash v2 format


{"id": "as_0017", "language": "assamese", "lang_code": "as", "filename": "as_0017.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0017.wav", "reference": "ঐতিহ্যমণ্ডিত ছাত্ৰক একতা সভাৰ সৰ্বোচ্চ পদত বহি ছাত্ৰ একতা সভা তথা মহাবিদ্যালয়খনক প্ৰতিনিধিত্ব কৰিব পৰা মুহূৰ্তটো মোৰ বাবে আটাইতকৈ গৌৰম্বিত সাফল্য বুলি মই কম", "duration": 9.759, "provider": "gemini", "model": "gemini-2.5-flash", "transcript": "ঐতিহ্যমন্ত্রীৰ ছাত্ৰকথা সভাৰ সৰ্বোচ্চ পদত বহি ছাত্ৰকথা সভা তথা মহাবিদ্যালয়খনক প্রতিনিধিত্ব কৰিব পৰা মুহূৰ্তটো মোৰ বাবে আটাইতকৈ গৌৰৱনীত সাফল্য বুলি মই কওঁ।", "status": "ok"}
{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "gemini", "model": "gemini-2.5-flash", "transcript": "বাৰ বিশ তেইশ পঁচিশ ত্ৰিশ", "status": "ok"}
{"id": "as_0020", "language": "assamese", "lang_code": "as", "filename": "as_0020.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0020.wav", "reference": "অঁ", "duration": 0.351, "provider": "gemini", "model": "gemini-2.5-flash", "transcript": "লগতে", "status": "ok"}


head -3 elevenlabs_v1_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 elevenlabs_v1_full_6k.jsonl

Check elevenlabs v1 format


{"id": "as_0001", "language": "assamese", "lang_code": "as", "filename": "as_0001.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0001.wav", "reference": "পাকিস্তান", "duration": 1.151, "provider": "elevenlabs", "model": "scribe_v1", "transcript": "Pakistan. (two seconds pause)", "status": "ok", "detected_lang": "asm"}
{"id": "as_0000", "language": "assamese", "lang_code": "as", "filename": "as_0000.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0000.wav", "reference": "অঁ দুদিনীয়া মই দুদিনীয়া বুলিয়ে শুনিছিলোঁ এনেকৈ ভালে লাগিব", "duration": 3.292, "provider": "elevenlabs", "model": "scribe_v1", "transcript": "ও দুদিন আগমা ৰুদ্ধি নেবো নাকি ভালে থাইব।", "status": "ok", "detected_lang": "asm"}
{"id": "as_0002", "language": "assamese", "lang_code": "as", "filename": "as_0002.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0002.wav", "reference": "বাৰ বিছ তেইছ পচিছ ত্ৰিছ", "duration": 6.46, "provider": "elevenlabs", "model": "scribe_v1", "transcript": "অ-বার, বিশ, তেইশি, পঁচিশ, ত্রিশ (coughing)।", "status": "ok", "detected_lang": "asm"}


head -3 gemini_3_flash_strict_full_6k.jsonl | python3 -m json.tool --no-ensure-ascii 2>/dev/null || head -3 gemini_3_flash_strict_full_6k.jsonl

Check gemini flash strict format


{"id": "as_0030", "language": "assamese", "lang_code": "as", "filename": "as_0030.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0030.wav", "reference": "আৰু গাখীৰ চাহ বেছি মানুহৰে প্ৰিয় হয় মোৰো গাখীৰ চাহ খুবেই প্ৰিয় কিন্তু এ ইয়াত অকণমান", "duration": 7.073, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "আৰু গাখীৰ চাহ বেছি মানুহৰে প্ৰিয় হয়। মোৰো গাখীৰ চাহ খুবেই প্ৰিয়। কিন্তু এতিয়া অলপমান কমাই দিছো।", "status": "ok"}
{"id": "as_0020", "language": "assamese", "lang_code": "as", "filename": "as_0020.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0020.wav", "reference": "অঁ", "duration": 0.351, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "ও", "status": "ok"}
{"id": "as_0019", "language": "assamese", "lang_code": "as", "filename": "as_0019.wav", "wav_path": "/workspace/parakeet_eval_tags/benchmark_5k/assamese_audio/as_0019.wav", "reference": "দুই", "duration": 0.545, "provider": "gemini", "model": "gemini-3-flash-preview", "transcript": "দুই", "status": "ok"}


for f in *_full_6k.jsonl; do echo "=== $f ===" && wc -l "$f" && python3 -c "
import json
with open('$f') as fh:
    lines = [json.loads(l) for l in fh]
provider = lines[0].get('provider','?')
model = lines[0].get('model','?')
langs = set(l['language'] for l in lines)
ok = sum(1 for l in lines if l.get('status')=='ok')
err = len(lines) - ok
print(f'  Provider: {provider}, Model: {model}')
print(f'  Total: {len(lines)}, OK: {ok}, Error: {err}')
print(f'  Languages: {len(langs)} -> {sorted(langs)}')
" && echo ""; done

Survey all full_6k files


=== elevenlabs_full_6k.jsonl ===
6000 elevenlabs_full_6k.jsonl
  Provider: elevenlabs, Model: scribe_v2
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== elevenlabs_v1_full_6k.jsonl ===
6000 elevenlabs_v1_full_6k.jsonl
  Provider: elevenlabs, Model: scribe_v1
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== gemini_3_flash_strict_full_6k.jsonl ===
6000 gemini_3_flash_strict_full_6k.jsonl
  Provider: gemini, Model: gemini-3-flash-preview
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== gemini_flash_v2_full_6k.jsonl ===
6000 gemini_flash_v2_full_6k.jsonl
  Provider: gemini, Model: gemini-2.5-flash
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== gemini_gemini_3_1_pro_preview_full_6k.jsonl ===
6000 gemini_gemini_3_1_pro_preview_full_6k.jsonl
  Provider: gemini, Model: gemini-3.1-pro-preview
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== gemini_gemini_3_flash_preview_full_6k.jsonl ===
6000 gemini_gemini_3_flash_preview_full_6k.jsonl
  Provider: gemini, Model: gemini-3-flash-preview
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']

=== sarvam_full_6k.jsonl ===
6000 sarvam_full_6k.jsonl
  Provider: sarvam, Model: saaras:v3
  Total: 6000, OK: 6000, Error: 0
  Languages: 12 -> ['assamese', 'bengali', 'english', 'gujarati', 'hindi', 'kannada', 'malayalam', 'marathi', 'odia', 'punjabi', 'tamil', 'telugu']


All 7 models, 6000 samples each, 12 languages, 100% OK. Now let me figure out the distinct models and check for duplicates:


# Check gemini_3_flash_strict vs gemini_gemini_3_flash_preview — same model name, different files
python3 -c "
import json
f1 = [json.loads(l) for l in open('gemini_3_flash_strict_full_6k.jsonl')]
f2 = [json.loads(l) for l in open('gemini_gemini_3_flash_preview_full_6k.jsonl')]
# Compare first 5 transcripts for same IDs
ids1 = {s['id']: s['transcript'] for s in f1}
ids2 = {s['id']: s['transcript'] for s in f2}
common = set(ids1) & set(ids2)
diffs = sum(1 for i in common if ids1[i] != ids2[i])
print(f'Common IDs: {len(common)}, Differences: {diffs}')
# Show a diff
for i in sorted(common)[:3]:
    if ids1[i] != ids2[i]:
        print(f'  {i}: strict={ids1[i][:60]}... vs preview={ids2[i][:60]}...')
"

Check if strict and preview are different runs


Common IDs: 6000, Differences: 5678
  as_0000: strict=নহয় অ ধুনীয়া ধুনীয়া বুলি শুনিছিলো আৰু ধুনীয়া আৰু ভাল লাগিব।... vs preview=অ দুনিয়া মোৰ দুনিয়া বুলি হ’ব পাৰিলে ভালে লাগিব...
  as_0001: strict=পাকিস্তান।... vs preview=পাকিস্তান...
  as_0002: strict=বাৰ বিছ তেইছ পঁচিশ ত্ৰিশ... vs preview=বাৰ বিশ তেইশ পঁচিশ ত্ৰিশ...


Definitely different runs. So we have 7 distinct baseline models:

1. **ElevenLabs Scribe v2** (`elevenlabs_full_6k.jsonl`)
2. **ElevenLabs Scribe v1** (`elevenlabs_v1_full_6k.jsonl`)
3. **Gemini 3 Flash (strict)** (`gemini_3_flash_strict_full_6k.jsonl`)
4. **Gemini 3 Flash (preview)** (`gemini_gemini_3_flash_preview_full_6k.jsonl`)
5. **Gemini 2.5 Flash** (`gemini_flash_v2_full_6k.jsonl`)
6. **Gemini 3.1 Pro** (`gemini_gemini_3_1_pro_preview_full_6k.jsonl`)
7. **Sarvam Saaras v3** (`sarvam_full_6k.jsonl`)

Now let me write the evaluation script that computes all 4 metric tiers using our exact normalization pipeline:


Install jiwer


Requirement already satisfied: rapidfuzz>=3.9.7 in /home/ubuntu/.local/lib/python3.10/site-packages (from jiwer) (3.14.3)
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)
WARNING: Ignoring invalid distribution -riton (/home/ubuntu/.local/lib/python3.10/site-packages)


Write

"""
Compute all 4 metric tiers for API baseline models.
Normalization version: v1
"""
import json
import re
import unicodedata
import sys
from collections import defaultdict
from pathlib import Path
from datetime import datetime, timezone

from jiwer import wer as compute_wer, cer as compute_cer

# ── Normalization pipeline ──

# Zero-width characters to strip
ZW_CHARS = re.compile(r'[\u200b-\u200f\u2028-\u202f\ufeff\u00ad]')

# Punctuation sets
# Common punctuation (ASCII + Unicode general)
PUNCT_COMMON = re.compile(r'[!"#$%&\'()*+,\-./:;<=>?@\[\\\]^_`{|}~]')
# Indic punctuation (dandas, etc.) — keep single danda for segmentation? No, remove for norm.
PUNCT_INDIC = re.compile(r'[\u0964\u0965\u0970\u0971]')  # danda, double danda, abbreviation signs
# Extended punctuation
PUNCT_EXTENDED = re.compile(r'[\u2010-\u2027\u2030-\u205e\u2e00-\u2e4f]')
# Quotes normalization
QUOTES = re.compile(r'[\u2018\u2019\u201a\u201b\u201c\u201d\u201e\u201f\u00ab\u00bb]')

# Parenthetical annotations like "(coughing)", "(two seconds pause)"
ANNOTATIONS = re.compile(r'\([^)]*\)')

def norm_raw(text: str) -> str:
    """Step for wer_raw: NFC unicode + trim only."""
    text = unicodedata.normalize('NFC', text)
    text = text.strip()
    return text

def norm_standard(text: str) -> str:
    """Steps for wer_norm: NFKC + strip ZW + whitespace + punct + case fold."""
    # Step 1: NFKC
    text = unicodedata.normalize('NFKC', text)
    # Step 2: Strip zero-width chars
    text = ZW_CHARS.sub('', text)
    # Step 2.5: Remove parenthetical annotations
    text = ANNOTATIONS.sub('', text)
    # Step 3: Normalize whitespace
    text = ' '.join(text.split())
    # Step 4: Standardize quotes/dashes
    text = QUOTES.sub("'", text)
    text = text.replace('\u2014', '-').replace('\u2013', '-').replace('\u2012', '-')
    # Step 5: Remove punctuation
    text = PUNCT_COMMON.sub('', text)
    text = PUNCT_INDIC.sub('', text)
    text = PUNCT_EXTENDED.sub('', text)
    # Step 6: Case fold (affects English only, Indic scripts have no case)
    text = text.lower()
    # Final whitespace cleanup
    text = ' '.join(text.split()).strip()
    return text

def norm_numcanon(text: str) -> str:
    """Steps for wer_numcanon: norm_standard + number canonicalization."""
    text = norm_standard(text)
    # Remove commas in numbers (e.g., "25,000" -> "25000")
    text = re.sub(r'(\d),(\d)', r'\1\2', text)
    # Normalize Devanagari digits to ASCII
    for offset, base in [(0x0966, '0'), (0x09E6, '0'), (0x0A66, '0'),
                          (0x0AE6, '0'), (0x0B66, '0'), (0x0BE6, '0'),
                          (0x0C66, '0'), (0x0CE6, '0'), (0x0D66, '0')]:
        for i in range(10):
            text = text.replace(chr(offset + i), str(i))
    return text

# ── Metric computation ──

def compute_metrics_for_samples(samples: list[dict]) -> dict:
    """Compute all 4 metric tiers for a list of samples."""
    by_lang = defaultdict(list)
    for s in samples:
        by_lang[s['language']].append(s)

    result = {}
    all_ref_raw, all_hyp_raw = [], []
    all_ref_norm, all_hyp_norm = [], []
    all_ref_nc, all_hyp_nc = [], []

    for lang in sorted(by_lang.keys()):
        lang_samples = by_lang[lang]
        refs_raw, hyps_raw = [], []
        refs_norm, hyps_norm = [], []
        refs_nc, hyps_nc = [], []
        empty_count = 0

        for s in lang_samples:
            ref = s['reference']
            hyp = s.get('transcript', '') or ''

            if not hyp.strip():
                empty_count += 1

            rr, hr = norm_raw(ref), norm_raw(hyp)
            rn, hn = norm_standard(ref), norm_standard(hyp)
            rnc, hnc = norm_numcanon(ref), norm_numcanon(hyp)

            # Skip empty refs (shouldn't happen but safety)
            if not rr.strip():
                continue

            refs_raw.append(rr); hyps_raw.append(hr if hr.strip() else '<empty>')
            refs_norm.append(rn if rn else '<empty>'); hyps_norm.append(hn if hn else '<empty>')
            refs_nc.append(rnc if rnc else '<empty>'); hyps_nc.append(hnc if hnc else '<empty>')

        w_raw = compute_wer(refs_raw, hyps_raw) * 100
        w_norm = compute_wer(refs_norm, hyps_norm) * 100
        w_nc = compute_wer(refs_nc, hyps_nc) * 100
        c_norm = compute_cer(refs_norm, hyps_norm) * 100

        result[lang] = {
            'n_samples': len(lang_samples),
            'wer_raw': round(w_raw, 2),
            'wer_norm': round(w_norm, 2),
            'wer_numcanon': round(w_nc, 2),
            'cer_norm': round(c_norm, 2),
            'empty_hypotheses': empty_count,
            'normalization_delta': {
                'raw_to_norm': round(w_raw - w_norm, 2),
                'norm_to_numcanon': round(w_norm - w_nc, 2),
            }
        }

        all_ref_raw.extend(refs_raw); all_hyp_raw.extend(hyps_raw)
        all_ref_norm.extend(refs_norm); all_hyp_norm.extend(hyps_norm)
        all_ref_nc.extend(refs_nc); all_hyp_nc.extend(hyps_nc)

    # Overall (micro-average)
    result['__overall__'] = {
        'n_samples': len(samples),
        'wer_raw': round(compute_wer(all_ref_raw, all_hyp_raw) * 100, 2),
        'wer_norm': round(compute_wer(all_ref_norm, all_hyp_norm) * 100, 2),
        'wer_numcanon': round(compute_wer(all_ref_nc, all_hyp_nc) * 100, 2),
        'cer_norm': round(compute_cer(all_ref_norm, all_hyp_norm) * 100, 2),
    }

    # Macro average
    lang_keys = [k for k in result if not k.startswith('_')]
    n_langs = len(lang_keys)
    result['__macro_avg__'] = {
        'n_languages': n_langs,
        'wer_raw': round(sum(result[l]['wer_raw'] for l in lang_keys) / n_langs, 2),
        'wer_norm': round(sum(result[l]['wer_norm'] for l in lang_keys) / n_langs, 2),
        'wer_numcanon': round(sum(result[l]['wer_numcanon'] for l in lang_keys) / n_langs, 2),
        'cer_norm': round(sum(result[l]['cer_norm'] for l in lang_keys) / n_langs, 2),
    }

    return result

def build_sample_analysis(samples: list[dict]) -> list[dict]:
    """Build sample_analysis.json entries."""
    out = []
    for s in samples:
        ref = s['reference']
        hyp = s.get('transcript', '') or ''
        rn = norm_standard(ref)
        hn = norm_standard(hyp)
        rnc = norm_numcanon(ref)
        hnc = norm_numcanon(hyp)

        flags = []
        if ref == hyp:
            flags.append('exact_match')
        if rn == hn:
            flags.append('exact_match_norm')
        if ref != hyp and rn == hn:
            flags.append('punctuation_only_diff')
        if not hyp.strip():
            flags.append('empty_hypothesis')
        # Numeric mismatch: digits present in either
        if re.search(r'\d', ref) or re.search(r'\d', hyp):
            if rn != hn:
                flags.append('numeric_mismatch')

        out.append({
            'id': s['id'],
            'language': s['language'],
            'reference': ref,
            'hypothesis': hyp,
            'ref_norm': rn,
            'hyp_norm': hn,
            'ref_numcanon': rnc,
            'hyp_numcanon': hnc,
            'detected_language': s.get('detected_lang', ''),
            'flags': flags,
        })
    return out

# ── Model registry ──

MODELS = {
    'elevenlabs_full_6k.jsonl': {
        'model_id': 'elevenlabs-scribe-v2',
        'display_name': 'ElevenLabs Scribe v2',
        'model_type': 'elevenlabs-scribe-v2',
    },
    'elevenlabs_v1_full_6k.jsonl': {
        'model_id': 'elevenlabs-scribe-v1',
        'display_name': 'ElevenLabs Scribe v1',
        'model_type': 'elevenlabs-scribe-v1',
    },
    'gemini_3_flash_strict_full_6k.jsonl': {
        'model_id': 'gemini-3-flash-strict',
        'display_name': 'Gemini 3 Flash (Strict)',
        'model_type': 'gemini-3-flash-preview',
    },
    'gemini_gemini_3_flash_preview_full_6k.jsonl': {
        'model_id': 'gemini-3-flash-preview',
        'display_name': 'Gemini 3 Flash (Preview)',
        'model_type': 'gemini-3-flash-preview',
    },
    'gemini_flash_v2_full_6k.jsonl': {
        'model_id': 'gemini-2.5-flash',
        'display_name': 'Gemini 2.5 Flash',
        'model_type': 'gemini-2.5-flash',
    },
    'gemini_gemini_3_1_pro_preview_full_6k.jsonl': {
        'model_id': 'gemini-3.1-pro',
        'display_name': 'Gemini 3.1 Pro',
        'model_type': 'gemini-3.1-pro-preview',
    },
    'sarvam_full_6k.jsonl': {
        'model_id': 'sarvam-saaras-v3',
        'display_name': 'Sarvam Saaras v3',
        'model_type': 'saaras:v3',
    },
}

OUTPUT_BASE = Path('/home/ubuntu/training/benchmark_outputs')

def process_model(filename: str, info: dict):
    model_id = info['model_id']
    print(f'\n{"="*60}')
    print(f'Processing: {info["display_name"]} ({filename})')
    print(f'{"="*60}')

    with open(filename) as f:
        samples = [json.loads(line) for line in f]

    print(f'  Loaded {len(samples)} samples')

    # Compute metrics
    metrics = compute_metrics_for_samples(samples)

    # Add meta
    total_audio = sum(s.get('duration', 0) for s in samples)
    metrics['__meta__'] = {
        'checkpoint': f'api/{model_id}',
        'checkpoint_name': 'baseline',
        'model_id': model_id,
        'model_type': info['model_type'],
        'dataset': 'BayAreaBoys/indic-asr-benchmark-6k',
        'batch_size': 1,
        'inference_time_sec': 0,
        'total_audio_sec': round(total_audio, 2),
        'rtf': 0,
        'timestamp': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
        'normalization_version': 'v1',
        'framework': 'api',
    }

    # Print summary
    o = metrics['__overall__']
    print(f'  Overall: wer_raw={o["wer_raw"]:.2f}  wer_norm={o["wer_norm"]:.2f}  wer_numcanon={o["wer_numcanon"]:.2f}  cer_norm={o["cer_norm"]:.2f}')

    # Build sample analysis
    sample_analysis = build_sample_analysis(samples)

    # Write outputs
    out_dir = OUTPUT_BASE / model_id / 'baseline'
    out_dir.mkdir(parents=True, exist_ok=True)

    with open(out_dir / 'metrics.json', 'w') as f:
        json.dump(metrics, f, indent=2, ensure_ascii=False)

    with open(out_dir / 'sample_analysis.json', 'w') as f:
        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)

    print(f'  Written to {out_dir}/')
    return metrics

if __name__ == '__main__':
    results = {}
    for filename, info in MODELS.items():
        fpath = Path(filename)
        if not fpath.exists():
            fpath = Path('/home/ubuntu/training/api_results') / filename
        if not fpath.exists():
            print(f'SKIP: {filename} not found')
            continue
        results[info['model_id']] = process_model(str(fpath), info)

    # Print comparison table
    print(f'\n\n{"="*80}')
    print('COMPARISON TABLE (all models)')
    print(f'{"="*80}')
    print(f'{"Model":<30} {"WER Raw":>8} {"WER Norm":>9} {"WER Num":>8} {"CER Norm":>9}')
    print('-' * 70)
    for mid, m in sorted(results.items(), key=lambda x: x[1]['__overall__']['wer_norm']):
        o = m['__overall__']
        print(f'{mid:<30} {o["wer_raw"]:>7.2f}% {o["wer_norm"]:>8.2f}% {o["wer_numcanon"]:>7.2f}% {o["cer_norm"]:>8.2f}%')


File created successfully at: /home/ubuntu/training/api_results/compute_metrics.py

"""
Compute all 4 metric tiers for API baseline models.
Normalization version: v1
"""
import json
import re
import unicodedata
import sys
from collections import defaultdict
from pathlib import Path
from datetime import datetime, timezone

from jiwer import wer as compute_wer, cer as compute_cer

# ── Normalization pipeline ──

# Zero-width characters to strip
ZW_CHARS = re.compile(r'[\u200b-\u200f\u2028-\u202f\ufeff\u00ad]')

# Punctuation sets
# Common punctuation (ASCII + Unicode general)
PUNCT_COMMON = re.compile(r'[!"#$%&\'()*+,\-./:;<=>?@\[\\\]^_`{|}~]')
# Indic punctuation (dandas, etc.) — keep single danda for segmentation? No, remove for norm.
PUNCT_INDIC = re.compile(r'[\u0964\u0965\u0970\u0971]')  # danda, double danda, abbreviation signs
# Extended punctuation
PUNCT_EXTENDED = re.compile(r'[\u2010-\u2027\u2030-\u205e\u2e00-\u2e4f]')
# Quotes normalization
QUOTES = re.compile(r'[\u2018\u2019\u201a\u201b\u201c\u201d\u201e\u201f\u00ab\u00bb]')

# Parenthetical annotations like "(coughing)", "(two seconds pause)"
ANNOTATIONS = re.compile(r'\([^)]*\)')

def norm_raw(text: str) -> str:
    """Step for wer_raw: NFC unicode + trim only."""
    text = unicodedata.normalize('NFC', text)
    text = text.strip()
    return text

def norm_standard(text: str) -> str:
    """Steps for wer_norm: NFKC + strip ZW + whitespace + punct + case fold."""
    # Step 1: NFKC
    text = unicodedata.normalize('NFKC', text)
    # Step 2: Strip zero-width chars
    text = ZW_CHARS.sub('', text)
    # Step 2.5: Remove parenthetical annotations
    text = ANNOTATIONS.sub('', text)
    # Step 3: Normalize whitespace
    text = ' '.join(text.split())
    # Step 4: Standardize quotes/dashes
    text = QUOTES.sub("'", text)
    text = text.replace('\u2014', '-').replace('\u2013', '-').replace('\u2012', '-')
    # Step 5: Remove punctuation
    text = PUNCT_COMMON.sub('', text)
    text = PUNCT_INDIC.sub('', text)
    text = PUNCT_EXTENDED.sub('', text)
    # Step 6: Case fold (affects English only, Indic scripts have no case)
    text = text.lower()
    # Final whitespace cleanup
    text = ' '.join(text.split()).strip()
    return text

def norm_numcanon(text: str) -> str:
    """Steps for wer_numcanon: norm_standard + number canonicalization."""
    text = norm_standard(text)
    # Remove commas in numbers (e.g., "25,000" -> "25000")
    text = re.sub(r'(\d),(\d)', r'\1\2', text)
    # Normalize Devanagari digits to ASCII
    for offset, base in [(0x0966, '0'), (0x09E6, '0'), (0x0A66, '0'),
                          (0x0AE6, '0'), (0x0B66, '0'), (0x0BE6, '0'),
                          (0x0C66, '0'), (0x0CE6, '0'), (0x0D66, '0')]:
        for i in range(10):
            text = text.replace(chr(offset + i), str(i))
    return text

# ── Metric computation ──

def compute_metrics_for_samples(samples: list[dict]) -> dict:
    """Compute all 4 metric tiers for a list of samples."""
    by_lang = defaultdict(list)
    for s in samples:
        by_lang[s['language']].append(s)

    result = {}
    all_ref_raw, all_hyp_raw = [], []
    all_ref_norm, all_hyp_norm = [], []
    all_ref_nc, all_hyp_nc = [], []

    for lang in sorted(by_lang.keys()):
        lang_samples = by_lang[lang]
        refs_raw, hyps_raw = [], []
        refs_norm, hyps_norm = [], []
        refs_nc, hyps_nc = [], []
        empty_count = 0

        for s in lang_samples:
            ref = s['reference']
            hyp = s.get('transcript', '') or ''

            if not hyp.strip():
                empty_count += 1

            rr, hr = norm_raw(ref), norm_raw(hyp)
            rn, hn = norm_standard(ref), norm_standard(hyp)
            rnc, hnc = norm_numcanon(ref), norm_numcanon(hyp)

            # Skip empty refs (shouldn't happen but safety)
            if not rr.strip():
                continue

            refs_raw.append(rr); hyps_raw.append(hr if hr.strip() else '<empty>')
            refs_norm.append(rn if rn else '<empty>'); hyps_norm.append(hn if hn else '<empty>')
            refs_nc.append(rnc if rnc else '<empty>'); hyps_nc.append(hnc if hnc else '<empty>')

        w_raw = compute_wer(refs_raw, hyps_raw) * 100
        w_norm = compute_wer(refs_norm, hyps_norm) * 100
        w_nc = compute_wer(refs_nc, hyps_nc) * 100
        c_norm = compute_cer(refs_norm, hyps_norm) * 100

        result[lang] = {
            'n_samples': len(lang_samples),
            'wer_raw': round(w_raw, 2),
            'wer_norm': round(w_norm, 2),
            'wer_numcanon': round(w_nc, 2),
            'cer_norm': round(c_norm, 2),
            'empty_hypotheses': empty_count,
            'normalization_delta': {
                'raw_to_norm': round(w_raw - w_norm, 2),
                'norm_to_numcanon': round(w_norm - w_nc, 2),
            }
        }

        all_ref_raw.extend(refs_raw); all_hyp_raw.extend(hyps_raw)
        all_ref_norm.extend(refs_norm); all_hyp_norm.extend(hyps_norm)
        all_ref_nc.extend(refs_nc); all_hyp_nc.extend(hyps_nc)

    # Overall (micro-average)
    result['__overall__'] = {
        'n_samples': len(samples),
        'wer_raw': round(compute_wer(all_ref_raw, all_hyp_raw) * 100, 2),
        'wer_norm': round(compute_wer(all_ref_norm, all_hyp_norm) * 100, 2),
        'wer_numcanon': round(compute_wer(all_ref_nc, all_hyp_nc) * 100, 2),
        'cer_norm': round(compute_cer(all_ref_norm, all_hyp_norm) * 100, 2),
    }

    # Macro average
    lang_keys = [k for k in result if not k.startswith('_')]
    n_langs = len(lang_keys)
    result['__macro_avg__'] = {
        'n_languages': n_langs,
        'wer_raw': round(sum(result[l]['wer_raw'] for l in lang_keys) / n_langs, 2),
        'wer_norm': round(sum(result[l]['wer_norm'] for l in lang_keys) / n_langs, 2),
        'wer_numcanon': round(sum(result[l]['wer_numcanon'] for l in lang_keys) / n_langs, 2),
        'cer_norm': round(sum(result[l]['cer_norm'] for l in lang_keys) / n_langs, 2),
    }

    return result

def build_sample_analysis(samples: list[dict]) -> list[dict]:
    """Build sample_analysis.json entries."""
    out = []
    for s in samples:
        ref = s['reference']
        hyp = s.get('transcript', '') or ''
        rn = norm_standard(ref)
        hn = norm_standard(hyp)
        rnc = norm_numcanon(ref)
        hnc = norm_numcanon(hyp)

        flags = []
        if ref == hyp:
            flags.append('exact_match')
        if rn == hn:
            flags.append('exact_match_norm')
        if ref != hyp and rn == hn:
            flags.append('punctuation_only_diff')
        if not hyp.strip():
            flags.append('empty_hypothesis')
        # Numeric mismatch: digits present in either
        if re.search(r'\d', ref) or re.search(r'\d', hyp):
            if rn != hn:
                flags.append('numeric_mismatch')

        out.append({
            'id': s['id'],
            'language': s['language'],
            'reference': ref,
            'hypothesis': hyp,
            'ref_norm': rn,
            'hyp_norm': hn,
            'ref_numcanon': rnc,
            'hyp_numcanon': hnc,
            'detected_language': s.get('detected_lang', ''),
            'flags': flags,
        })
    return out

# ── Model registry ──

MODELS = {
    'elevenlabs_full_6k.jsonl': {
        'model_id': 'elevenlabs-scribe-v2',
        'display_name': 'ElevenLabs Scribe v2',
        'model_type': 'elevenlabs-scribe-v2',
    },
    'elevenlabs_v1_full_6k.jsonl': {
        'model_id': 'elevenlabs-scribe-v1',
        'display_name': 'ElevenLabs Scribe v1',
        'model_type': 'elevenlabs-scribe-v1',
    },
    'gemini_3_flash_strict_full_6k.jsonl': {
        'model_id': 'gemini-3-flash-strict',
        'display_name': 'Gemini 3 Flash (Strict)',
        'model_type': 'gemini-3-flash-preview',
    },
    'gemini_gemini_3_flash_preview_full_6k.jsonl': {
        'model_id': 'gemini-3-flash-preview',
        'display_name': 'Gemini 3 Flash (Preview)',
        'model_type': 'gemini-3-flash-preview',
    },
    'gemini_flash_v2_full_6k.jsonl': {
        'model_id': 'gemini-2.5-flash',
        'display_name': 'Gemini 2.5 Flash',
        'model_type': 'gemini-2.5-flash',
    },
    'gemini_gemini_3_1_pro_preview_full_6k.jsonl': {
        'model_id': 'gemini-3.1-pro',
        'display_name': 'Gemini 3.1 Pro',
        'model_type': 'gemini-3.1-pro-preview',
    },
    'sarvam_full_6k.jsonl': {
        'model_id': 'sarvam-saaras-v3',
        'display_name': 'Sarvam Saaras v3',
        'model_type': 'saaras:v3',
    },
}

OUTPUT_BASE = Path('/home/ubuntu/training/benchmark_outputs')

def process_model(filename: str, info: dict):
    model_id = info['model_id']
    print(f'\n{"="*60}')
    print(f'Processing: {info["display_name"]} ({filename})')
    print(f'{"="*60}')

    with open(filename) as f:
        samples = [json.loads(line) for line in f]

    print(f'  Loaded {len(samples)} samples')

    # Compute metrics
    metrics = compute_metrics_for_samples(samples)

    # Add meta
    total_audio = sum(s.get('duration', 0) for s in samples)
    metrics['__meta__'] = {
        'checkpoint': f'api/{model_id}',
        'checkpoint_name': 'baseline',
        'model_id': model_id,
        'model_type': info['model_type'],
        'dataset': 'BayAreaBoys/indic-asr-benchmark-6k',
        'batch_size': 1,
        'inference_time_sec': 0,
        'total_audio_sec': round(total_audio, 2),
        'rtf': 0,
        'timestamp': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
        'normalization_version': 'v1',
        'framework': 'api',
    }

    # Print summary
    o = metrics['__overall__']
    print(f'  Overall: wer_raw={o["wer_raw"]:.2f}  wer_norm={o["wer_norm"]:.2f}  wer_numcanon={o["wer_numcanon"]:.2f}  cer_norm={o["cer_norm"]:.2f}')

    # Build sample analysis
    sample_analysis = build_sample_analysis(samples)

    # Write outputs
    out_dir = OUTPUT_BASE / model_id / 'baseline'
    out_dir.mkdir(parents=True, exist_ok=True)

    with open(out_dir / 'metrics.json', 'w') as f:
        json.dump(metrics, f, indent=2, ensure_ascii=False)

    with open(out_dir / 'sample_analysis.json', 'w') as f:
        json.dump(sample_analysis, f, indent=2, ensure_ascii=False)

    print(f'  Written to {out_dir}/')
    return metrics

if __name__ == '__main__':
    results = {}
    for filename, info in MODELS.items():
        fpath = Path(filename)
        if not fpath.exists():
            fpath = Path('/home/ubuntu/training/api_results') / filename
        if not fpath.exists():
            print(f'SKIP: {filename} not found')
            continue
        results[info['model_id']] = process_model(str(fpath), info)

    # Print comparison table
    print(f'\n\n{"="*80}')
    print('COMPARISON TABLE (all models)')
    print(f'{"="*80}')
    print(f'{"Model":<30} {"WER Raw":>8} {"WER Norm":>9} {"WER Num":>8} {"CER Norm":>9}')
    print('-' * 70)
    for mid, m in sorted(results.items(), key=lambda x: x[1]['__overall__']['wer_norm']):
        o = m['__overall__']
        print(f'{mid:<30} {o["wer_raw"]:>7.2f}% {o["wer_norm"]:>8.2f}% {o["wer_numcanon"]:>7.2f}% {o["cer_norm"]:>8.2f}%')


Bash

Compute metrics for all API models


============================================================
Processing: ElevenLabs Scribe v2 (elevenlabs_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=21.94  wer_norm=17.74  wer_numcanon=17.74  cer_norm=5.28
  Written to /home/ubuntu/training/benchmark_outputs/elevenlabs-scribe-v2/baseline/

============================================================
Processing: ElevenLabs Scribe v1 (elevenlabs_v1_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=25.82  wer_norm=21.11  wer_numcanon=21.11  cer_norm=6.29
  Written to /home/ubuntu/training/benchmark_outputs/elevenlabs-scribe-v1/baseline/

============================================================
Processing: Gemini 3 Flash (Strict) (gemini_3_flash_strict_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=27.62  wer_norm=20.12  wer_numcanon=20.12  cer_norm=5.79
  Written to /home/ubuntu/training/benchmark_outputs/gemini-3-flash-strict/baseline/

============================================================
Processing: Gemini 3 Flash (Preview) (gemini_gemini_3_flash_preview_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=71.47  wer_norm=67.68  wer_numcanon=67.68  cer_norm=95.00
  Written to /home/ubuntu/training/benchmark_outputs/gemini-3-flash-preview/baseline/

============================================================
Processing: Gemini 2.5 Flash (gemini_flash_v2_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=63.80  wer_norm=57.71  wer_numcanon=57.71  cer_norm=63.57
  Written to /home/ubuntu/training/benchmark_outputs/gemini-2.5-flash/baseline/

============================================================
Processing: Gemini 3.1 Pro (gemini_gemini_3_1_pro_preview_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=20.50  wer_norm=18.98  wer_numcanon=18.98  cer_norm=4.92
  Written to /home/ubuntu/training/benchmark_outputs/gemini-3.1-pro/baseline/

============================================================
Processing: Sarvam Saaras v3 (sarvam_full_6k.jsonl)
============================================================
  Loaded 6000 samples
  Overall: wer_raw=26.36  wer_norm=18.26  wer_numcanon=18.26  cer_norm=5.01
  Written to /home/ubuntu/training/benchmark_outputs/sarvam-saaras-v3/baseline/

================================================================================
COMPARISON TABLE (all models)
================================================================================
Model                           WER Raw  WER Norm  WER Num  CER Norm
----------------------------------------------------------------------
elevenlabs-scribe-v2             21.94%    17.74%   17.74%     5.28%
sarvam-saaras-v3                 26.36%    18.26%   18.26%     5.01%
gemini-3.1-pro                   20.50%    18.98%   18.98%     4.92%
gemini-3-flash-strict            27.62%    20.12%   20.12%     5.79%
elevenlabs-scribe-v1             25.82%    21.11%   21.11%     6.29%
gemini-2.5-flash                 63.80%    57.71%   57.71%    63.57%
gemini-3-flash-preview           71.47%    67.68%   67.68%    95.00%


Interesting — Gemini 2.5 Flash and 3 Flash Preview have terrible scores (likely partial/failed transcriptions). The top models are ElevenLabs Scribe v2, Sarvam, and Gemini 3.1 Pro. Now let me also recompute our own models with the same normalization pipeline for true apples-to-apples:


Write

"""
Recompute our models (qwen3-asr, gemma3n-e2b) using the SAME v1 normalization pipeline
so comparisons against API baselines are apples-to-apples.
"""
import json
from pathlib import Path
from datetime import datetime, timezone
from compute_metrics import compute_metrics_for_samples, build_sample_analysis, norm_raw, norm_standard

OUTPUT_BASE = Path('/home/ubuntu/training/benchmark_outputs')

OUR_MODELS = {
    'qwen3-asr': ['ckpt-24000', 'ckpt-72000'],
    'gemma3n-e2b': ['ckpt-10000', 'ckpt-20000'],
}

for model_id, checkpoints in OUR_MODELS.items():
    for ckpt in checkpoints:
        sa_path = OUTPUT_BASE / model_id / ckpt / 'sample_analysis.json'
        if not sa_path.exists():
            print(f'SKIP: {sa_path} not found')
            continue

        print(f'\nRecomputing {model_id}/{ckpt}...')
        with open(sa_path) as f:
            raw_samples = json.load(f)

        # Convert old sample format to match our compute pipeline
        samples = []
        for s in raw_samples:
            samples.append({
                'id': s['id'],
                'language': s['language'],
                'reference': s['reference'],
                'transcript': s['hypothesis'],
                'detected_lang': s.get('detected_language', ''),
            })

        metrics = compute_metrics_for_samples(samples)

        # Read existing meta
        old_metrics_path = OUTPUT_BASE / model_id / ckpt / 'metrics.json'
        old_meta = {}
        if old_metrics_path.exists():
            with open(old_metrics_path) as f:
                old_data = json.load(f)
                old_meta = old_data.get('__meta__', {})

        metrics['__meta__'] = {
            **old_meta,
            'model_id': model_id,
            'checkpoint_name': ckpt,
            'normalization_version': 'v1',
            'timestamp': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
        }

        # Write updated metrics
        with open(old_metrics_path, 'w') as f:
            json.dump(metrics, f, indent=2, ensure_ascii=False)

        # Rebuild sample_analysis with new schema
        new_samples = build_sample_analysis(samples)
        with open(sa_path, 'w') as f:
            json.dump(new_samples, f, indent=2, ensure_ascii=False)

        o = metrics['__overall__']
        print(f'  {model_id}/{ckpt}: wer_raw={o["wer_raw"]:.2f}  wer_norm={o["wer_norm"]:.2f}  wer_numcanon={o["wer_numcanon"]:.2f}  cer_norm={o["cer_norm"]:.2f}')


File created successfully at: /home/ubuntu/training/api_results/recompute_ours.py

"""
Recompute our models (qwen3-asr, gemma3n-e2b) using the SAME v1 normalization pipeline
so comparisons against API baselines are apples-to-apples.
"""
import json
from pathlib import Path
from datetime import datetime, timezone
from compute_metrics import compute_metrics_for_samples, build_sample_analysis, norm_raw, norm_standard

OUTPUT_BASE = Path('/home/ubuntu/training/benchmark_outputs')

OUR_MODELS = {
    'qwen3-asr': ['ckpt-24000', 'ckpt-72000'],
    'gemma3n-e2b': ['ckpt-10000', 'ckpt-20000'],
}

for model_id, checkpoints in OUR_MODELS.items():
    for ckpt in checkpoints:
        sa_path = OUTPUT_BASE / model_id / ckpt / 'sample_analysis.json'
        if not sa_path.exists():
            print(f'SKIP: {sa_path} not found')
            continue

        print(f'\nRecomputing {model_id}/{ckpt}...')
        with open(sa_path) as f:
            raw_samples = json.load(f)

        # Convert old sample format to match our compute pipeline
        samples = []
        for s in raw_samples:
            samples.append({
                'id': s['id'],
                'language': s['language'],
                'reference': s['reference'],
                'transcript': s['hypothesis'],
                'detected_lang': s.get('detected_language', ''),
            })

        metrics = compute_metrics_for_samples(samples)

        # Read existing meta
        old_metrics_path = OUTPUT_BASE / model_id / ckpt / 'metrics.json'
        old_meta = {}
        if old_metrics_path.exists():
            with open(old_metrics_path) as f:
                old_data = json.load(f)
                old_meta = old_data.get('__meta__', {})

        metrics['__meta__'] = {
            **old_meta,
            'model_id': model_id,
            'checkpoint_name': ckpt,
            'normalization_version': 'v1',
            'timestamp': datetime.now(timezone.utc).strftime('%Y-%m-%dT%H:%M:%SZ'),
        }

        # Write updated metrics
        with open(old_metrics_path, 'w') as f:
            json.dump(metrics, f, indent=2, ensure_ascii=False)

        # Rebuild sample_analysis with new schema
        new_samples = build_sample_analysis(samples)
        with open(sa_path, 'w') as f:
            json.dump(new_samples, f, indent=2, ensure_ascii=False)

        o = metrics['__overall__']
        print(f'  {model_id}/{ckpt}: wer_raw={o["wer_raw"]:.2f}  wer_norm={o["wer_norm"]:.2f}  wer_numcanon={o["wer_numcanon"]:.2f}  cer_norm={o["cer_norm"]:.2f}')


Bash

Recompute our models with v1 normalization


Recomputing qwen3-asr/ckpt-24000...
  qwen3-asr/ckpt-24000: wer_raw=40.81  wer_norm=39.12  wer_numcanon=39.12  cer_norm=13.64

Recomputing qwen3-asr/ckpt-72000...
  qwen3-asr/ckpt-72000: wer_raw=38.20  wer_norm=36.77  wer_numcanon=36.77  cer_norm=12.40

Recomputing gemma3n-e2b/ckpt-10000...
  gemma3n-e2b/ckpt-10000: wer_raw=37.33  wer_norm=36.24  wer_numcanon=36.24  cer_norm=17.38

Recomputing gemma3n-e2b/ckpt-20000...
  gemma3n-e2b/ckpt-20000: wer_raw=34.97  wer_norm=33.95  wer_numcanon=33.95  cer_norm=15.25


All metrics recomputed with the same v1 normalization. Let me verify what's in benchmark_outputs now:


List all model directories


Good — permissions need updating for nginx, then I need to update the dashboard MODEL_REGISTRY and add the baselines comparison view. Let me do this in parallel:


chmod -R o+rx /home/ubuntu/training/benchmark_outputs/

Fix permissions for nginx


(Bash completed with no output)


Now update the dashboard types to include the new API baseline models:


   100→  top_substitutions: SubstitutionEntry[];
   101→  top_insertions: WordCountEntry[];
   102→  top_deletions: WordCountEntry[];
   103→  error_buckets: ErrorBuckets;
   104→  examples: {
   105→    worst_samples: string[];
   106→    best_samples: string[];
   107→    numeric_mismatch_samples: string[];
   108→    entity_mismatch_samples: string[];
   109→  };
   110→}
   111→
   112→export interface ErrorAnalysisSummary {
   113→  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
   114→  primary_error_source: string;
   115→  numeric_verbalization_impact: string;
   116→  formatting_impact: string;
   117→  worst_languages: string[];
   118→  best_languages: string[];
   119→}
   120→
   121→export type ErrorAnalysisFile = {
   122→  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
   123→} & {
   124→  __summary__: ErrorAnalysisSummary;
   125→};
   126→
   127→export interface ModelInfo {
   128→  id: string;
   129→  displayName: string;
   130→  checkpoints: string[];
   131→}
   132→
   133→export interface CheckpointComparison {
   134→  language: string;
   135→  [key: string]: number | string;
   136→}
   137→
   138→// === Constants ===
   139→

  top_substitutions: SubstitutionEntry[];
  top_insertions: WordCountEntry[];
  top_deletions: WordCountEntry[];
  error_buckets: ErrorBuckets;
  examples: {
    worst_samples: string[];
    best_samples: string[];
    numeric_mismatch_samples: string[];
    entity_mismatch_samples: string[];
  };
}

export interface ErrorAnalysisSummary {
  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
  primary_error_source: string;
  numeric_verbalization_impact: string;
  formatting_impact: string;
  worst_languages: string[];
  best_languages: string[];
}

export type ErrorAnalysisFile = {
  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
} & {
  __summary__: ErrorAnalysisSummary;
};

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string;
}

// === Constants ===


Edit

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
};

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
  "elevenlabs-scribe-v2": "ElevenLabs Scribe v2",
  "elevenlabs-scribe-v1": "ElevenLabs Scribe v1",
  "gemini-3-flash-strict": "Gemini 3 Flash (Strict)",
  "gemini-3-flash-preview": "Gemini 3 Flash (Preview)",
  "gemini-2.5-flash": "Gemini 2.5 Flash",
  "gemini-3.1-pro": "Gemini 3.1 Pro",
  "sarvam-saaras-v3": "Sarvam Saaras v3",
};

// Models grouped by category
export const OUR_MODELS = ["qwen3-asr", "gemma3n-e2b", "parakeet-1.1b"] as const;
export const BASELINE_MODELS = [
  "elevenlabs-scribe-v2", "elevenlabs-scribe-v1",
  "gemini-3-flash-strict", "gemini-3-flash-preview", "gemini-2.5-flash", "gemini-3.1-pro",
  "sarvam-saaras-v3",
] as const;


The file /home/ubuntu/training/asr-benchmark-dashboard/src/lib/types.ts has been updated successfully.

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
  "elevenlabs-scribe-v2": "ElevenLabs Scribe v2",
  "elevenlabs-scribe-v1": "ElevenLabs Scribe v1",
  "gemini-3-flash-strict": "Gemini 3 Flash (Strict)",
  "gemini-3-flash-preview": "Gemini 3 Flash (Preview)",
  "gemini-2.5-flash": "Gemini 2.5 Flash",
  "gemini-3.1-pro": "Gemini 3.1 Pro",
  "sarvam-saaras-v3": "Sarvam Saaras v3",
};

// Models grouped by category
export const OUR_MODELS = ["qwen3-asr", "gemma3n-e2b", "parakeet-1.1b"] as const;
export const BASELINE_MODELS = [
  "elevenlabs-scribe-v2", "elevenlabs-scribe-v1",
  "gemini-3-flash-strict", "gemini-3-flash-preview", "gemini-2.5-flash", "gemini-3.1-pro",
  "sarvam-saaras-v3",
] as const;

// === Metric tiers ===
// wer_raw:      minimal normalization, strict fidelity
// wer_norm:     PRIMARY metric — NFKC + whitespace + punctuation + case fold
// wer_numcanon: wer_norm + number canonicalization
// cer_norm:     character-level normalized — critical for Indic scripts

export interface NormalizationDelta {
  raw_to_norm: number;       // wer_raw - wer_norm (negative = norm helped)
  norm_to_numcanon: number;  // wer_norm - wer_numcanon
}

export interface LanguageMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
  empty_hypotheses: number;
  normalization_delta: NormalizationDelta;
}

export interface AggregateMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MacroAvgMetrics {
  n_languages: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MetaMeta {
  checkpoint: string;
  checkpoint_name: string;
  model_id: string;
  model_type?: string;
  dataset: string;
  batch_size?: number;
  force_language?: boolean;
  inference_time_sec: number;
  total_audio_sec: number;
  rtf: number;
  timestamp?: string;
  gpu?: string;
  framework?: string;
  normalization_version?: string;
  jiwer_version?: string;
}

export type MetricsFile = {
  [language: string]: LanguageMetrics | AggregateMetrics | MacroAvgMetrics | MetaMeta;
} & {
  __overall__: AggregateMetrics;
  __macro_avg__: MacroAvgMetrics;
  __meta__: MetaMeta;
};

export interface SampleAnalysis {
  id: string;
  language: string;
  reference: string;
  hypothesis: string;
  ref_norm?: string;
  hyp_norm?: string;
  ref_numcanon?: string;
  hyp_numcanon?: string;
  detected_language?: string;
  wer_raw?: number;
  wer_norm?: number;
  flags?: string[];
}

export interface ErrorBuckets {
  numeric_mismatch_count: number;
  punctuation_only_count: number;
  spacing_tokenization_count: number;
  entity_mismatch_count: number;
  script_confusion_count: number;
  empty_hypothesis_count: number;
}

export interface SubstitutionEntry {
  ref: string;
  hyp: string;
  count: number;
}

export interface WordCountEntry {
  word: string;
  count: number;
}

export interface LanguageErrorAnalysis {
  top_substitutions: SubstitutionEntry[];
  top_insertions: WordCountEntry[];
  top_deletions: WordCountEntry[];
  error_buckets: ErrorBuckets;
  examples: {
    worst_samples: string[];
    best_samples: string[];
    numeric_mismatch_samples: string[];
    entity_mismatch_samples: string[];
  };
}

export interface ErrorAnalysisSummary {
  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
  primary_error_source: string;
  numeric_verbalization_impact: string;
  formatting_impact: string;
  worst_languages: string[];
  best_languages: string[];
}

export type ErrorAnalysisFile = {
  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
} & {
  __summary__: ErrorAnalysisSummary;
};

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string;
}

// === Constants ===

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
};

export const LANGUAGES = [
  "assamese", "bengali", "english", "gujarati", "hindi", "kannada",
  "malayalam", "marathi", "odia", "punjabi", "tamil", "telugu",
] as const;

export const LANGUAGE_CODES: Record<string, string> = {
  assamese: "as", bengali: "bn", english: "en", gujarati: "gu",
  hindi: "hi", kannada: "kn", malayalam: "ml", marathi: "mr",
  odia: "or", punjabi: "pa", tamil: "ta", telugu: "te",
};

export const METRIC_TIERS = ["wer_raw", "wer_norm", "wer_numcanon", "cer_norm"] as const;
export type MetricTier = typeof METRIC_TIERS[number];

export const METRIC_LABELS: Record<MetricTier, string> = {
  wer_raw: "WER Raw",
  wer_norm: "WER Norm",
  wer_numcanon: "WER NumCanon",
  cer_norm: "CER Norm",
};

export const METRIC_DESCRIPTIONS: Record<MetricTier, string> = {
  wer_raw: "Strict transcript fidelity",
  wer_norm: "Primary ASR metric",
  wer_numcanon: "Numeric robustness",
  cer_norm: "Script-sensitive (Indic)",
};

export const CHART_COLORS = [
  "#6366f1", "#8b5cf6", "#a78bfa", "#c084fc",
  "#e879f9", "#f472b6", "#fb7185", "#f87171",
  "#fb923c", "#fbbf24", "#a3e635", "#34d399",
];

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};

// === Backward compatibility ===
// Converts old-schema metrics into new-schema shape so dashboard works
// with both old data (pre-migration) and new data.

export interface OldLanguageMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
  empty_hypotheses: number;
}

export function isOldSchema(obj: Record<string, unknown>): boolean {
  return "wer" in obj && !("wer_raw" in obj);
}

export function migrateLanguageMetrics(old: OldLanguageMetrics): LanguageMetrics {
  return {
    n_samples: old.n_samples,
    wer_raw: old.wer,
    wer_norm: old.wer_normalized,
    wer_numcanon: old.wer_normalized, // no numcanon data in old schema, use norm as fallback
    cer_norm: old.cer_normalized,
    empty_hypotheses: old.empty_hypotheses,
    normalization_delta: {
      raw_to_norm: old.wer - old.wer_normalized,
      norm_to_numcanon: 0,
    },
  };
}

+  "elevenlabs-scribe-v2": "ElevenLabs Scribe v2",

+  "elevenlabs-scribe-v1": "ElevenLabs Scribe v1",

+  "gemini-3-flash-strict": "Gemini 3 Flash (Strict)",

+  "gemini-3-flash-preview": "Gemini 3 Flash (Preview)",

+  "gemini-2.5-flash": "Gemini 2.5 Flash",

+  "sarvam-saaras-v3": "Sarvam Saaras v3",

+export const OUR_MODELS = ["qwen3-asr", "gemma3n-e2b", "parakeet-1.1b"] as const;

+  "elevenlabs-scribe-v2", "elevenlabs-scribe-v1",

+  "gemini-3-flash-strict", "gemini-3-flash-preview", "gemini-2.5-flash", "gemini-3.1-pro",


export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
  "elevenlabs-scribe-v2": "#f472b6",
  "elevenlabs-scribe-v1": "#fb7185",
  "gemini-3-flash-strict": "#fbbf24",
  "gemini-3-flash-preview": "#fb923c",
  "gemini-2.5-flash": "#f87171",
  "gemini-3.1-pro": "#a3e635",
  "sarvam-saaras-v3": "#22d3ee",
};


export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
  "elevenlabs-scribe-v2": "#f472b6",
  "elevenlabs-scribe-v1": "#fb7185",
  "gemini-3-flash-strict": "#fbbf24",
  "gemini-3-flash-preview": "#fb923c",
  "gemini-2.5-flash": "#f87171",
  "gemini-3.1-pro": "#a3e635",
  "sarvam-saaras-v3": "#22d3ee",
};

// === Metric tiers ===
// wer_raw:      minimal normalization, strict fidelity
// wer_norm:     PRIMARY metric — NFKC + whitespace + punctuation + case fold
// wer_numcanon: wer_norm + number canonicalization
// cer_norm:     character-level normalized — critical for Indic scripts

export interface NormalizationDelta {
  raw_to_norm: number;       // wer_raw - wer_norm (negative = norm helped)
  norm_to_numcanon: number;  // wer_norm - wer_numcanon
}

export interface LanguageMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
  empty_hypotheses: number;
  normalization_delta: NormalizationDelta;
}

export interface AggregateMetrics {
  n_samples: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MacroAvgMetrics {
  n_languages: number;
  wer_raw: number;
  wer_norm: number;
  wer_numcanon: number;
  cer_norm: number;
}

export interface MetaMeta {
  checkpoint: string;
  checkpoint_name: string;
  model_id: string;
  model_type?: string;
  dataset: string;
  batch_size?: number;
  force_language?: boolean;
  inference_time_sec: number;
  total_audio_sec: number;
  rtf: number;
  timestamp?: string;
  gpu?: string;
  framework?: string;
  normalization_version?: string;
  jiwer_version?: string;
}

export type MetricsFile = {
  [language: string]: LanguageMetrics | AggregateMetrics | MacroAvgMetrics | MetaMeta;
} & {
  __overall__: AggregateMetrics;
  __macro_avg__: MacroAvgMetrics;
  __meta__: MetaMeta;
};

export interface SampleAnalysis {
  id: string;
  language: string;
  reference: string;
  hypothesis: string;
  ref_norm?: string;
  hyp_norm?: string;
  ref_numcanon?: string;
  hyp_numcanon?: string;
  detected_language?: string;
  wer_raw?: number;
  wer_norm?: number;
  flags?: string[];
}

export interface ErrorBuckets {
  numeric_mismatch_count: number;
  punctuation_only_count: number;
  spacing_tokenization_count: number;
  entity_mismatch_count: number;
  script_confusion_count: number;
  empty_hypothesis_count: number;
}

export interface SubstitutionEntry {
  ref: string;
  hyp: string;
  count: number;
}

export interface WordCountEntry {
  word: string;
  count: number;
}

export interface LanguageErrorAnalysis {
  top_substitutions: SubstitutionEntry[];
  top_insertions: WordCountEntry[];
  top_deletions: WordCountEntry[];
  error_buckets: ErrorBuckets;
  examples: {
    worst_samples: string[];
    best_samples: string[];
    numeric_mismatch_samples: string[];
    entity_mismatch_samples: string[];
  };
}

export interface ErrorAnalysisSummary {
  model_diagnosis: "recognition-limited" | "formatting-limited" | "numeric-limited" | "mixed";
  primary_error_source: string;
  numeric_verbalization_impact: string;
  formatting_impact: string;
  worst_languages: string[];
  best_languages: string[];
}

export type ErrorAnalysisFile = {
  [language: string]: LanguageErrorAnalysis | ErrorAnalysisSummary;
} & {
  __summary__: ErrorAnalysisSummary;
};

export interface ModelInfo {
  id: string;
  displayName: string;
  checkpoints: string[];
}

export interface CheckpointComparison {
  language: string;
  [key: string]: number | string;
}

// === Constants ===

export const MODEL_REGISTRY: Record<string, string> = {
  "qwen3-asr": "Qwen3-ASR-1.7B",
  "gemma3n-e2b": "Gemma-3n-E2B-ASR",
  "parakeet-1.1b": "Parakeet-1.1B-Language-Guided",
  "elevenlabs-scribe-v2": "ElevenLabs Scribe v2",
  "elevenlabs-scribe-v1": "ElevenLabs Scribe v1",
  "gemini-3-flash-strict": "Gemini 3 Flash (Strict)",
  "gemini-3-flash-preview": "Gemini 3 Flash (Preview)",
  "gemini-2.5-flash": "Gemini 2.5 Flash",
  "gemini-3.1-pro": "Gemini 3.1 Pro",
  "sarvam-saaras-v3": "Sarvam Saaras v3",
};

// Models grouped by category
export const OUR_MODELS = ["qwen3-asr", "gemma3n-e2b", "parakeet-1.1b"] as const;
export const BASELINE_MODELS = [
  "elevenlabs-scribe-v2", "elevenlabs-scribe-v1",
  "gemini-3-flash-strict", "gemini-3-flash-preview", "gemini-2.5-flash", "gemini-3.1-pro",
  "sarvam-saaras-v3",
] as const;

export const LANGUAGES = [
  "assamese", "bengali", "english", "gujarati", "hindi", "kannada",
  "malayalam", "marathi", "odia", "punjabi", "tamil", "telugu",
] as const;

export const LANGUAGE_CODES: Record<string, string> = {
  assamese: "as", bengali: "bn", english: "en", gujarati: "gu",
  hindi: "hi", kannada: "kn", malayalam: "ml", marathi: "mr",
  odia: "or", punjabi: "pa", tamil: "ta", telugu: "te",
};

export const METRIC_TIERS = ["wer_raw", "wer_norm", "wer_numcanon", "cer_norm"] as const;
export type MetricTier = typeof METRIC_TIERS[number];

export const METRIC_LABELS: Record<MetricTier, string> = {
  wer_raw: "WER Raw",
  wer_norm: "WER Norm",
  wer_numcanon: "WER NumCanon",
  cer_norm: "CER Norm",
};

export const METRIC_DESCRIPTIONS: Record<MetricTier, string> = {
  wer_raw: "Strict transcript fidelity",
  wer_norm: "Primary ASR metric",
  wer_numcanon: "Numeric robustness",
  cer_norm: "Script-sensitive (Indic)",
};

export const CHART_COLORS = [
  "#6366f1", "#8b5cf6", "#a78bfa", "#c084fc",
  "#e879f9", "#f472b6", "#fb7185", "#f87171",
  "#fb923c", "#fbbf24", "#a3e635", "#34d399",
];

export const MODEL_COLORS: Record<string, string> = {
  "qwen3-asr": "#6366f1",
  "gemma3n-e2b": "#8b5cf6",
  "parakeet-1.1b": "#34d399",
};

// === Backward compatibility ===
// Converts old-schema metrics into new-schema shape so dashboard works
// with both old data (pre-migration) and new data.

export interface OldLanguageMetrics {
  n_samples: number;
  wer: number;
  cer: number;
  wer_normalized: number;
  cer_normalized: number;
  empty_hypotheses: number;
}

export function isOldSchema(obj: Record<string, unknown>): boolean {
  return "wer" in obj && !("wer_raw" in obj);
}

export function migrateLanguageMetrics(old: OldLanguageMetrics): LanguageMetrics {
  return {
    n_samples: old.n_samples,
    wer_raw: old.wer,
    wer_norm: old.wer_normalized,
    wer_numcanon: old.wer_normalized, // no numcanon data in old schema, use norm as fallback
    cer_norm: old.cer_normalized,
    empty_hypotheses: old.empty_hypotheses,
    normalization_delta: {
      raw_to_norm: old.wer - old.wer_normalized,
      norm_to_numcanon: 0,
    },
  };
}

"qwen3-asr": "#6366f1",
"gemma3n-e2b": "#8b5cf6",
"parakeet-1.1b": "#34d399",
+  "elevenlabs-scribe-v2": "#f472b6",
+  "elevenlabs-scribe-v1": "#fb7185",
+  "gemini-3-flash-strict": "#fbbf24",
+  "gemini-3-flash-preview": "#fb923c",
+  "gemini-2.5-flash": "#f87171",
+  "gemini-3.1-pro": "#a3e635",
+  "sarvam-saaras-v3": "#22d3ee",
};
// === Backward compatibility ===


Now add the "Baselines" navigation item in the sidebar and create the baselines comparison view. Let me update the sidebar first:


1→"use client";
     2→
     3→import { ModelInfo, MODEL_COLORS } from "@/lib/types";
     4→
     5→interface SidebarProps {
     6→  models: ModelInfo[];
     7→  selectedModel: string | null;
     8→  selectedCheckpoint: string | null;
     9→  activeView: "overview" | "wer-analysis";
    10→  onSelectModel: (id: string) => void;
    11→  onSelectCheckpoint: (ckpt: string) => void;
    12→  onViewChange: (view: "overview" | "wer-analysis") => void;
    13→}
    14→
    15→export default function Sidebar({
    16→  models,
    17→  selectedModel,
    18→  selectedCheckpoint,
    19→  activeView,
    20→  onSelectModel,
    21→  onSelectCheckpoint,
    22→  onViewChange,
    23→}: SidebarProps) {
    24→  return (
    25→
    26→      {/* Logo */}
    27→

    28→

    29→          ASR Benchmark
    30→

    31→

    32→          Indic Language Evaluation
    33→

    34→

    35→
    36→      {/* Navigation */}
    37→

    38→

    39→          Navigation
    40→

    41→        <button
    42→          onClick={() => onViewChange("overview")}
    43→          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors ${
    44→            activeView === "overview"
    45→              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
    46→              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
    47→          }`}
    48→        >
    49→          Overview & Plots
    50→        </button>
    51→        <button
    52→          onClick={() => onViewChange("wer-analysis")}
    53→          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-1 ${
    54→            activeView === "wer-analysis"
    55→              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
    56→              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
    57→          }`}
    58→        >
    59→          WER Analysis
    60→        </button>
    61→

    62→
    63→      {/* Models */}
    64→

    65→

    66→          Models
    67→

    68→        {models.map((model) => (
    69→

    70→            <button
    71→              onClick={() => {
    72→                onSelectModel(model.id);
    73→                if (model.checkpoints.length > 0) {
    74→                  onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);
    75→                }
    76→              }}
    77→              className={`w-full text-left px-3 py-2 rounded text-xs transition-colors flex items-center gap-2 ${
    78→                selectedModel === model.id
    79→                  ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
    80→                  : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
    81→              }`}
    82→            >
    83→
    87→              {model.displayName}
    88→            </button>
    89→
    90→            {selectedModel === model.id && model.checkpoints.length > 0 && (
    91→

    92→                {model.checkpoints.map((ckpt) => (
    93→                  <button
    94→                    key={ckpt}
    95→                    onClick={() => onSelectCheckpoint(ckpt)}
    96→                    className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors ${
    97→                      selectedCheckpoint === ckpt
    98→                        ? "text-[var(--color-foreground)] bg-[var(--color-card)]"
    99→                        : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"
   100→                    }`}
   101→                  >
   102→                    {ckpt}
   103→                  </button>
   104→                ))}
   105→

   106→            )}
   107→
   108→            {model.checkpoints.length === 0 && selectedModel === model.id && (
   109→

   110→                No benchmarks yet
   111→

   112→            )}
   113→

   114→        ))}
   115→

   116→
   117→      {/* Footer */}
   118→

   119→

   120→          Dataset: indic-asr-benchmark-6k
   121→

   122→

   123→          12 languages · 6,000 samples
   124→

   125→

   126→
   127→  );
   128→}
   129→

"use client";

import { ModelInfo, MODEL_COLORS } from "@/lib/types";

interface SidebarProps {
  models: ModelInfo[];
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  activeView: "overview" | "wer-analysis";
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
  onViewChange: (view: "overview" | "wer-analysis") => void;
}

export default function Sidebar({
  models,
  selectedModel,
  selectedCheckpoint,
  activeView,
  onSelectModel,
  onSelectCheckpoint,
  onViewChange,
}: SidebarProps) {
  return (

      {/* Logo */}

          ASR Benchmark

          Indic Language Evaluation

      {/* Navigation */}

          Navigation

        <button
          onClick={() => onViewChange("overview")}
          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors ${
            activeView === "overview"
              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
          }`}
        >
          Overview & Plots
        </button>
        <button
          onClick={() => onViewChange("wer-analysis")}
          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-1 ${
            activeView === "wer-analysis"
              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
          }`}
        >
          WER Analysis
        </button>

      {/* Models */}

          Models

        {models.map((model) => (

            <button
              onClick={() => {
                onSelectModel(model.id);
                if (model.checkpoints.length > 0) {
                  onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);
                }
              }}
              className={`w-full text-left px-3 py-2 rounded text-xs transition-colors flex items-center gap-2 ${
                selectedModel === model.id
                  ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
                  : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
              }`}
            >

              {model.displayName}
            </button>

            {selectedModel === model.id && model.checkpoints.length > 0 && (

                {model.checkpoints.map((ckpt) => (
                  <button
                    key={ckpt}
                    onClick={() => onSelectCheckpoint(ckpt)}
                    className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors ${
                      selectedCheckpoint === ckpt
                        ? "text-[var(--color-foreground)] bg-[var(--color-card)]"
                        : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"
                    }`}
                  >
                    {ckpt}
                  </button>
                ))}

            )}

            {model.checkpoints.length === 0 && selectedModel === model.id && (

                No benchmarks yet

            )}

        ))}

      {/* Footer */}

          Dataset: indic-asr-benchmark-6k

          12 languages · 6,000 samples

  );
}


Write

"use client";

import { ModelInfo, MODEL_COLORS, OUR_MODELS, BASELINE_MODELS } from "@/lib/types";

export type ViewMode = "overview" | "wer-analysis" | "baselines";

interface SidebarProps {
  models: ModelInfo[];
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  activeView: ViewMode;
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
  onViewChange: (view: ViewMode) => void;
}

export default function Sidebar({
  models,
  selectedModel,
  selectedCheckpoint,
  activeView,
  onSelectModel,
  onSelectCheckpoint,
  onViewChange,
}: SidebarProps) {
  const ourModels = models.filter((m) => (OUR_MODELS as readonly string[]).includes(m.id));
  const baselineModels = models.filter((m) => (BASELINE_MODELS as readonly string[]).includes(m.id));

  const navItems: { key: ViewMode; label: string }[] = [
    { key: "overview", label: "Overview & Plots" },
    { key: "wer-analysis", label: "WER Analysis" },
    { key: "baselines", label: "vs Baselines" },
  ];

  return (

      {/* Logo */}

          ASR Benchmark

          Indic Language Evaluation

      {/* Navigation */}

          Navigation

        {navItems.map((item) => (
          <button
            key={item.key}
            onClick={() => onViewChange(item.key)}
            className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-0.5 ${
              activeView === item.key
                ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
                : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
            }`}
          >
            {item.label}
          </button>
        ))}

      {/* Our Models */}

          Our Models

        {ourModels.map((model) => (
          <ModelEntry
            key={model.id}
            model={model}
            selectedModel={selectedModel}
            selectedCheckpoint={selectedCheckpoint}
            onSelectModel={onSelectModel}
            onSelectCheckpoint={onSelectCheckpoint}
          />
        ))}

      {/* Baseline Models */}
      {baselineModels.length > 0 && (

            API Baselines

          {baselineModels.map((model) => (
            <ModelEntry
              key={model.id}
              model={model}
              selectedModel={selectedModel}
              selectedCheckpoint={selectedCheckpoint}
              onSelectModel={onSelectModel}
              onSelectCheckpoint={onSelectCheckpoint}
            />
          ))}

      )}

      {/* Footer */}

          Dataset: indic-asr-benchmark-6k

          12 languages · 6,000 samples

  );
}

function ModelEntry({
  model, selectedModel, selectedCheckpoint, onSelectModel, onSelectCheckpoint,
}: {
  model: ModelInfo;
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
}) {
  return (

      <button
        onClick={() => {
          onSelectModel(model.id);
          if (model.checkpoints.length > 0) {
            onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);
          }
        }}
        className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors flex items-center gap-2 ${
          selectedModel === model.id
            ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
            : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
        }`}
      >

        {model.displayName}
      </button>

      {selectedModel === model.id && model.checkpoints.length > 1 && (

          {model.checkpoints.map((ckpt) => (
            <button
              key={ckpt}
              onClick={() => onSelectCheckpoint(ckpt)}
              className={`w-full text-left px-3 py-1 rounded text-[10px] transition-colors ${
                selectedCheckpoint === ckpt
                  ? "text-[var(--color-foreground)] bg-[var(--color-card)]"
                  : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"
              }`}
            >
              {ckpt}
            </button>
          ))}

      )}

  );
}


The file /home/ubuntu/training/asr-benchmark-dashboard/src/components/Sidebar.tsx has been updated successfully.

"use client";

import { ModelInfo, MODEL_COLORS, OUR_MODELS, BASELINE_MODELS } from "@/lib/types";

export type ViewMode = "overview" | "wer-analysis" | "baselines";

interface SidebarProps {
  models: ModelInfo[];
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  activeView: ViewMode;
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
  onViewChange: (view: ViewMode) => void;
}

export default function Sidebar({
  models,
  selectedModel,
  selectedCheckpoint,
  activeView,
  onSelectModel,
  onSelectCheckpoint,
  onViewChange,
}: SidebarProps) {
  const ourModels = models.filter((m) => (OUR_MODELS as readonly string[]).includes(m.id));
  const baselineModels = models.filter((m) => (BASELINE_MODELS as readonly string[]).includes(m.id));

  const navItems: { key: ViewMode; label: string }[] = [
    { key: "overview", label: "Overview & Plots" },
    { key: "wer-analysis", label: "WER Analysis" },
    { key: "baselines", label: "vs Baselines" },
  ];

  return (

      {/* Logo */}

          ASR Benchmark

          Indic Language Evaluation

      {/* Navigation */}

          Navigation

        {navItems.map((item) => (
          <button
            key={item.key}
            onClick={() => onViewChange(item.key)}
            className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-0.5 ${
              activeView === item.key
                ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
                : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
            }`}
          >
            {item.label}
          </button>
        ))}

      {/* Our Models */}

          Our Models

        {ourModels.map((model) => (
          <ModelEntry
            key={model.id}
            model={model}
            selectedModel={selectedModel}
            selectedCheckpoint={selectedCheckpoint}
            onSelectModel={onSelectModel}
            onSelectCheckpoint={onSelectCheckpoint}
          />
        ))}

      {/* Baseline Models */}
      {baselineModels.length > 0 && (

            API Baselines

          {baselineModels.map((model) => (
            <ModelEntry
              key={model.id}
              model={model}
              selectedModel={selectedModel}
              selectedCheckpoint={selectedCheckpoint}
              onSelectModel={onSelectModel}
              onSelectCheckpoint={onSelectCheckpoint}
            />
          ))}

      )}

      {/* Footer */}

          Dataset: indic-asr-benchmark-6k

          12 languages · 6,000 samples

  );
}

function ModelEntry({
  model, selectedModel, selectedCheckpoint, onSelectModel, onSelectCheckpoint,
}: {
  model: ModelInfo;
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
}) {
  return (

      <button
        onClick={() => {
          onSelectModel(model.id);
          if (model.checkpoints.length > 0) {
            onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);
          }
        }}
        className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors flex items-center gap-2 ${
          selectedModel === model.id
            ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
            : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
        }`}
      >

        {model.displayName}
      </button>

      {selectedModel === model.id && model.checkpoints.length > 1 && (

          {model.checkpoints.map((ckpt) => (
            <button
              key={ckpt}
              onClick={() => onSelectCheckpoint(ckpt)}
              className={`w-full text-left px-3 py-1 rounded text-[10px] transition-colors ${
                selectedCheckpoint === ckpt
                  ? "text-[var(--color-foreground)] bg-[var(--color-card)]"
                  : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"
              }`}
            >
              {ckpt}
            </button>
          ))}

      )}

  );
}

-import { ModelInfo, MODEL_COLORS } from "@/lib/types";

+import { ModelInfo, MODEL_COLORS, OUR_MODELS, BASELINE_MODELS } from "@/lib/types";

+export type ViewMode = "overview" | "wer-analysis" | "baselines";

-  activeView: "overview" | "wer-analysis";

-  onViewChange: (view: "overview" | "wer-analysis") => void;

+  onViewChange: (view: ViewMode) => void;

+  const ourModels = models.filter((m) => (OUR_MODELS as readonly string[]).includes(m.id));

+  const baselineModels = models.filter((m) => (BASELINE_MODELS as readonly string[]).includes(m.id));

+  const navItems: { key: ViewMode; label: string }[] = [

+    { key: "overview", label: "Overview & Plots" },

+    { key: "wer-analysis", label: "WER Analysis" },

+    { key: "baselines", label: "vs Baselines" },

<aside className="w-64 border-r border-[var(--color-border)] bg-[var(--color-background)] flex flex-col h-full overflow-y-auto">

-          onClick={() => onViewChange("overview")}

-          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors ${

-              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"

-              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"

-          onClick={() => onViewChange("wer-analysis")}

-          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-1 ${

-            activeView === "wer-analysis"

+            onClick={() => onViewChange(item.key)}

+            className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-0.5 ${

+                ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"

+                : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"

+      <div className="p-4 border-b border-[var(--color-border)]">

-          <div key={model.id} className="mb-4">

-                onSelectModel(model.id);

-                if (model.checkpoints.length > 0) {

-                  onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);

-              className={`w-full text-left px-3 py-2 rounded text-xs transition-colors flex items-center gap-2 ${

-                selectedModel === model.id

-                className="w-2 h-2 rounded-full flex-shrink-0"

-                style={{ backgroundColor: MODEL_COLORS[model.id] || "#6366f1" }}

-            {selectedModel === model.id && model.checkpoints.length > 0 && (

-              <div className="ml-6 mt-1 space-y-0.5">

-                {model.checkpoints.map((ckpt) => (

-                    onClick={() => onSelectCheckpoint(ckpt)}

-                    className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors ${

-                      selectedCheckpoint === ckpt

-                        ? "text-[var(--color-foreground)] bg-[var(--color-card)]"

-                        : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"

-            {model.checkpoints.length === 0 && selectedModel === model.id && (

-              <p className="ml-6 mt-1 text-[10px] text-[var(--color-muted)] italic px-3">

+            selectedModel={selectedModel}

+            selectedCheckpoint={selectedCheckpoint}

+            onSelectModel={onSelectModel}

+            onSelectCheckpoint={onSelectCheckpoint}

+          <p className="text-[10px] uppercase tracking-widest text-[var(--color-muted)] mb-3">

+          {baselineModels.map((model) => (

+              selectedModel={selectedModel}

+              onSelectModel={onSelectModel}

<div className="p-4 border-t border-[var(--color-border)]">

+  model, selectedModel, selectedCheckpoint, onSelectModel, onSelectCheckpoint,

+  onSelectCheckpoint: (ckpt: string) => void;

+          if (model.checkpoints.length > 0) {

+            onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);

+        className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors flex items-center gap-2 ${

+          className="w-2 h-2 rounded-full flex-shrink-0"

+          style={{ backgroundColor: MODEL_COLORS[model.id] || "#6366f1" }}

+      {selectedModel === model.id && model.checkpoints.length > 1 && (

+        <div className="ml-6 mt-0.5 space-y-0.5">

+          {model.checkpoints.map((ckpt) => (

+              onClick={() => onSelectCheckpoint(ckpt)}

+              className={`w-full text-left px-3 py-1 rounded text-[10px] transition-colors ${

+                selectedCheckpoint === ckpt

+                  ? "text-[var(--color-foreground)] bg-[var(--color-card)]"

+                  : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"

"use client";

import { ModelInfo, MODEL_COLORS } from "@/lib/types";

interface SidebarProps {
  models: ModelInfo[];
  selectedModel: string | null;
  selectedCheckpoint: string | null;
  activeView: "overview" | "wer-analysis";
  onSelectModel: (id: string) => void;
  onSelectCheckpoint: (ckpt: string) => void;
  onViewChange: (view: "overview" | "wer-analysis") => void;
}

export default function Sidebar({
  models,
  selectedModel,
  selectedCheckpoint,
  activeView,
  onSelectModel,
  onSelectCheckpoint,
  onViewChange,
}: SidebarProps) {
  return (

      {/* Logo */}

          ASR Benchmark

          Indic Language Evaluation

      {/* Navigation */}

          Navigation

        <button
          onClick={() => onViewChange("overview")}
          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors ${
            activeView === "overview"
              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
          }`}
        >
          Overview & Plots
        </button>
        <button
          onClick={() => onViewChange("wer-analysis")}
          className={`w-full text-left px-3 py-2 rounded text-xs transition-colors mt-1 ${
            activeView === "wer-analysis"
              ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
              : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
          }`}
        >
          WER Analysis
        </button>

      {/* Models */}

          Models

        {models.map((model) => (

            <button
              onClick={() => {
                onSelectModel(model.id);
                if (model.checkpoints.length > 0) {
                  onSelectCheckpoint(model.checkpoints[model.checkpoints.length - 1]);
                }
              }}
              className={`w-full text-left px-3 py-2 rounded text-xs transition-colors flex items-center gap-2 ${
                selectedModel === model.id
                  ? "bg-[var(--color-card-hover)] text-[var(--color-foreground)]"
                  : "text-[var(--color-muted)] hover:text-[var(--color-foreground)] hover:bg-[var(--color-card)]"
              }`}
            >

              {model.displayName}
            </button>

            {selectedModel === model.id && model.checkpoints.length > 0 && (

                {model.checkpoints.map((ckpt) => (
                  <button
                    key={ckpt}
                    onClick={() => onSelectCheckpoint(ckpt)}
                    className={`w-full text-left px-3 py-1.5 rounded text-[11px] transition-colors ${
                      selectedCheckpoint === ckpt
                        ? "text-[var(--color-foreground)] bg-[var(--color-card)]"
                        : "text-[var(--color-muted)] hover:text-[var(--color-accent)]"
                    }`}
                  >
                    {ckpt}
                  </button>
                ))}

            )}

            {model.checkpoints.length === 0 && selectedModel === model.id && (

                No benchmarks yet

            )}

        ))}

      {/* Footer */}

          Dataset: indic-asr-benchmark-6k

          12 languages · 6,000 samples

  );
}


Now update the page.tsx to add the baselines comparison view and use the new ViewMode type:


     1→"use client";
     2→
     3→import { useState, useEffect, useCallback } from "react";
     4→import Sidebar from "@/components/Sidebar";
     5→import MetricCard from "@/components/MetricCard";
     6→import LanguageBarChart from "@/components/LanguageBarChart";
     7→import CheckpointLineChart from "@/components/CheckpointLineChart";
     8→import CrossModelChart from "@/components/CrossModelChart";
     9→import LanguageHeatmap from "@/components/LanguageHeatmap";
    10→import SampleTable from "@/components/SampleTable";
    11→import {
    12→  ModelInfo, LANGUAGES, MODEL_REGISTRY,
    13→  MetricTier, METRIC_TIERS, METRIC_LABELS, METRIC_DESCRIPTIONS,
    14→  LanguageMetrics, AggregateMetrics,
    15→} from "@/lib/types";
    16→
    17→type ViewMode = "overview" | "wer-analysis";
    18→
    19→interface MetaInfo {
    20→  checkpoint?: string;

"use client";

import { useState, useEffect, useCallback } from "react";
import Sidebar from "@/components/Sidebar";
import MetricCard from "@/components/MetricCard";
import LanguageBarChart from "@/components/LanguageBarChart";
import CheckpointLineChart from "@/components/CheckpointLineChart";
import CrossModelChart from "@/components/CrossModelChart";
import LanguageHeatmap from "@/components/LanguageHeatmap";
import SampleTable from "@/components/SampleTable";
import {
  ModelInfo, LANGUAGES, MODEL_REGISTRY,
  MetricTier, METRIC_TIERS, METRIC_LABELS, METRIC_DESCRIPTIONS,
  LanguageMetrics, AggregateMetrics,
} from "@/lib/types";

type ViewMode = "overview" | "wer-analysis";

interface MetaInfo {
  checkpoint?: string;


The page.tsx is long. Let me make the targeted edits:


Edit

import Sidebar from "@/components/Sidebar";
import MetricCard from "@/components/MetricCard";
import LanguageBarChart from "@/components/LanguageBarChart";
import CheckpointLineChart from "@/components/CheckpointLineChart";
import CrossModelChart from "@/components/CrossModelChart";
import LanguageHeatmap from "@/components/LanguageHeatmap";
import SampleTable from "@/components/SampleTable";
import {
  ModelInfo, LANGUAGES, MODEL_REGISTRY,
  MetricTier, METRIC_TIERS, METRIC_LABELS, METRIC_DESCRIPTIONS,
  LanguageMetrics, AggregateMetrics,
} from "@/lib/types";

type ViewMode = "overview" | "wer-analysis";

import Sidebar, { ViewMode } from "@/components/Sidebar";
import MetricCard from "@/components/MetricCard";
import LanguageBarChart from "@/components/LanguageBarChart";
import CheckpointLineChart from "@/components/CheckpointLineChart";
import CrossModelChart from "@/components/CrossModelChart";
import LanguageHeatmap from "@/components/LanguageHeatmap";
import SampleTable from "@/components/SampleTable";
import BaselinesView from "@/components/BaselinesView";
import {
  ModelInfo, LANGUAGES, MODEL_REGISTRY,
  MetricTier, METRIC_TIERS, METRIC_LABELS, METRIC_DESCRIPTIONS,
  LanguageMetrics, AggregateMetrics,
} from "@/lib/types";