https://huggingface.co/google/gemma-3-270m/tree/main and https://huggingface.co/Qwen/Qwen3-ASR-1.7B, test the tokenizer performance of these both models. efficiency and other tests too. rigorous.


Let me first check the current environment and then build a rigorous tokenizer benchmark.


Bash

ls /home/ubuntu/training/ && python3 -c "import transformers; print(transformers.__version__)" 2>&1

Check workspace and transformers version


pip list 2>/dev/null | grep -iE "transformers|tokenizers|sentencepiece|tiktoken"

Check installed tokenizer packages


api_results
api_results.zip
asr-benchmark-dashboard
benchmark_gemma3n_asr.py
benchmark_maya_asr_tdt.py
benchmark_outputs
benchmark_qwen3_asr.py
benchmark_results
benchmark_results_gemma3n
benchmark_schema
checkpoints
convert_to_schema_v1.py
datasets
gemma3n-E2B-asr-ft
generate_schema_outputs.py
indic-asr-benchmark-6k
maya-asr-hybrid-fastconformer-rnnt-stage1
maya-tts-llama32-1b-pretraining
__pycache__
qwen3-asr-1.7b-phase2-sft
test_audio
4.57.6


curated-tokenizers                       0.0.9
curated-transformers                     0.1.1
sentencepiece                            0.2.1
spacy-curated-transformers               0.3.1
tiktoken                                 0.12.0
tokenizers                               0.22.2
transformers                             4.57.6
transformers-stream-generator            0.0.5
x-transformers                           2.16.1


Good, all dependencies are available. Now let me create a comprehensive tokenizer benchmark script.


Write

#!/usr/bin/env python3
"""
Rigorous Tokenizer Benchmark: Gemma-3-270M vs Qwen3-ASR-1.7B
Tests: efficiency, speed, vocabulary coverage, multilingual handling,
       edge cases, roundtrip fidelity, and token distribution analysis.
"""

import time
import json
import sys
import os
import statistics
import unicodedata
from collections import Counter
from transformers import AutoTokenizer

# ─── Configuration ───────────────────────────────────────────────────────────

MODELS = {
    "gemma3-270m": "google/gemma-3-270m",
    "qwen3-asr-1.7b": "Qwen/Qwen3-ASR-1.7B",
}

# ─── Test Data ───────────────────────────────────────────────────────────────

# English texts of varying complexity
ENGLISH_TEXTS = {
    "simple": "The quick brown fox jumps over the lazy dog.",
    "technical": "The transformer architecture uses multi-head self-attention mechanisms with scaled dot-product attention, layer normalization, and residual connections.",
    "code": 'def fibonacci(n: int) -> int:\n    """Return the nth Fibonacci number."""\n    if n <= 1:\n        return n\n    return fibonacci(n - 1) + fibonacci(n - 2)',
    "numbers": "In 2024, the GDP grew by 3.14159% to $21,427,700,000,000. Temperature was -40.5°C.",
    "punctuation_heavy": 'He said, "Wait—no! Don\'t do that!!!" She replied: "Why not?" [silence] ...then (nothing).',
    "urls_emails": "Visit https://www.example.com/path?key=value&foo=bar or email user@domain.co.uk for info.",
    "long_paragraph": (
        "Artificial intelligence has transformed numerous industries, from healthcare diagnostics "
        "to autonomous vehicle navigation. Large language models, in particular, have demonstrated "
        "remarkable capabilities in natural language understanding, generation, and reasoning. "
        "These models are trained on vast corpora of text data, learning statistical patterns "
        "that enable them to produce coherent and contextually appropriate responses. The "
        "architecture underlying most modern LLMs is the Transformer, which relies on self-attention "
        "mechanisms to capture long-range dependencies in sequential data. Fine-tuning these "
        "pre-trained models on domain-specific datasets has become a standard practice for "
        "achieving state-of-the-art performance on specialized tasks."
    ),
}

# Multilingual / Indic texts
MULTILINGUAL_TEXTS = {
    "hindi": "भारत एक विविधताओं से भरा देश है। यहाँ की संस्कृति, भाषा और परंपराएँ अद्वितीय हैं।",
    "hindi_technical": "स्वचालित वाक् पहचान प्रणाली ध्वनि संकेतों को पाठ में परिवर्तित करती है।",
    "tamil": "தமிழ் உலகின் மிகப் பழமையான மொழிகளில் ஒன்றாகும்.",
    "bengali": "বাংলাদেশ দক্ষিণ এশিয়ার একটি দেশ। এর রাজধানী ঢাকা।",
    "telugu": "తెలుగు భాషా దక్షిణ భారతదేశంలో ఒక ప్రధాన భాష.",
    "marathi": "महाराष्ट्र हे भारतातील एक राज्य आहे. मुंबई ही त्याची राजधानी आहे.",
    "gujarati": "ગુજરાત ભારતનું એક રાજ્ય છે. તેની રાજધાની ગાંધીનગર છે.",
    "kannada": "ಕರ್ನಾಟಕ ದಕ್ಷಿಣ ಭಾರತದ ಒಂದು ರಾಜ್ಯ. ಬೆಂಗಳೂರು ಅದರ ರಾಜಧಾನಿ.",
    "malayalam": "കേരളം ഇന്ത്യയിലെ ഒരു സംസ്ഥാനമാണ്. തിരുവനന്തപുരം ആണ് തലസ്ഥാനം.",
    "odia": "ଓଡ଼ିଶା ଭାରତର ଏକ ରାଜ୍ୟ। ଭୁବନେଶ୍ୱର ଏହାର ରାଜଧାନୀ।",
    "punjabi": "ਪੰਜਾਬ ਭਾਰਤ ਦਾ ਇੱਕ ਰਾਜ ਹੈ। ਚੰਡੀਗੜ੍ਹ ਇਸ ਦੀ ਰਾਜਧਾਨੀ ਹੈ।",
    "chinese": "人工智能技术正在改变世界各地的产业格局。",
    "japanese": "日本語の自然言語処理は、形態素解析から始まります。",
    "arabic": "الذكاء الاصطناعي يغير العالم بسرعة كبيرة.",
    "korean": "한국어 자연어 처리는 형태소 분석에서 시작됩니다.",
    "mixed_hindi_english": "AI technology ने भारत में healthcare को transform किया है। Machine learning models अब Hindi speech recognition में भी काम करते हैं।",
    "mixed_code_switching": "मैंने yesterday एक new model train किया, accuracy 95.3% आई with batch_size=32.",
}

# Edge cases
EDGE_CASES = {
    "empty": "",
    "single_char": "a",
    "single_unicode": "अ",
    "whitespace_only": "   \t\n  ",
    "repeated_char": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "emoji": "🎉🚀🤖💡🔥 AI is amazing! 🌍🎯✨",
    "emoji_complex": "👨👩👧👦 Family emoji, 🏳🌈 flag, 👩🏽💻 person",
    "special_tokens": "<s> </s> <pad> <unk> <mask> [CLS] [SEP] [PAD]",
    "html_xml": '

Hello & welcome

',
    "json_str": '{"model": "gemma", "params": 270000000, "layers": [12, 24]}',
    "math": "∫₀^∞ e^(-x²) dx = √π/2, ∑_{n=1}^{∞} 1/n² = π²/6",
    "zalgo": "H̸̡̪̯ͨ͊̽̅̾̎E̮̟͈̣̖̰̤̤ C̷̙̲̝͖OME̶̡̛S",
    "long_word": "Pneumonoultramicroscopicsilicovolcanoconiosis",
    "camelCase": "thisIsACamelCaseVariableNameThatIsVeryLong",
    "snake_case": "this_is_a_snake_case_variable_name_that_is_very_long",
    "numbers_formats": "1,234,567.89 -42 0xFF 0b1010 1e-10 3.14159265358979",
    "newlines_tabs": "line1\nline2\n\nline4\ttabbed\t\there",
    "null_and_control": "text\x00with\x01control\x02chars",
    "rtl_mixed": "Hello مرحبا World عالم",
    "devanagari_conjuncts": "क्ष त्र ज्ञ श्र द्ध क्त स्त्र",
}

# ASR-specific texts (since Qwen3 is ASR-focused)
ASR_TEXTS = {
    "disfluent": "So um I was like uh going to the uh store and um yeah it was like really um crowded.",
    "filler_words": "you know what I mean like basically essentially actually literally",
    "transcript_style": "okay so first we need to uh open the file and then we can um process the data right",
    "noisy_transcript": "I [unintelligible] went to the [noise] store yesterday [laughter] and bought groceries",
    "timestamps": "[0.0-1.5] Hello everyone [1.5-3.0] welcome to the presentation [3.0-5.5] today we will discuss",
    "hindi_transcript": "नमस्ते आज हम बात करेंगे artificial intelligence के बारे में जो कि बहुत important topic है",
}

def load_tokenizers():
    """Load both tokenizers."""
    tokenizers = {}
    for name, model_id in MODELS.items():
        print(f"Loading tokenizer: {name} ({model_id})...")
        try:
            tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
            tokenizers[name] = tok
            print(f"  ✓ Loaded. Vocab size: {tok.vocab_size:,}")
        except Exception as e:
            print(f"  ✗ Failed: {e}")
            sys.exit(1)
    return tokenizers

def print_header(title):
    print(f"\n{'='*80}")
    print(f"  {title}")
    print(f"{'='*80}")

def print_subheader(title):
    print(f"\n--- {title} ---")

# ─── Test 1: Vocabulary Analysis ─────────────────────────────────────────────

def test_vocabulary_analysis(tokenizers):
    print_header("TEST 1: VOCABULARY ANALYSIS")

    for name, tok in tokenizers.items():
        print(f"\n[{name}]")
        vocab = tok.get_vocab()
        vocab_size = len(vocab)
        print(f"  Vocabulary size:        {vocab_size:,}")
        print(f"  Reported vocab_size:    {tok.vocab_size:,}")

        # Analyze token types
        byte_tokens = sum(1 for t in vocab if t.startswith("<0x") or t.startswith("\\x"))
        special_tokens = set()
        if hasattr(tok, 'all_special_tokens'):
            special_tokens = set(tok.all_special_tokens)
        print(f"  Special tokens:         {len(special_tokens)}")
        if special_tokens:
            shown = list(special_tokens)[:10]
            print(f"    Examples: {shown}")
        print(f"  Byte-level tokens:      {byte_tokens}")

        # Token length distribution
        token_lengths = [len(t) for t in vocab if t not in special_tokens]
        if token_lengths:
            print(f"  Token length (chars):   min={min(token_lengths)}, max={max(token_lengths)}, "
                  f"mean={statistics.mean(token_lengths):.2f}, median={statistics.median(token_lengths):.1f}")

        # Script coverage analysis
        scripts = Counter()
        for token in vocab:
            for ch in token:
                try:
                    script = unicodedata.name(ch, "UNKNOWN").split()[0]
                    scripts[script] += 1
                except:
                    scripts["UNKNOWN"] += 1
        top_scripts = scripts.most_common(15)
        print(f"  Top Unicode scripts in vocab:")
        for script, count in top_scripts:
            print(f"    {script:20s}: {count:>8,}")

# ─── Test 2: Tokenization Efficiency ─────────────────────────────────────────

def test_efficiency(tokenizers):
    print_header("TEST 2: TOKENIZATION EFFICIENCY (tokens per text)")

    all_text_groups = {
        "English": ENGLISH_TEXTS,
        "Multilingual": MULTILINGUAL_TEXTS,
        "Edge Cases": EDGE_CASES,
        "ASR-Specific": ASR_TEXTS,
    }

    results = {name: {} for name in tokenizers}

    for group_name, texts in all_text_groups.items():
        print_subheader(group_name)
        print(f"  {'Text':<25s}", end="")
        for name in tokenizers:
            print(f"  {name:>16s}", end="")
        print(f"  {'Ratio':>10s}  {'Chars':>6s}")
        print(f"  {'-'*25}", end="")
        for _ in tokenizers:
            print(f"  {'-'*16}", end="")
        print(f"  {'-'*10}  {'-'*6}")

        for text_name, text in texts.items():
            if not text:
                continue
            char_count = len(text)
            token_counts = {}
            for name, tok in tokenizers.items():
                ids = tok.encode(text, add_special_tokens=False)
                token_counts[name] = len(ids)

            names = list(tokenizers.keys())
            if token_counts[names[1]] > 0:
                ratio = token_counts[names[0]] / token_counts[names[1]]
            else:
                ratio = float('inf')

            print(f"  {text_name:<25s}", end="")
            for name in tokenizers:
                count = token_counts[name]
                # chars per token
                cpt = char_count / count if count > 0 else 0
                print(f"  {count:>8d} ({cpt:.1f})", end="")
            print(f"  {ratio:>9.3f}x  {char_count:>6d}")

            for name in tokenizers:
                results[name][f"{group_name}/{text_name}"] = token_counts[name]

    # Summary statistics
    print_subheader("Efficiency Summary")
    for name in tokenizers:
        counts = [v for v in results[name].values() if v > 0]
        if counts:
            print(f"  [{name}] Total tokens across all tests: {sum(counts):,}, "
                  f"Mean: {statistics.mean(counts):.1f}, Median: {statistics.median(counts):.1f}")

    return results

# ─── Test 3: Speed Benchmark ─────────────────────────────────────────────────

def test_speed(tokenizers):
    print_header("TEST 3: ENCODING/DECODING SPEED")

    test_texts = {
        "short (45 chars)": ENGLISH_TEXTS["simple"],
        "medium (200 chars)": ENGLISH_TEXTS["technical"],
        "long (700 chars)": ENGLISH_TEXTS["long_paragraph"],
        "hindi (100 chars)": MULTILINGUAL_TEXTS["hindi"],
        "code (180 chars)": ENGLISH_TEXTS["code"],
        "mixed hi-en": MULTILINGUAL_TEXTS["mixed_hindi_english"],
    }

    N_WARMUP = 50
    N_ITER = 500

    for text_name, text in test_texts.items():
        print_subheader(f"Text: {text_name}")
        print(f"  {'Model':<20s} {'Encode(ms)':>12s} {'Decode(ms)':>12s} {'Tok/s(enc)':>12s} {'Roundtrip':>10s}")

        for name, tok in tokenizers.items():
            # Warmup
            for _ in range(N_WARMUP):
                ids = tok.encode(text, add_special_tokens=False)
                tok.decode(ids)

            # Encode benchmark
            encode_times = []
            for _ in range(N_ITER):
                start = time.perf_counter()
                ids = tok.encode(text, add_special_tokens=False)
                encode_times.append(time.perf_counter() - start)

            # Decode benchmark
            decode_times = []
            for _ in range(N_ITER):
                start = time.perf_counter()
                tok.decode(ids)
                decode_times.append(time.perf_counter() - start)

            enc_median = statistics.median(encode_times) * 1000
            dec_median = statistics.median(decode_times) * 1000
            tokens_per_sec = len(ids) / statistics.median(encode_times) if statistics.median(encode_times) > 0 else 0

            # Roundtrip check
            decoded = tok.decode(ids, skip_special_tokens=True)
            roundtrip_ok = "✓" if decoded.strip() == text.strip() else "≈"

            print(f"  {name:<20s} {enc_median:>11.4f} {dec_median:>11.4f} {tokens_per_sec:>11,.0f} {roundtrip_ok:>10s}")

# ─── Test 4: Roundtrip Fidelity ──────────────────────────────────────────────

def test_roundtrip_fidelity(tokenizers):
    print_header("TEST 4: ROUNDTRIP FIDELITY (encode → decode)")

    all_texts = {}
    all_texts.update(ENGLISH_TEXTS)
    all_texts.update(MULTILINGUAL_TEXTS)
    all_texts.update(EDGE_CASES)
    all_texts.update(ASR_TEXTS)

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        failures = []
        for text_name, text in all_texts.items():
            if not text:
                continue
            try:
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                # Normalize whitespace for comparison
                orig_norm = " ".join(text.split())
                dec_norm = " ".join(decoded.split())
                if orig_norm != dec_norm:
                    failures.append((text_name, text[:60], decoded[:60]))
            except Exception as e:
                failures.append((text_name, text[:60], f"ERROR: {e}"))

        if failures:
            print(f"  Roundtrip mismatches: {len(failures)}/{len(all_texts)}")
            for tname, orig, dec in failures[:10]:
                print(f"    [{tname}]")
                print(f"      Original: {repr(orig)}")
                print(f"      Decoded:  {repr(dec)}")
        else:
            print(f"  All {len(all_texts)} texts roundtrip perfectly! ✓")

# ─── Test 5: Batch Throughput ─────────────────────────────────────────────────

def test_batch_throughput(tokenizers):
    print_header("TEST 5: BATCH THROUGHPUT")

    # Create a batch of 100 sentences
    batch = []
    for texts in [ENGLISH_TEXTS, MULTILINGUAL_TEXTS, ASR_TEXTS]:
        for text in texts.values():
            if text:
                batch.append(text)
    # Repeat to get ~100 items
    while len(batch) < 100:
        batch = batch + batch
    batch = batch[:100]

    total_chars = sum(len(t) for t in batch)
    print(f"  Batch size: {len(batch)} texts, {total_chars:,} total chars")

    N_ITER = 20

    for name, tok in tokenizers.items():
        # Warmup
        for _ in range(3):
            for text in batch:
                tok.encode(text, add_special_tokens=False)

        times = []
        total_tokens = 0
        for _ in range(N_ITER):
            start = time.perf_counter()
            for text in batch:
                ids = tok.encode(text, add_special_tokens=False)
                total_tokens += len(ids)
            times.append(time.perf_counter() - start)

        avg_tokens = total_tokens // N_ITER
        med_time = statistics.median(times)
        print(f"  [{name}] Median batch time: {med_time*1000:.2f}ms, "
              f"Tokens/batch: {avg_tokens:,}, "
              f"Throughput: {total_chars/med_time:,.0f} chars/s, "
              f"{avg_tokens/med_time:,.0f} tokens/s")

# ─── Test 6: Token Distribution Analysis ─────────────────────────────────────

def test_token_distribution(tokenizers):
    print_header("TEST 6: TOKEN DISTRIBUTION ANALYSIS")

    # Combine all texts
    corpus = ""
    for texts in [ENGLISH_TEXTS, MULTILINGUAL_TEXTS, ASR_TEXTS]:
        for text in texts.values():
            if text:
                corpus += text + " "

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        ids = tok.encode(corpus, add_special_tokens=False)

        # Token frequency
        token_freq = Counter(ids)
        tokens_decoded = [(tok.decode([tid]), count) for tid, count in token_freq.most_common(15)]

        print(f"  Total tokens: {len(ids):,}")
        print(f"  Unique tokens used: {len(token_freq):,} / {tok.vocab_size:,} ({100*len(token_freq)/tok.vocab_size:.2f}%)")
        print(f"  Top 15 most frequent tokens:")
        for token_str, count in tokens_decoded:
            print(f"    {repr(token_str):>25s}: {count:>4d} ({100*count/len(ids):.1f}%)")

        # Token length distribution (in characters)
        token_char_lengths = []
        for tid in ids:
            decoded = tok.decode([tid])
            token_char_lengths.append(len(decoded))

        print(f"  Token character lengths: min={min(token_char_lengths)}, max={max(token_char_lengths)}, "
              f"mean={statistics.mean(token_char_lengths):.2f}")

# ─── Test 7: Indic Language Deep Dive ─────────────────────────────────────────

def test_indic_deep_dive(tokenizers):
    print_header("TEST 7: INDIC LANGUAGE TOKENIZATION DEEP DIVE")

    indic_texts = {k: v for k, v in MULTILINGUAL_TEXTS.items()
                   if k in ["hindi", "hindi_technical", "tamil", "bengali", "telugu",
                            "marathi", "gujarati", "kannada", "malayalam", "odia", "punjabi",
                            "mixed_hindi_english", "mixed_code_switching"]}

    print(f"\n  {'Language':<22s}", end="")
    for name in tokenizers:
        print(f"  {'Tokens':>8s} {'Ch/Tok':>7s}", end="")
    print(f"  {'Winner':>12s}")

    for text_name, text in indic_texts.items():
        char_count = len(text)
        print(f"  {text_name:<22s}", end="")
        token_counts = {}
        for name, tok in tokenizers.items():
            ids = tok.encode(text, add_special_tokens=False)
            token_counts[name] = len(ids)
            cpt = char_count / len(ids) if len(ids) > 0 else 0
            print(f"  {len(ids):>8d} {cpt:>7.2f}", end="")

        names = list(tokenizers.keys())
        winner = names[0] if token_counts[names[0]] <= token_counts[names[1]] else names[1]
        print(f"  {winner:>12s}")

    # Show actual tokenization for Hindi
    print_subheader("Hindi Tokenization Detail")
    hindi = MULTILINGUAL_TEXTS["hindi"]
    for name, tok in tokenizers.items():
        ids = tok.encode(hindi, add_special_tokens=False)
        tokens = [tok.decode([tid]) for tid in ids]
        print(f"\n  [{name}] ({len(ids)} tokens)")
        print(f"  Text: {hindi}")
        print(f"  Tokens: {tokens}")

# ─── Test 8: Compression Ratio ───────────────────────────────────────────────

def test_compression_ratio(tokenizers):
    print_header("TEST 8: COMPRESSION RATIO (bytes per token)")

    all_groups = {
        "English": ENGLISH_TEXTS,
        "Indic": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                  if k in ["hindi", "tamil", "bengali", "telugu", "marathi"]},
        "CJK": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                if k in ["chinese", "japanese", "korean"]},
        "Code": {"code": ENGLISH_TEXTS["code"]},
        "Mixed": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                  if "mixed" in k},
        "ASR": ASR_TEXTS,
    }

    print(f"\n  {'Category':<15s}", end="")
    for name in tokenizers:
        print(f"  {name + ' (B/tok)':>20s}", end="")
    print(f"  {'Better':>15s}")

    for group_name, texts in all_groups.items():
        combined = " ".join(t for t in texts.values() if t)
        byte_count = len(combined.encode('utf-8'))

        print(f"  {group_name:<15s}", end="")
        bpt = {}
        for name, tok in tokenizers.items():
            ids = tok.encode(combined, add_special_tokens=False)
            bpt[name] = byte_count / len(ids) if len(ids) > 0 else 0
            print(f"  {bpt[name]:>20.3f}", end="")

        names = list(tokenizers.keys())
        # Higher bytes per token = more efficient (fewer tokens needed)
        winner = names[0] if bpt[names[0]] >= bpt[names[1]] else names[1]
        print(f"  {winner:>15s}")

# ─── Test 9: Special / Unusual Input Handling ─────────────────────────────────

def test_special_inputs(tokenizers):
    print_header("TEST 9: SPECIAL INPUT HANDLING")

    tests = {
        "empty string": "",
        "single space": " ",
        "BOM": "\ufeff",
        "zero-width space": "\u200b",
        "zero-width joiner": "\u200d",
        "right-to-left mark": "\u200f",
        "null byte": "\x00",
        "very long repeat": "a" * 10000,
        "alternating scripts": "हa ि b क c",
        "combining diacritics": "a\u0301 e\u0300 o\u0302",  # á è ô
        "surrogate-safe emoji": "😀" * 50,
    }

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        for test_name, text in tests.items():
            try:
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                status = "✓"
                info = f"{len(ids)} tokens"
                if text and decoded.strip() != text.strip() and test_name not in ["null byte", "BOM", "zero-width space", "zero-width joiner", "right-to-left mark"]:
                    status = "≈"
                    info += " (lossy)"
            except Exception as e:
                status = "✗"
                info = str(e)[:60]

            print(f"  {status} {test_name:<25s}: {info}")

# ─── Test 10: Subword Segmentation Quality ───────────────────────────────────

def test_subword_quality(tokenizers):
    print_header("TEST 10: SUBWORD SEGMENTATION QUALITY")

    # Words where we can judge segmentation quality
    words = [
        "unhappiness", "unbelievable", "internationalization",
        "preprocessing", "tokenization", "transformer",
        "भारतीय", "स्वतंत्रता", "अंतरराष्ट्रीय",
        "counterproductive", "misunderstanding",
        "autobiography", "electromagnetic",
    ]

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        for word in words:
            ids = tok.encode(word, add_special_tokens=False)
            subwords = [tok.decode([tid]) for tid in ids]
            print(f"  {word:<30s} → {subwords}")

# ─── Main ────────────────────────────────────────────────────────────────────

def main():
    print("=" * 80)
    print("  TOKENIZER BENCHMARK: Gemma-3-270M vs Qwen3-ASR-1.7B")
    print("=" * 80)

    tokenizers = load_tokenizers()

    test_vocabulary_analysis(tokenizers)
    test_efficiency(tokenizers)
    test_speed(tokenizers)
    test_roundtrip_fidelity(tokenizers)
    test_batch_throughput(tokenizers)
    test_token_distribution(tokenizers)
    test_indic_deep_dive(tokenizers)
    test_compression_ratio(tokenizers)
    test_special_inputs(tokenizers)
    test_subword_quality(tokenizers)

    print_header("BENCHMARK COMPLETE")
    print("\nAll 10 test suites finished.")

if __name__ == "__main__":
    main()


File created successfully at: /home/ubuntu/training/benchmark_tokenizers.py

#!/usr/bin/env python3
"""
Rigorous Tokenizer Benchmark: Gemma-3-270M vs Qwen3-ASR-1.7B
Tests: efficiency, speed, vocabulary coverage, multilingual handling,
       edge cases, roundtrip fidelity, and token distribution analysis.
"""

import time
import json
import sys
import os
import statistics
import unicodedata
from collections import Counter
from transformers import AutoTokenizer

# ─── Configuration ───────────────────────────────────────────────────────────

MODELS = {
    "gemma3-270m": "google/gemma-3-270m",
    "qwen3-asr-1.7b": "Qwen/Qwen3-ASR-1.7B",
}

# ─── Test Data ───────────────────────────────────────────────────────────────

# English texts of varying complexity
ENGLISH_TEXTS = {
    "simple": "The quick brown fox jumps over the lazy dog.",
    "technical": "The transformer architecture uses multi-head self-attention mechanisms with scaled dot-product attention, layer normalization, and residual connections.",
    "code": 'def fibonacci(n: int) -> int:\n    """Return the nth Fibonacci number."""\n    if n <= 1:\n        return n\n    return fibonacci(n - 1) + fibonacci(n - 2)',
    "numbers": "In 2024, the GDP grew by 3.14159% to $21,427,700,000,000. Temperature was -40.5°C.",
    "punctuation_heavy": 'He said, "Wait—no! Don\'t do that!!!" She replied: "Why not?" [silence] ...then (nothing).',
    "urls_emails": "Visit https://www.example.com/path?key=value&foo=bar or email user@domain.co.uk for info.",
    "long_paragraph": (
        "Artificial intelligence has transformed numerous industries, from healthcare diagnostics "
        "to autonomous vehicle navigation. Large language models, in particular, have demonstrated "
        "remarkable capabilities in natural language understanding, generation, and reasoning. "
        "These models are trained on vast corpora of text data, learning statistical patterns "
        "that enable them to produce coherent and contextually appropriate responses. The "
        "architecture underlying most modern LLMs is the Transformer, which relies on self-attention "
        "mechanisms to capture long-range dependencies in sequential data. Fine-tuning these "
        "pre-trained models on domain-specific datasets has become a standard practice for "
        "achieving state-of-the-art performance on specialized tasks."
    ),
}

# Multilingual / Indic texts
MULTILINGUAL_TEXTS = {
    "hindi": "भारत एक विविधताओं से भरा देश है। यहाँ की संस्कृति, भाषा और परंपराएँ अद्वितीय हैं।",
    "hindi_technical": "स्वचालित वाक् पहचान प्रणाली ध्वनि संकेतों को पाठ में परिवर्तित करती है।",
    "tamil": "தமிழ் உலகின் மிகப் பழமையான மொழிகளில் ஒன்றாகும்.",
    "bengali": "বাংলাদেশ দক্ষিণ এশিয়ার একটি দেশ। এর রাজধানী ঢাকা।",
    "telugu": "తెలుగు భాషా దక్షిణ భారతదేశంలో ఒక ప్రధాన భాష.",
    "marathi": "महाराष्ट्र हे भारतातील एक राज्य आहे. मुंबई ही त्याची राजधानी आहे.",
    "gujarati": "ગુજરાત ભારતનું એક રાજ્ય છે. તેની રાજધાની ગાંધીનગર છે.",
    "kannada": "ಕರ್ನಾಟಕ ದಕ್ಷಿಣ ಭಾರತದ ಒಂದು ರಾಜ್ಯ. ಬೆಂಗಳೂರು ಅದರ ರಾಜಧಾನಿ.",
    "malayalam": "കേരളം ഇന്ത്യയിലെ ഒരു സംസ്ഥാനമാണ്. തിരുവനന്തപുരം ആണ് തലസ്ഥാനം.",
    "odia": "ଓଡ଼ିଶା ଭାରତର ଏକ ରାଜ୍ୟ। ଭୁବନେଶ୍ୱର ଏହାର ରାଜଧାନୀ।",
    "punjabi": "ਪੰਜਾਬ ਭਾਰਤ ਦਾ ਇੱਕ ਰਾਜ ਹੈ। ਚੰਡੀਗੜ੍ਹ ਇਸ ਦੀ ਰਾਜਧਾਨੀ ਹੈ।",
    "chinese": "人工智能技术正在改变世界各地的产业格局。",
    "japanese": "日本語の自然言語処理は、形態素解析から始まります。",
    "arabic": "الذكاء الاصطناعي يغير العالم بسرعة كبيرة.",
    "korean": "한국어 자연어 처리는 형태소 분석에서 시작됩니다.",
    "mixed_hindi_english": "AI technology ने भारत में healthcare को transform किया है। Machine learning models अब Hindi speech recognition में भी काम करते हैं।",
    "mixed_code_switching": "मैंने yesterday एक new model train किया, accuracy 95.3% आई with batch_size=32.",
}

# Edge cases
EDGE_CASES = {
    "empty": "",
    "single_char": "a",
    "single_unicode": "अ",
    "whitespace_only": "   \t\n  ",
    "repeated_char": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
    "emoji": "🎉🚀🤖💡🔥 AI is amazing! 🌍🎯✨",
    "emoji_complex": "👨👩👧👦 Family emoji, 🏳🌈 flag, 👩🏽💻 person",
    "special_tokens": "<s> </s> <pad> <unk> <mask> [CLS] [SEP] [PAD]",
    "html_xml": '

Hello & welcome

',
    "json_str": '{"model": "gemma", "params": 270000000, "layers": [12, 24]}',
    "math": "∫₀^∞ e^(-x²) dx = √π/2, ∑_{n=1}^{∞} 1/n² = π²/6",
    "zalgo": "H̸̡̪̯ͨ͊̽̅̾̎E̮̟͈̣̖̰̤̤ C̷̙̲̝͖OME̶̡̛S",
    "long_word": "Pneumonoultramicroscopicsilicovolcanoconiosis",
    "camelCase": "thisIsACamelCaseVariableNameThatIsVeryLong",
    "snake_case": "this_is_a_snake_case_variable_name_that_is_very_long",
    "numbers_formats": "1,234,567.89 -42 0xFF 0b1010 1e-10 3.14159265358979",
    "newlines_tabs": "line1\nline2\n\nline4\ttabbed\t\there",
    "null_and_control": "text\x00with\x01control\x02chars",
    "rtl_mixed": "Hello مرحبا World عالم",
    "devanagari_conjuncts": "क्ष त्र ज्ञ श्र द्ध क्त स्त्र",
}

# ASR-specific texts (since Qwen3 is ASR-focused)
ASR_TEXTS = {
    "disfluent": "So um I was like uh going to the uh store and um yeah it was like really um crowded.",
    "filler_words": "you know what I mean like basically essentially actually literally",
    "transcript_style": "okay so first we need to uh open the file and then we can um process the data right",
    "noisy_transcript": "I [unintelligible] went to the [noise] store yesterday [laughter] and bought groceries",
    "timestamps": "[0.0-1.5] Hello everyone [1.5-3.0] welcome to the presentation [3.0-5.5] today we will discuss",
    "hindi_transcript": "नमस्ते आज हम बात करेंगे artificial intelligence के बारे में जो कि बहुत important topic है",
}

def load_tokenizers():
    """Load both tokenizers."""
    tokenizers = {}
    for name, model_id in MODELS.items():
        print(f"Loading tokenizer: {name} ({model_id})...")
        try:
            tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
            tokenizers[name] = tok
            print(f"  ✓ Loaded. Vocab size: {tok.vocab_size:,}")
        except Exception as e:
            print(f"  ✗ Failed: {e}")
            sys.exit(1)
    return tokenizers

def print_header(title):
    print(f"\n{'='*80}")
    print(f"  {title}")
    print(f"{'='*80}")

def print_subheader(title):
    print(f"\n--- {title} ---")

# ─── Test 1: Vocabulary Analysis ─────────────────────────────────────────────

def test_vocabulary_analysis(tokenizers):
    print_header("TEST 1: VOCABULARY ANALYSIS")

    for name, tok in tokenizers.items():
        print(f"\n[{name}]")
        vocab = tok.get_vocab()
        vocab_size = len(vocab)
        print(f"  Vocabulary size:        {vocab_size:,}")
        print(f"  Reported vocab_size:    {tok.vocab_size:,}")

        # Analyze token types
        byte_tokens = sum(1 for t in vocab if t.startswith("<0x") or t.startswith("\\x"))
        special_tokens = set()
        if hasattr(tok, 'all_special_tokens'):
            special_tokens = set(tok.all_special_tokens)
        print(f"  Special tokens:         {len(special_tokens)}")
        if special_tokens:
            shown = list(special_tokens)[:10]
            print(f"    Examples: {shown}")
        print(f"  Byte-level tokens:      {byte_tokens}")

        # Token length distribution
        token_lengths = [len(t) for t in vocab if t not in special_tokens]
        if token_lengths:
            print(f"  Token length (chars):   min={min(token_lengths)}, max={max(token_lengths)}, "
                  f"mean={statistics.mean(token_lengths):.2f}, median={statistics.median(token_lengths):.1f}")

        # Script coverage analysis
        scripts = Counter()
        for token in vocab:
            for ch in token:
                try:
                    script = unicodedata.name(ch, "UNKNOWN").split()[0]
                    scripts[script] += 1
                except:
                    scripts["UNKNOWN"] += 1
        top_scripts = scripts.most_common(15)
        print(f"  Top Unicode scripts in vocab:")
        for script, count in top_scripts:
            print(f"    {script:20s}: {count:>8,}")

# ─── Test 2: Tokenization Efficiency ─────────────────────────────────────────

def test_efficiency(tokenizers):
    print_header("TEST 2: TOKENIZATION EFFICIENCY (tokens per text)")

    all_text_groups = {
        "English": ENGLISH_TEXTS,
        "Multilingual": MULTILINGUAL_TEXTS,
        "Edge Cases": EDGE_CASES,
        "ASR-Specific": ASR_TEXTS,
    }

    results = {name: {} for name in tokenizers}

    for group_name, texts in all_text_groups.items():
        print_subheader(group_name)
        print(f"  {'Text':<25s}", end="")
        for name in tokenizers:
            print(f"  {name:>16s}", end="")
        print(f"  {'Ratio':>10s}  {'Chars':>6s}")
        print(f"  {'-'*25}", end="")
        for _ in tokenizers:
            print(f"  {'-'*16}", end="")
        print(f"  {'-'*10}  {'-'*6}")

        for text_name, text in texts.items():
            if not text:
                continue
            char_count = len(text)
            token_counts = {}
            for name, tok in tokenizers.items():
                ids = tok.encode(text, add_special_tokens=False)
                token_counts[name] = len(ids)

            names = list(tokenizers.keys())
            if token_counts[names[1]] > 0:
                ratio = token_counts[names[0]] / token_counts[names[1]]
            else:
                ratio = float('inf')

            print(f"  {text_name:<25s}", end="")
            for name in tokenizers:
                count = token_counts[name]
                # chars per token
                cpt = char_count / count if count > 0 else 0
                print(f"  {count:>8d} ({cpt:.1f})", end="")
            print(f"  {ratio:>9.3f}x  {char_count:>6d}")

            for name in tokenizers:
                results[name][f"{group_name}/{text_name}"] = token_counts[name]

    # Summary statistics
    print_subheader("Efficiency Summary")
    for name in tokenizers:
        counts = [v for v in results[name].values() if v > 0]
        if counts:
            print(f"  [{name}] Total tokens across all tests: {sum(counts):,}, "
                  f"Mean: {statistics.mean(counts):.1f}, Median: {statistics.median(counts):.1f}")

    return results

# ─── Test 3: Speed Benchmark ─────────────────────────────────────────────────

def test_speed(tokenizers):
    print_header("TEST 3: ENCODING/DECODING SPEED")

    test_texts = {
        "short (45 chars)": ENGLISH_TEXTS["simple"],
        "medium (200 chars)": ENGLISH_TEXTS["technical"],
        "long (700 chars)": ENGLISH_TEXTS["long_paragraph"],
        "hindi (100 chars)": MULTILINGUAL_TEXTS["hindi"],
        "code (180 chars)": ENGLISH_TEXTS["code"],
        "mixed hi-en": MULTILINGUAL_TEXTS["mixed_hindi_english"],
    }

    N_WARMUP = 50
    N_ITER = 500

    for text_name, text in test_texts.items():
        print_subheader(f"Text: {text_name}")
        print(f"  {'Model':<20s} {'Encode(ms)':>12s} {'Decode(ms)':>12s} {'Tok/s(enc)':>12s} {'Roundtrip':>10s}")

        for name, tok in tokenizers.items():
            # Warmup
            for _ in range(N_WARMUP):
                ids = tok.encode(text, add_special_tokens=False)
                tok.decode(ids)

            # Encode benchmark
            encode_times = []
            for _ in range(N_ITER):
                start = time.perf_counter()
                ids = tok.encode(text, add_special_tokens=False)
                encode_times.append(time.perf_counter() - start)

            # Decode benchmark
            decode_times = []
            for _ in range(N_ITER):
                start = time.perf_counter()
                tok.decode(ids)
                decode_times.append(time.perf_counter() - start)

            enc_median = statistics.median(encode_times) * 1000
            dec_median = statistics.median(decode_times) * 1000
            tokens_per_sec = len(ids) / statistics.median(encode_times) if statistics.median(encode_times) > 0 else 0

            # Roundtrip check
            decoded = tok.decode(ids, skip_special_tokens=True)
            roundtrip_ok = "✓" if decoded.strip() == text.strip() else "≈"

            print(f"  {name:<20s} {enc_median:>11.4f} {dec_median:>11.4f} {tokens_per_sec:>11,.0f} {roundtrip_ok:>10s}")

# ─── Test 4: Roundtrip Fidelity ──────────────────────────────────────────────

def test_roundtrip_fidelity(tokenizers):
    print_header("TEST 4: ROUNDTRIP FIDELITY (encode → decode)")

    all_texts = {}
    all_texts.update(ENGLISH_TEXTS)
    all_texts.update(MULTILINGUAL_TEXTS)
    all_texts.update(EDGE_CASES)
    all_texts.update(ASR_TEXTS)

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        failures = []
        for text_name, text in all_texts.items():
            if not text:
                continue
            try:
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                # Normalize whitespace for comparison
                orig_norm = " ".join(text.split())
                dec_norm = " ".join(decoded.split())
                if orig_norm != dec_norm:
                    failures.append((text_name, text[:60], decoded[:60]))
            except Exception as e:
                failures.append((text_name, text[:60], f"ERROR: {e}"))

        if failures:
            print(f"  Roundtrip mismatches: {len(failures)}/{len(all_texts)}")
            for tname, orig, dec in failures[:10]:
                print(f"    [{tname}]")
                print(f"      Original: {repr(orig)}")
                print(f"      Decoded:  {repr(dec)}")
        else:
            print(f"  All {len(all_texts)} texts roundtrip perfectly! ✓")

# ─── Test 5: Batch Throughput ─────────────────────────────────────────────────

def test_batch_throughput(tokenizers):
    print_header("TEST 5: BATCH THROUGHPUT")

    # Create a batch of 100 sentences
    batch = []
    for texts in [ENGLISH_TEXTS, MULTILINGUAL_TEXTS, ASR_TEXTS]:
        for text in texts.values():
            if text:
                batch.append(text)
    # Repeat to get ~100 items
    while len(batch) < 100:
        batch = batch + batch
    batch = batch[:100]

    total_chars = sum(len(t) for t in batch)
    print(f"  Batch size: {len(batch)} texts, {total_chars:,} total chars")

    N_ITER = 20

    for name, tok in tokenizers.items():
        # Warmup
        for _ in range(3):
            for text in batch:
                tok.encode(text, add_special_tokens=False)

        times = []
        total_tokens = 0
        for _ in range(N_ITER):
            start = time.perf_counter()
            for text in batch:
                ids = tok.encode(text, add_special_tokens=False)
                total_tokens += len(ids)
            times.append(time.perf_counter() - start)

        avg_tokens = total_tokens // N_ITER
        med_time = statistics.median(times)
        print(f"  [{name}] Median batch time: {med_time*1000:.2f}ms, "
              f"Tokens/batch: {avg_tokens:,}, "
              f"Throughput: {total_chars/med_time:,.0f} chars/s, "
              f"{avg_tokens/med_time:,.0f} tokens/s")

# ─── Test 6: Token Distribution Analysis ─────────────────────────────────────

def test_token_distribution(tokenizers):
    print_header("TEST 6: TOKEN DISTRIBUTION ANALYSIS")

    # Combine all texts
    corpus = ""
    for texts in [ENGLISH_TEXTS, MULTILINGUAL_TEXTS, ASR_TEXTS]:
        for text in texts.values():
            if text:
                corpus += text + " "

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        ids = tok.encode(corpus, add_special_tokens=False)

        # Token frequency
        token_freq = Counter(ids)
        tokens_decoded = [(tok.decode([tid]), count) for tid, count in token_freq.most_common(15)]

        print(f"  Total tokens: {len(ids):,}")
        print(f"  Unique tokens used: {len(token_freq):,} / {tok.vocab_size:,} ({100*len(token_freq)/tok.vocab_size:.2f}%)")
        print(f"  Top 15 most frequent tokens:")
        for token_str, count in tokens_decoded:
            print(f"    {repr(token_str):>25s}: {count:>4d} ({100*count/len(ids):.1f}%)")

        # Token length distribution (in characters)
        token_char_lengths = []
        for tid in ids:
            decoded = tok.decode([tid])
            token_char_lengths.append(len(decoded))

        print(f"  Token character lengths: min={min(token_char_lengths)}, max={max(token_char_lengths)}, "
              f"mean={statistics.mean(token_char_lengths):.2f}")

# ─── Test 7: Indic Language Deep Dive ─────────────────────────────────────────

def test_indic_deep_dive(tokenizers):
    print_header("TEST 7: INDIC LANGUAGE TOKENIZATION DEEP DIVE")

    indic_texts = {k: v for k, v in MULTILINGUAL_TEXTS.items()
                   if k in ["hindi", "hindi_technical", "tamil", "bengali", "telugu",
                            "marathi", "gujarati", "kannada", "malayalam", "odia", "punjabi",
                            "mixed_hindi_english", "mixed_code_switching"]}

    print(f"\n  {'Language':<22s}", end="")
    for name in tokenizers:
        print(f"  {'Tokens':>8s} {'Ch/Tok':>7s}", end="")
    print(f"  {'Winner':>12s}")

    for text_name, text in indic_texts.items():
        char_count = len(text)
        print(f"  {text_name:<22s}", end="")
        token_counts = {}
        for name, tok in tokenizers.items():
            ids = tok.encode(text, add_special_tokens=False)
            token_counts[name] = len(ids)
            cpt = char_count / len(ids) if len(ids) > 0 else 0
            print(f"  {len(ids):>8d} {cpt:>7.2f}", end="")

        names = list(tokenizers.keys())
        winner = names[0] if token_counts[names[0]] <= token_counts[names[1]] else names[1]
        print(f"  {winner:>12s}")

    # Show actual tokenization for Hindi
    print_subheader("Hindi Tokenization Detail")
    hindi = MULTILINGUAL_TEXTS["hindi"]
    for name, tok in tokenizers.items():
        ids = tok.encode(hindi, add_special_tokens=False)
        tokens = [tok.decode([tid]) for tid in ids]
        print(f"\n  [{name}] ({len(ids)} tokens)")
        print(f"  Text: {hindi}")
        print(f"  Tokens: {tokens}")

# ─── Test 8: Compression Ratio ───────────────────────────────────────────────

def test_compression_ratio(tokenizers):
    print_header("TEST 8: COMPRESSION RATIO (bytes per token)")

    all_groups = {
        "English": ENGLISH_TEXTS,
        "Indic": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                  if k in ["hindi", "tamil", "bengali", "telugu", "marathi"]},
        "CJK": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                if k in ["chinese", "japanese", "korean"]},
        "Code": {"code": ENGLISH_TEXTS["code"]},
        "Mixed": {k: v for k, v in MULTILINGUAL_TEXTS.items()
                  if "mixed" in k},
        "ASR": ASR_TEXTS,
    }

    print(f"\n  {'Category':<15s}", end="")
    for name in tokenizers:
        print(f"  {name + ' (B/tok)':>20s}", end="")
    print(f"  {'Better':>15s}")

    for group_name, texts in all_groups.items():
        combined = " ".join(t for t in texts.values() if t)
        byte_count = len(combined.encode('utf-8'))

        print(f"  {group_name:<15s}", end="")
        bpt = {}
        for name, tok in tokenizers.items():
            ids = tok.encode(combined, add_special_tokens=False)
            bpt[name] = byte_count / len(ids) if len(ids) > 0 else 0
            print(f"  {bpt[name]:>20.3f}", end="")

        names = list(tokenizers.keys())
        # Higher bytes per token = more efficient (fewer tokens needed)
        winner = names[0] if bpt[names[0]] >= bpt[names[1]] else names[1]
        print(f"  {winner:>15s}")

# ─── Test 9: Special / Unusual Input Handling ─────────────────────────────────

def test_special_inputs(tokenizers):
    print_header("TEST 9: SPECIAL INPUT HANDLING")

    tests = {
        "empty string": "",
        "single space": " ",
        "BOM": "\ufeff",
        "zero-width space": "\u200b",
        "zero-width joiner": "\u200d",
        "right-to-left mark": "\u200f",
        "null byte": "\x00",
        "very long repeat": "a" * 10000,
        "alternating scripts": "हa ि b क c",
        "combining diacritics": "a\u0301 e\u0300 o\u0302",  # á è ô
        "surrogate-safe emoji": "😀" * 50,
    }

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        for test_name, text in tests.items():
            try:
                ids = tok.encode(text, add_special_tokens=False)
                decoded = tok.decode(ids, skip_special_tokens=True)
                status = "✓"
                info = f"{len(ids)} tokens"
                if text and decoded.strip() != text.strip() and test_name not in ["null byte", "BOM", "zero-width space", "zero-width joiner", "right-to-left mark"]:
                    status = "≈"
                    info += " (lossy)"
            except Exception as e:
                status = "✗"
                info = str(e)[:60]

            print(f"  {status} {test_name:<25s}: {info}")

# ─── Test 10: Subword Segmentation Quality ───────────────────────────────────

def test_subword_quality(tokenizers):
    print_header("TEST 10: SUBWORD SEGMENTATION QUALITY")

    # Words where we can judge segmentation quality
    words = [
        "unhappiness", "unbelievable", "internationalization",
        "preprocessing", "tokenization", "transformer",
        "भारतीय", "स्वतंत्रता", "अंतरराष्ट्रीय",
        "counterproductive", "misunderstanding",
        "autobiography", "electromagnetic",
    ]

    for name, tok in tokenizers.items():
        print_subheader(f"Model: {name}")
        for word in words:
            ids = tok.encode(word, add_special_tokens=False)
            subwords = [tok.decode([tid]) for tid in ids]
            print(f"  {word:<30s} → {subwords}")

# ─── Main ────────────────────────────────────────────────────────────────────

def main():
    print("=" * 80)
    print("  TOKENIZER BENCHMARK: Gemma-3-270M vs Qwen3-ASR-1.7B")
    print("=" * 80)

    tokenizers = load_tokenizers()

    test_vocabulary_analysis(tokenizers)
    test_efficiency(tokenizers)
    test_speed(tokenizers)
    test_roundtrip_fidelity(tokenizers)
    test_batch_throughput(tokenizers)
    test_token_distribution(tokenizers)
    test_indic_deep_dive(tokenizers)
    test_compression_ratio(tokenizers)
    test_special_inputs(tokenizers)
    test_subword_quality(tokenizers)

    print_header("BENCHMARK COMPLETE")
    print("\nAll 10 test suites finished.")

if __name__ == "__main__":
    main()


Bash

Run tokenizer benchmark


================================================================================
  TOKENIZER BENCHMARK: Gemma-3-270M vs Qwen3-ASR-1.7B
================================================================================
Loading tokenizer: gemma3-270m (google/gemma-3-270m)...
  ✓ Loaded. Vocab size: 262,144
Loading tokenizer: qwen3-asr-1.7b (Qwen/Qwen3-ASR-1.7B)...
  ✓ Loaded. Vocab size: 151,643

================================================================================
  TEST 1: VOCABULARY ANALYSIS
================================================================================

[gemma3-270m]
  Vocabulary size:        262,145
  Reported vocab_size:    262,144
  Special tokens:         7
    Examples: ['<pad>', '<image_soft_token>', '<unk>', '<start_of_image>', '<end_of_image>', '<eos>', '<bos>']
  Byte-level tokens:      256
  Token length (chars):   min=1, max=31, mean=5.63, median=5.0
  Top Unicode scripts in vocab:
    LATIN               :  960,974
    LOWER               :  138,023
    CYRILLIC            :   64,746
    DEVANAGARI          :   61,878
    BENGALI             :   46,018
    CJK                 :   31,227
    ARABIC              :   29,853
    DIGIT               :   24,461
    TAMIL               :   18,552
    HANGUL              :    7,539
    THAI                :    7,428
    MALAYALAM           :    7,313
    GREATER-THAN        :    6,974
    LESS-THAN           :    6,849
    HIRAGANA            :    6,439

[qwen3-asr-1.7b]
  Vocabulary size:        151,705
  Reported vocab_size:    151,643
  Special tokens:         20
    Examples: ['<|box_end|>', '<|quad_start|>', '<|audio_end|>', '<tts_text_bos>', '<|vision_end|>', '<tts_pad>', '<|audio_pad|>', '<|endoftext|>', '<|vision_pad|>', '<|image_pad|>']
  Byte-level tokens:      9
  Token length (chars):   min=1, max=128, mean=6.44, median=6.0
  Top Unicode scripts in vocab:
    LATIN               :  818,686
    CEDILLA             :   14,294
    MULTIPLICATION      :   11,197
    VULGAR              :   10,259
    SUPERSCRIPT         :   10,153
    FULL                :    7,105
    HYPHEN-MINUS        :    6,393
    LOW                 :    6,069
    INVERTED            :    5,598
    SECTION             :    4,844
    MASCULINE           :    4,750
    LEFT                :    4,122
    DEGREE              :    4,094
    RIGHT-POINTING      :    4,003
    PLUS-MINUS          :    3,608

================================================================================
  TEST 2: TOKENIZATION EFFICIENCY (tokens per text)
================================================================================

--- English ---
  Text                            gemma3-270m    qwen3-asr-1.7b       Ratio   Chars
  -------------------------  ----------------  ----------------  ----------  ------
  simple                           10 (4.4)        10 (4.4)      1.000x      44
  technical                        25 (6.1)        23 (6.6)      1.087x     152
  code                             52 (2.9)        43 (3.5)      1.209x     151
  numbers                          51 (1.6)        50 (1.6)      1.020x      82
  punctuation_heavy                30 (3.0)        30 (3.0)      1.000x      89
  urls_emails                      30 (3.0)        23 (3.9)      1.304x      89
  long_paragraph                  122 (6.1)       119 (6.3)      1.025x     749

--- Multilingual ---
  Text                            gemma3-270m    qwen3-asr-1.7b       Ratio   Chars
  -------------------------  ----------------  ----------------  ----------  ------
  hindi                            20 (4.0)        79 (1.0)      0.253x      81
  hindi_technical                  15 (4.7)        62 (1.1)      0.242x      71
  tamil                             9 (5.2)        51 (0.9)      0.176x      47
  bengali                          12 (4.2)        52 (1.0)      0.231x      50
  telugu                           11 (4.0)        64 (0.7)      0.172x      44
  marathi                          13 (5.0)        61 (1.1)      0.213x      65
  gujarati                         17 (3.1)        77 (0.7)      0.221x      53
  kannada                          17 (3.2)        76 (0.7)      0.224x      54
  malayalam                        16 (3.8)        83 (0.7)      0.193x      61
  odia                             36 (1.3)        89 (0.5)      0.404x      46
  punjabi                          26 (2.0)        75 (0.7)      0.347x      52
  chinese                          10 (2.0)         9 (2.2)      1.111x      20
  japanese                         14 (1.8)        20 (1.2)      0.700x      25
  arabic                           13 (3.2)        16 (2.6)      0.812x      41
  korean                           13 (2.1)        15 (1.8)      0.867x      27
  mixed_hindi_english              24 (5.5)        52 (2.5)      0.462x     131
  mixed_code_switching             24 (3.2)        35 (2.2)      0.686x      78

--- Edge Cases ---
  Text                            gemma3-270m    qwen3-asr-1.7b       Ratio   Chars
  -------------------------  ----------------  ----------------  ----------  ------
  single_char                       1 (1.0)         1 (1.0)      1.000x       1
  single_unicode                    1 (1.0)         1 (1.0)      1.000x       1
  whitespace_only                   4 (1.8)         3 (2.3)      1.333x       7
  repeated_char                     8 (8.0)         8 (8.0)      1.000x      64
  emoji                            13 (1.8)        14 (1.7)      0.929x      24
  emoji_complex                    23 (1.9)        29 (1.5)      0.793x      44
  special_tokens                   18 (2.5)        23 (2.0)      0.783x      45
  html_xml                         17 (3.2)        16 (3.4)      1.062x      55
  json_str                         32 (1.8)        32 (1.8)      1.000x      59
  math                             32 (1.5)        37 (1.3)      0.865x      47
  zalgo                            32 (1.1)        57 (0.6)      0.561x      34
  long_word                        14 (3.2)        17 (2.6)      0.824x      45
  camelCase                        10 (4.2)        11 (3.8)      0.909x      42
  snake_case                       21 (2.5)        13 (4.0)      1.615x      52
  numbers_formats                  48 (1.1)        48 (1.1)      1.000x      51
  newlines_tabs                    13 (2.4)        13 (2.4)      1.000x      31
  null_and_control                  7 (3.3)         7 (3.3)      1.000x      23
  rtl_mixed                         4 (5.5)         7 (3.1)      0.571x      22
  devanagari_conjuncts              9 (3.2)        24 (1.2)      0.375x      29

--- ASR-Specific ---
  Text                            gemma3-270m    qwen3-asr-1.7b       Ratio   Chars
  -------------------------  ----------------  ----------------  ----------  ------
  disfluent                        21 (4.0)        21 (4.0)      1.000x      84
  filler_words                     10 (6.6)        10 (6.6)      1.000x      66
  transcript_style                 19 (4.4)        19 (4.4)      1.000x      83
  noisy_transcript                 21 (4.1)        21 (4.1)      1.000x      86
  timestamps                       37 (2.5)        37 (2.5)      1.000x      94
  hindi_transcript                 16 (5.6)        48 (1.9)      0.333x      89

--- Efficiency Summary ---
  [gemma3-270m] Total tokens across all tests: 1,041, Mean: 21.2, Median: 17.0
  [qwen3-asr-1.7b] Total tokens across all tests: 1,731, Mean: 35.3, Median: 24.0

================================================================================
  TEST 3: ENCODING/DECODING SPEED
================================================================================

--- Text: short (45 chars) ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.0347      0.0107     287,959          ✓
  qwen3-asr-1.7b            0.0461      0.0076     217,119          ✓

--- Text: medium (200 chars) ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.0463      0.0168     540,046          ✓
  qwen3-asr-1.7b            0.0776      0.0106     296,429          ✓

--- Text: long (700 chars) ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.2609      0.0649     467,549          ✓
  qwen3-asr-1.7b            0.2744      0.0331     433,654          ✓

--- Text: hindi (100 chars) ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.0467      0.0157     428,520          ✓
  qwen3-asr-1.7b            0.1071      0.0203     737,819          ✓

--- Text: code (180 chars) ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.0698      0.0291     744,699          ✓
  qwen3-asr-1.7b            0.0990      0.0135     434,304          ✓

--- Text: mixed hi-en ---
  Model                  Encode(ms)   Decode(ms)   Tok/s(enc)  Roundtrip
  gemma3-270m               0.0506      0.0182     473,971          ✓
  qwen3-asr-1.7b            0.0970      0.0156     536,093          ✓

================================================================================
  TEST 4: ROUNDTRIP FIDELITY (encode → decode)
================================================================================

--- Model: gemma3-270m ---
  Roundtrip mismatches: 1/50
    [special_tokens]
      Original: '<s> </s> <pad> <unk> <mask> [CLS] [SEP] [PAD]'
      Decoded:  '<s> </s>   <mask> [CLS] [SEP] [PAD]'

--- Model: qwen3-asr-1.7b ---
  All 50 texts roundtrip perfectly! ✓

================================================================================
  TEST 5: BATCH THROUGHPUT
================================================================================
  Batch size: 100 texts, 9,967 total chars
  [gemma3-270m] Median batch time: 5.52ms, Tokens/batch: 2,566, Throughput: 1,805,756 chars/s, 464,891 tokens/s
  [qwen3-asr-1.7b] Median batch time: 8.95ms, Tokens/batch: 4,600, Throughput: 1,113,245 chars/s, 513,788 tokens/s

================================================================================
  TEST 6: TOKEN DISTRIBUTION ANALYSIS
================================================================================

--- Model: gemma3-270m ---
  Total tokens: 732
  Unique tokens used: 485 / 262,144 (0.19%)
  Top 15 most frequent tokens:
                          '.':   37 (5.1%)
                          ',':   17 (2.3%)
                          '-':   14 (1.9%)
                          '0':   14 (1.9%)
                          '।':   11 (1.5%)
                       ' the':    9 (1.2%)
                          ' ':    9 (1.2%)
                        ' to':    8 (1.1%)
                          '1':    7 (1.0%)
                          '5':    7 (1.0%)
                         ' [':    7 (1.0%)
                          ']':    7 (1.0%)
                       ' and':    6 (0.8%)
                          '2':    6 (0.8%)
                          '3':    5 (0.7%)
  Token character lengths: min=1, max=14, mean=3.87

--- Model: qwen3-asr-1.7b ---
  Total tokens: 1,382
  Unique tokens used: 493 / 151,643 (0.33%)
  Top 15 most frequent tokens:
                          '.':   33 (2.4%)
                          '�':   33 (2.4%)
                         ' �':   30 (2.2%)
                          '�':   23 (1.7%)
                         'ा�':   22 (1.6%)
                          '�':   22 (1.6%)
                          '�':   21 (1.5%)
                          '�':   18 (1.3%)
                          ',':   17 (1.2%)
                          '�':   16 (1.2%)
                          '�':   15 (1.1%)
                          '�':   15 (1.1%)
                          '0':   14 (1.0%)
                         ' �':   14 (1.0%)
                          '�':   13 (0.9%)
  Token character lengths: min=1, max=14, mean=2.29

================================================================================
  TEST 7: INDIC LANGUAGE TOKENIZATION DEEP DIVE
================================================================================

  Language                  Tokens  Ch/Tok    Tokens  Ch/Tok        Winner
  hindi                         20    4.05        79    1.03   gemma3-270m
  hindi_technical               15    4.73        62    1.15   gemma3-270m
  tamil                          9    5.22        51    0.92   gemma3-270m
  bengali                       12    4.17        52    0.96   gemma3-270m
  telugu                        11    4.00        64    0.69   gemma3-270m
  marathi                       13    5.00        61    1.07   gemma3-270m
  gujarati                      17    3.12        77    0.69   gemma3-270m
  kannada                       17    3.18        76    0.71   gemma3-270m
  malayalam                     16    3.81        83    0.73   gemma3-270m
  odia                          36    1.28        89    0.52   gemma3-270m
  punjabi                       26    2.00        75    0.69   gemma3-270m
  mixed_hindi_english           24    5.46        52    2.52   gemma3-270m
  mixed_code_switching          24    3.25        35    2.23   gemma3-270m

--- Hindi Tokenization Detail ---

  [gemma3-270m] (20 tokens)
  Text: भारत एक विविधताओं से भरा देश है। यहाँ की संस्कृति, भाषा और परंपराएँ अद्वितीय हैं।
  Tokens: ['भारत', ' एक', ' विविध', 'ताओं', ' से', ' भरा', ' देश', ' है', '।', ' यहाँ', ' की', ' संस्कृति', ',', ' भाषा', ' और', ' परंपरा', 'एँ', ' अद्वितीय', ' हैं', '।']

  [qwen3-asr-1.7b] (79 tokens)
  Text: भारत एक विविधताओं से भरा देश है। यहाँ की संस्कृति, भाषा और परंपराएँ अद्वितीय हैं।
  Tokens: ['भ', 'ा�', '�', 'त', ' �', '�', 'क', ' �', '�', 'ि�', '�', 'ि�', '�', 'त', 'ा�', '�', 'ं', ' स', 'े', ' �', '�', 'र', 'ा', ' �', '�', 'े�', '�', ' ह', '�', '�', '।', ' �', '�', 'ह', 'ा�', '�', ' क', 'ी', ' स', 'ं', 'स', '्�', '�', '�', '�', 'त', 'ि', ',', ' �', '�', 'ा�', '�', 'ा', ' �', '�', 'र', ' प', 'र', 'ं', 'प', 'र', 'ा�', '�', '�', '�', ' �', '�', 'द', '्�', '�', 'ि�', '�', 'ी', 'य', ' ह', '�', '�', 'ं', '।']

================================================================================
  TEST 8: COMPRESSION RATIO (bytes per token)
================================================================================

  Category          gemma3-270m (B/tok)  qwen3-asr-1.7b (B/tok)           Better
  English                         4.266                 4.596   qwen3-asr-1.7b
  Indic                          12.047                 2.487      gemma3-270m
  CJK                             5.421                 4.682      gemma3-270m
  Code                            2.904                 3.512   qwen3-asr-1.7b
  Mixed                           6.417                 3.540      gemma3-270m
  ASR                             4.702                 3.713      gemma3-270m

================================================================================
  TEST 9: SPECIAL INPUT HANDLING
================================================================================

--- Model: gemma3-270m ---
  ✓ empty string             : 0 tokens
  ✓ single space             : 1 tokens
  ✓ BOM                      : 1 tokens
  ✓ zero-width space         : 1 tokens
  ✓ zero-width joiner        : 1 tokens
  ✓ right-to-left mark       : 1 tokens
  ✓ null byte                : 1 tokens
  ✓ very long repeat         : 1250 tokens
  ✓ alternating scripts      : 6 tokens
  ✓ combining diacritics     : 6 tokens
  ✓ surrogate-safe emoji     : 50 tokens

--- Model: qwen3-asr-1.7b ---
  ✓ empty string             : 0 tokens
  ✓ single space             : 1 tokens
  ✓ BOM                      : 1 tokens
  ✓ zero-width space         : 1 tokens
  ✓ zero-width joiner        : 2 tokens
  ✓ right-to-left mark       : 2 tokens
  ✓ null byte                : 1 tokens
  ✓ very long repeat         : 1250 tokens
  ✓ alternating scripts      : 7 tokens
  ≈ combining diacritics     : 3 tokens (lossy)
  ✓ surrogate-safe emoji     : 50 tokens

================================================================================
  TEST 10: SUBWORD SEGMENTATION QUALITY
================================================================================

--- Model: gemma3-270m ---
  unhappiness                    → ['un', 'happiness']
  unbelievable                   → ['un', 'bel', 'ievable']
  internationalization           → ['international', 'ization']
  preprocessing                  → ['preprocessing']
  tokenization                   → ['token', 'ization']
  transformer                    → ['transformer']
  भारतीय                         → ['भारतीय']
  स्वतंत्रता                     → ['स्व', 'तंत्र', 'ता']
  अंतरराष्ट्रीय                  → ['अंतर', 'राष्ट्रीय']
  counterproductive              → ['counter', 'productive']
  misunderstanding               → ['mis', 'understanding']
  autobiography                  → ['aut', 'obi', 'ography']
  electromagnetic                → ['elect', 'romagnetic']

--- Model: qwen3-asr-1.7b ---
  unhappiness                    → ['un', 'h', 'appiness']
  unbelievable                   → ['un', 'belie', 'vable']
  internationalization           → ['international', 'ization']
  preprocessing                  → ['pre', 'processing']
  tokenization                   → ['token', 'ization']
  transformer                    → ['transform', 'er']
  भारतीय                         → ['भ', 'ा�', '�', 'त', 'ी', 'य']
  स्वतंत्रता                     → ['स', '्�', '�', 'त', 'ं', 'त', '्र', 'त', 'ा']
  अंतरराष्ट्रीय                  → ['अ', 'ं', 'त', 'र', 'र', 'ा�', '�', '्�', '�', '्र', 'ी', 'य']
  counterproductive              → ['counter', 'productive']
  misunderstanding               → ['mis', 'under', 'standing']
  autobiography                  → ['aut', 'obi', 'ography']
  electromagnetic                → ['elect', 'rom', 'agnetic']

================================================================================
  BENCHMARK COMPLETE
================================================================================

All 10 test suites finished.


## Results Summary: Gemma-3-270M vs Qwen3-ASR-1.7B Tokenizers

### Vocabulary
| Metric | Gemma-3-270M | Qwen3-ASR-1.7B |
|---|---|---|
| Vocab size | **262,144** | 151,643 |
| Byte-level tokens | 256 | 9 |
| Special tokens | 7 | 20 (incl. audio/vision) |

### Key Findings

**1. Indic Languages — Gemma dominates (3-5x more efficient)**
- Hindi: Gemma uses **20 tokens** vs Qwen's **79 tokens** (4.0 vs 1.0 chars/token)
- Qwen tokenizes Devanagari at the **byte level** — individual characters get split into replacement characters (`�`), meaning it has essentially no Indic vocabulary
- This holds across all Indic scripts: Tamil, Bengali, Telugu, Marathi, Gujarati, Kannada, Malayalam, Odia, Punjabi
- Gemma tokenizes Hindi morphologically: `भारत`, `विविध`, `ताओं`, `संस्कृति` — clean, meaningful subwords

**2. English — Near parity, slight Qwen edge**
- Both perform similarly on plain English text
- Qwen is slightly better on code (43 vs 52 tokens) and URLs (23 vs 30 tokens)
- Qwen handles `snake_case` better (13 vs 21 tokens)

**3. Speed — Gemma is faster**
- Gemma encodes ~1.4-1.7x faster across most inputs
- Batch throughput: Gemma 1.8M chars/s vs Qwen 1.1M chars/s
- Decoding speed is comparable

**4. Roundtrip Fidelity**
- Qwen: **perfect** roundtrip on all 50 tests
- Gemma: 1 failure (its own special tokens `<pad>`/`<unk>` get consumed during decode — expected behavior)
- Qwen has a **lossy** combining diacritics issue

**5. Compression Ratio (bytes per token)**
- Indic: Gemma **12.0 B/tok** vs Qwen **2.5 B/tok** — Gemma is ~5x more efficient
- English: Qwen slightly better (4.6 vs 4.3)
- Code: Qwen better (3.5 vs 2.9)

**6. Subword Segmentation Quality**
- Both produce linguistically sensible English splits (`un|happiness`, `international|ization`)
- Gemma produces **clean Hindi morphemes**: `भारतीय` (single token), `स्व|तंत्र|ता`
- Qwen shatters Hindi into **byte-level garbage**: `भ|ा�|�|त|ी|य`

### Bottom Line

| Dimension | Winner |
|---|---|
| Indic languages | **Gemma** (massively) |
| English text | Tie / slight Qwen edge |
| Code | **Qwen** |
| Speed | **Gemma** |
| Vocab coverage | **Gemma** (broader Unicode) |
| Roundtrip fidelity | **Qwen** (marginally) |

**Gemma-3-270M's tokenizer is dramatically superior for any Indic/multilingual workload** — its 262K vocabulary has dedicated Devanagari, Bengali, Tamil, and other Indic subwords. Qwen3-ASR-1.7B's tokenizer was clearly built for English/Chinese and falls back to byte-level encoding for Indic scripts, producing ~4x more tokens and semantically meaningless segmentations.