Thoroughly explore the repo at /home/ubuntu/training/gemma3n-E2B-asr-ft/. I need to understand:

1. The overall structure - list all files and directories
2. The model architecture - what kind of ASR model is this?
3. Inference support - how do you load the model and run transcription? Look for any inference scripts, model loading code, transcribe methods
4. Any existing benchmark or evaluation scripts
5. The training configuration - what languages, what base model
6. Dependencies and requirements

Read all key Python files, configs, and READMEs. I need enough detail to write a benchmark script that loads a checkpoint and runs inference on audio data.


I'll thoroughly explore the gemma3n-E2B-asr-ft repository to understand its structure, architecture, and inference capabilities.


Bash

find /home/ubuntu/training/gemma3n-E2B-asr-ft -type f -name "*.md" -o -name "*.txt" -o -name "*.json" -o -name "*.yaml" -o -name "*.yml" | head -30


Let me get a broader view of the directory structure:


cd /home/ubuntu/training/gemma3n-E2B-asr-ft && find . -maxdepth 3 -type d | head -50


.
./docs
./scripts
./artifacts
./artifacts/phase2
./artifacts/phase4
./vendor
./vendor/LlamaFactory
./vendor/LlamaFactory/requirements
./vendor/LlamaFactory/docs
./vendor/LlamaFactory/docker
./vendor/LlamaFactory/src
./vendor/LlamaFactory/tests
./vendor/LlamaFactory/scripts
./vendor/LlamaFactory/tests_v1
./vendor/LlamaFactory/assets
./vendor/LlamaFactory/.github
./vendor/LlamaFactory/examples
./tools
./configs
./benchmarks
./benchmarks/phase1
./.git
./.git/hooks
./.git/branches
./.git/logs
./.git/logs/refs
./.git/refs
./.git/refs/heads
./.git/refs/tags
./.git/refs/remotes
./.git/objects
./.git/objects/pack
./.git/objects/info
./.git/info


cd /home/ubuntu/training/gemma3n-E2B-asr-ft && ls -la


total 44
drwxrwxr-x 10 ubuntu ubuntu 4096 Mar 25 02:16 .
drwxrwxr-x 11 ubuntu ubuntu 4096 Mar 26 00:46 ..
drwxrwxr-x  4 ubuntu ubuntu 4096 Mar 25 02:16 artifacts
drwxrwxr-x  3 ubuntu ubuntu 4096 Mar 25 02:16 benchmarks
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 configs
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 docs
drwxrwxr-x  8 ubuntu ubuntu 4096 Mar 25 21:14 .git
-rw-rw-r--  1 ubuntu ubuntu  608 Mar 25 02:16 .gitignore
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 scripts
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 tools
drwxrwxr-x  3 ubuntu ubuntu 4096 Mar 25 02:16 vendor


Now let me examine the key files and directories:


cd /home/ubuntu/training/gemma3n-E2B-asr-ft && find . -maxdepth 2 -name "*.py" -o -name "*.md" -o -name "requirements*.txt" | grep -v vendor | sort


./docs/overnight_plan.md
./docs/perf_contract.md
./docs/phase2_runbook.md
./tools/bench_raw_audio_pipeline.py
./tools/build_bench_manifest.py
./tools/debug_full_ft_nan_probe.py
./tools/phase2_bench_decode_only.py
./tools/phase2_build_global_manifest.py
./tools/phase2_convert_shard_audio16.py
./tools/phase2_orchestrate_conversion.py
./tools/phase2_scan_inventory.py
./tools/phase2_validate_conversion.py
./tools/phase2_worker_sweep.py
./tools/phase3_fix_offsets.py
./tools/phase3_uri_bench.py
./tools/phase4_build_lf_dataset.py
./tools/phase4_rebuild_final.py
./tools/phase4_rebuild_splits.py
./tools/phase4_rebuild_v2.py
./tools/phase5_bs_sweep.py
./tools/phase6_preflight.py


     1→# Phase 2 Runbook: Zero-Resample Audio Conversion
     2→
     3→## Overview
     4→
     5→Convert the full 78M-sample corpus from mixed sample rates (16k/48k) to uniform
     6→16 kHz mono FLAC, keeping shard-level tar packaging. Post-conversion, the training
     7→dataloader needs only to decode FLAC — no runtime resampling.
     8→
     9→## Prerequisites
    10→
    11→```bash
    12→source /root/gemini-asr/lf_asr/.venv/bin/activate
    13→```
    14→
    15→## Step-by-Step Execution
    16→
    17→### Step A: Build Inventory
    18→
    19→```bash
    20→python /root/gemini-asr/lf_asr/tools/phase2_scan_inventory.py
    21→```
    22→
    23→Outputs:
    24→- `artifacts/phase2/shard_inventory.parquet`
    25→- `artifacts/phase2/conversion_queue.parquet` (sorted largest-first)
    26→
    27→### Step B+C: Run Conversion (Orchestrated)
    28→
    29→```bash
    30→python /root/gemini-asr/lf_asr/tools/phase2_orchestrate_conversion.py --workers 8
    31→```
    32→
    33→- Processes shards in parallel with `--workers` processes.
    34→- Resumable: re-run the same command to pick up where it left off.
    35→- Per-shard status in `artifacts/phase2/shard_status/<shard_id>.json`.
    36→- Per-shard index in `artifacts/phase2/shard_indices/<shard_id>.parquet`.
    37→
    38→Atomic swap strategy:
    39→1. Write `audio_16k.tar.tmp` alongside original `audio.tar`
    40→2. Validate: count match + random decode checks
    41→3. `audio.tar` → `audio_48k.bak`
    42→4. `audio_16k.tar.tmp` → `audio.tar`
    43→5. Remove `.bak` only after validation passes
    44→
    45→### Step D: Build Global Manifest
    46→
    47→```bash
    48→python /root/gemini-asr/lf_asr/tools/phase2_build_global_manifest.py
    49→```
    50→
    51→Outputs:
    52→- `artifacts/phase2/global_manifest.parquet` (all rows)
    53→- `artifacts/phase2/train.parquet` (96%)
    54→- `artifacts/phase2/dev.parquet` (2%)
    55→- `artifacts/phase2/test.parquet` (2%)
    56→
    57→Split is deterministic by `video_id` MD5 hash — leakage-safe.
    58→
    59→### Step E: Validate
    60→
    61→```bash
    62→python /root/gemini-asr/lf_asr/tools/phase2_validate_conversion.py
    63→```
    64→
    65→Outputs:
    66→- `artifacts/phase2/validation_report.json`
    67→- `artifacts/phase2/validation_report.md`
    68→
    69→### Step F: Decode-Only Benchmark
    70→
    71→```bash
    72→python /root/gemini-asr/lf_asr/tools/phase2_bench_decode_only.py
    73→```
    74→
    75→Outputs:
    76→- `artifacts/phase2/decode_only_bench_results.json`
    77→- `artifacts/phase2/decode_only_bench_report.md`
    78→
    79→## Failure Recovery
    80→
    81→If the orchestrator is interrupted:
    82→1. Re-run the same command — it skips completed shards automatically.
    83→2. Check `artifacts/phase2/shard_status/` for individual shard results.
    84→3. If a shard fails after max retries, its `.json` status has the error details.
    85→
    86→If a shard's atomic swap fails mid-way:
    87→- If `audio_48k.bak` exists but `audio.tar` is missing: rename `.bak` back.
    88→- If `audio_16k.tar.tmp` exists: the conversion completed but swap didn't — can be manually swapped after validation.
    89→
    90→## Tuning
    91→
    92→- **Workers:** Default 8. With 240 CPU cores and I/O-bound tar reads, can go up to 16-24.
    93→  Monitor with `htop`. Beyond 16 workers, Phase 1 showed diminishing returns.
    94→- **Disk I/O:** Conversion is read-write. If on shared storage, fewer workers may be better.
    95→- **Memory:** Each worker holds one audio file in memory at a time (~50MB max for long 48kHz files).
    96→  8 workers ≈ 400MB peak. Safe with 3TB RAM.
    97→


     1→# Performance Contract — ASR Fine-Tuning Pipeline
     2→
     3→## Non-Negotiables
     4→
     5→### 1. Data Integrity
     6→- **All valid rows retained.** No data loss except rows that fail integrity checks (corrupt FLAC, missing metadata, zero-length audio).
     7→- Every dropped row must be logged with reason.
     8→
     9→### 2. No Runtime Resample/Decode in Training Path
    10→- The final training dataloader must read pre-converted 16 kHz mono FLAC (or PCM) directly.
    11→- No `torchaudio.transforms.Resample` or `soundfile.read` inside the training hot path.
    12→- All format conversion happens offline in Phase 2 preprocessing.
    13→
    14→### 3. GPU Data-Wait Budget
    15→- **Target:** Dataloader wall-time < 10% of step wall-time in smoke tests.
    16→- Measured via `torch.cuda.Event` timing around `next(dataloader)` vs full step.
    17→- If exceeded, pipeline must be rearchitected before scaling up.
    18→
    19→### 4. Reproducible Benchmarking
    20→- Fixed random seed: `42` for all benchmark scripts.
    21→- Fixed benchmark manifest: `benchmarks/phase1/bench_manifest.parquet`.
    22→- All benchmark results saved as JSON (machine-readable) + Markdown (human-readable).
    23→- Environment matrix (torch, torchaudio, transformers, LlamaFactory commit) recorded in every report.
    24→
    25→## Measurement Protocol
    26→
    27→| Metric | Tool | Target |
    28→|--------|------|--------|
    29→| samples/sec | `bench_raw_audio_pipeline.py` | Establish baseline |
    30→| audio-seconds/sec | `bench_raw_audio_pipeline.py` | Establish baseline |
    31→| p50/p95 latency | `bench_raw_audio_pipeline.py` | Establish baseline |
    32→| CPU utilization | `psutil` | Document |
    33→| Dataloader % of step | Phase 5 smoke test | < 10% |
    34→| Row drop rate | Phase 2 conversion logs | < 0.1% |
    35→


     1→# Overnight Execution Plan — Phases 3-6
     2→
     3→## STATUS TRACKER
     4→
     5→### Phase 3: URI Loader Patch [DONE]
     6→- [x] Patch mm_plugin.py: tar:// URI support with pread + FD cache
     7→- [x] Patch converter.py: skip os.path.isfile for URI media
     8→- [x] Tests: test_converter.py URI passthrough, test_mm_plugin.py tar decode — PASSED
     9→- [x] URI decode microbench: PASS at 4-24 workers (111-8223% of baseline)
    10→- [x] Fixed tar offset computation (3.7M offsets corrected)
    11→- [x] 10k random URI decode: 10,000/10,000 OK, 0 failures, 0 SR mismatches
    12→
    13→### Phase 4: LlamaFactory Dataset Build [DONE]
    14→- [x] Build train/dev/test parquet with audios=["tar://..."], instruction with <audio>, output=transcript
    15→- [x] ALL buckets included (dispose/golden/redo/finalsftdata/indicvoices — no filtering)
    16→- [x] Duration bucket columns (0-4s, 4-8s, 8-16s, 16-32s, >32s)
    17→- [x] I/O locality ordering: group by tar_path, 256-row blocks, shuffle blocks not rows
    18→- [x] dataset_info.json with formatting: alpaca
    19→- [x] disable_shuffling: true in config
    20→- [x] train: 74,950,509 rows (148,021h), dev: 1,572,083 (3,108h), test: 1,566,152 (3,105h)
    21→
    22→### Phase 5: Training Config [DONE]
    23→- [x] gemma3n_asr_lora_smoke.yaml created
    24→- [x] template: gemma3n, trust_remote_code: true
    25→- [x] freeze_multi_modal_projector: false
    26→- [x] bf16, flash_attn: auto
    27→- [x] preprocessing_num_workers: 64
    28→- [x] dataloader_num_workers: 8, persistent_workers, prefetch_factor: 4
    29→- [x] TORCH_NUM_THREADS=1, OMP_NUM_THREADS=1, MKL_NUM_THREADS=1
    30→- [ ] Per-bucket batch size binary search (requires GPU access)
    31→- [ ] ulimit -n 1048576
    32→
    33→### Phase 6: Smoke Test [READY TO LAUNCH]
    34→- [x] 10k random URI decode check: ALL PASSED
    35→- [x] Zero SR mismatches
    36→- [ ] 2k-step smoke on 8 GPUs
    37→- [ ] Report: train_samples_per_sec, train_steps_per_sec, GPU util, dataloader stalls
    38→- [ ] 300+ steps no OOM, no NaN loss
    39→- [ ] Final greenlight doc
    40→
    41→## KEY DECISIONS
    42→- ALL data used (all buckets). No filtering by quality bucket.
    43→- No runtime resample. ValueError if SR mismatch.
    44→- Keep tar+offset, never extract 78M files.
    45→- Block-shuffle for I/O locality, disable_shuffling: true.
    46→
    47→## LAUNCH COMMAND
    48→```bash
    49→ulimit -n 1048576
    50→export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 TORCH_NUM_THREADS=1
    51→source /root/gemini-asr/lf_asr/.venv/bin/activate
    52→cd /root/gemini-asr/lf_asr/vendor/LlamaFactory
    53→torchrun --nproc_per_node=8 --master_port=29500 \
    54→    -m llamafactory.train \
    55→    /root/gemini-asr/lf_asr/configs/gemma3n_asr_lora_smoke.yaml
    56→```
    57→


Now let me examine the configs and scripts:


total 24
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 .
drwxrwxr-x 10 ubuntu ubuntu 4096 Mar 25 02:16 ..
-rw-rw-r--  1 ubuntu ubuntu 1254 Mar 25 02:16 gemma3n_asr_full_ft.yaml
-rw-rw-r--  1 ubuntu ubuntu 1271 Mar 25 02:16 gemma3n_asr_lora_smoke.yaml
-rw-rw-r--  1 ubuntu ubuntu 1004 Mar 25 02:16 smoke_100step.yaml
-rw-rw-r--  1 ubuntu ubuntu  999 Mar 25 02:16 smoke_500step.yaml


     1→### Gemma3n ASR LoRA Smoke Test Config
     2→### Phase 5: Throughput-first training configuration
     3→
     4→### Model
     5→model_name_or_path: /workspace/gemini-asr/gemma-model
     6→trust_remote_code: true
     7→
     8→### Method
     9→stage: sft
    10→do_train: true
    11→finetuning_type: lora
    12→lora_target: self_attn.q_proj,self_attn.k_proj,self_attn.v_proj,self_attn.o_proj,mlp.gate_proj,mlp.up_proj,mlp.down_proj
    13→lora_rank: 16
    14→lora_alpha: 32
    15→
    16→### Multimodal
    17→freeze_multi_modal_projector: false
    18→
    19→### Dataset
    20→dataset_dir: /root/gemini-asr/lf_asr/artifacts/phase4
    21→dataset: asr_train
    22→eval_dataset: asr_dev
    23→template: gemma3n
    24→
    25→### Data loading
    26→preprocessing_num_workers: 64
    27→dataloader_num_workers: 8
    28→dataloader_persistent_workers: true
    29→dataloader_prefetch_factor: 4
    30→dataloader_pin_memory: true
    31→remove_unused_columns: false
    32→disable_shuffling: true
    33→
    34→### Training
    35→per_device_train_batch_size: 4
    36→per_device_eval_batch_size: 4
    37→gradient_accumulation_steps: 4
    38→learning_rate: 2.0e-4
    39→num_train_epochs: 1.0
    40→max_steps: 2000
    41→lr_scheduler_type: cosine
    42→warmup_steps: 100
    43→
    44→### Precision
    45→bf16: true
    46→flash_attn: auto
    47→
    48→### Logging
    49→logging_steps: 10
    50→save_steps: 500
    51→eval_steps: 500
    52→save_total_limit: 3
    53→
    54→### Output
    55→output_dir: /root/gemini-asr/lf_asr/outputs/smoke_run
    56→overwrite_output_dir: true
    57→report_to: none
    58→
    59→### Misc
    60→seed: 42
    61→cutoff_len: 2048
    62→


     1→### Gemma3n ASR Full Fine-Tune Config
     2→### Full parameter training (not LoRA)
     3→
     4→### Model
     5→model_name_or_path: /workspace/gemini-asr/gemma-model
     6→trust_remote_code: true
     7→
     8→### Method
     9→stage: sft
    10→do_train: true
    11→do_eval: true
    12→finetuning_type: full
    13→
    14→### Multimodal
    15→freeze_multi_modal_projector: false
    16→freeze_vision_tower: false
    17→
    18→### Dataset
    19→dataset_dir: /root/gemini-asr/lf_asr/artifacts/phase4
    20→dataset: asr_train
    21→eval_dataset: asr_dev
    22→template: gemma3n
    23→
    24→### Data loading
    25→preprocessing_num_workers: 64
    26→dataloader_num_workers: 8
    27→dataloader_persistent_workers: true
    28→dataloader_prefetch_factor: 4
    29→dataloader_pin_memory: true
    30→remove_unused_columns: false
    31→disable_shuffling: true
    32→
    33→### Training
    34→per_device_train_batch_size: 8
    35→per_device_eval_batch_size: 4
    36→gradient_accumulation_steps: 8
    37→learning_rate: 5.0e-5
    38→num_train_epochs: 1.0
    39→lr_scheduler_type: cosine
    40→warmup_ratio: 0.02
    41→
    42→### Memory optimization
    43→gradient_checkpointing: true
    44→
    45→### Precision + Speed
    46→bf16: true
    47→flash_attn: fa2
    48→
    49→### Logging + Checkpointing
    50→logging_steps: 10
    51→save_steps: 5000
    52→eval_strategy: steps
    53→eval_steps: 5000
    54→save_total_limit: 5
    55→
    56→### Output
    57→output_dir: /root/gemini-asr/lf_asr/outputs/full_ft_run
    58→overwrite_output_dir: true
    59→report_to: wandb
    60→run_name: gemma-3n
    61→
    62→### Misc
    63→seed: 42
    64→cutoff_len: 2048
    65→


total 28
drwxrwxr-x  2 ubuntu ubuntu 4096 Mar 25 02:16 .
drwxrwxr-x 10 ubuntu ubuntu 4096 Mar 25 02:16 ..
-rwxrwxr-x  1 ubuntu ubuntu 1600 Mar 25 02:16 launch_full_ft.sh
-rwxrwxr-x  1 ubuntu ubuntu 1399 Mar 25 02:16 launch_smoke.sh
-rwxrwxr-x  1 ubuntu ubuntu 1697 Mar 25 02:16 push_ckpt_r2.sh
-rwxrwxr-x  1 ubuntu ubuntu 6367 Mar 25 02:16 watch_training.sh


     1→#!/bin/bash
     2→# Launch Gemma3n ASR full fine-tune on 8x B200 GPUs
     3→set -euo pipefail
     4→
     5→# Thread controls
     6→export OMP_NUM_THREADS=1
     7→export MKL_NUM_THREADS=1
     8→export OPENBLAS_NUM_THREADS=1
     9→export NUMEXPR_NUM_THREADS=1
    10→export TORCH_NUM_THREADS=1
    11→
    12→# Audio decode safety
    13→export LLAMAFACTORY_ALLOW_RESAMPLE=0
    14→export LLAMAFACTORY_AUDIO_DECODE_FALLBACK=1
    15→export LLAMAFACTORY_AUDIO_DECODE_FALLBACK_SEC=1.0
    16→
    17→# CUDA optimizations
    18→export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
    19→export CUDA_LAUNCH_BLOCKING=0
    20→export NCCL_P2P_DISABLE=0
    21→
    22→# HF
    23→export HF_TOKEN="${HF_TOKEN:-}"
    24→export HF_HOME="${HF_HOME:-/workspace/.hf_home}"
    25→
    26→# File descriptor limit for tar FD cache
    27→ulimit -n 1048576
    28→
    29→# Activate venv
    30→source /root/gemini-asr/lf_asr/.venv/bin/activate
    31→
    32→# Preflight check
    33→echo "Running preflight..."
    34→python3 /root/gemini-asr/lf_asr/tools/phase6_preflight.py
    35→PREFLIGHT_EXIT=$?
    36→if [ $PREFLIGHT_EXIT -ne 0 ]; then
    37→    echo "PREFLIGHT FAILED (exit $PREFLIGHT_EXIT). Aborting launch."
    38→    exit 1
    39→fi
    40→
    41→# Launch
    42→cd /root/gemini-asr/lf_asr/vendor/LlamaFactory
    43→
    44→CONFIG="${1:-/root/gemini-asr/lf_asr/configs/gemma3n_asr_full_ft.yaml}"
    45→echo ""
    46→echo "Starting training..."
    47→echo "Config: $CONFIG"
    48→echo "Model: /workspace/gemini-asr/gemma-model"
    49→echo "Dataset: /root/gemini-asr/lf_asr/artifacts/phase4/"
    50→echo ""
    51→
    52→llamafactory-cli train "$CONFIG" \
    53→    2>&1 | tee /root/gemini-asr/lf_asr/outputs/train_$(date +%Y%m%d_%H%M%S).log
    54→


Now let me examine the tool scripts to understand inference and benchmarking:


     1→#!/usr/bin/env python3
     2→"""Benchmark the raw audio pipeline: tar read → FLAC decode → resample 16k mono → ndarray.
     3→
     4→Runs a scaling sweep across different worker counts and reports throughput + latency.
     5→"""
     6→
     7→import io
     8→import json
     9→import os
    10→import sys
    11→import tarfile
    12→import time
    13→from collections import defaultdict
    14→from concurrent.futures import ProcessPoolExecutor, as_completed
    15→from dataclasses import dataclass, asdict
    16→from pathlib import Path
    17→
    18→import numpy as np
    19→import pandas as pd
    20→import psutil
    21→import soundfile as sf
    22→
    23→# Optional torchaudio for resampling comparison
    24→try:
    25→    import torchaudio
    26→    import torch
    27→    HAS_TORCHAUDIO = True
    28→except ImportError:
    29→    HAS_TORCHAUDIO = False
    30→
    31→SEED = 42
    32→MANIFEST_PATH = "/root/gemini-asr/lf_asr/benchmarks/phase1/bench_manifest.parquet"
    33→OUTPUT_DIR = "/root/gemini-asr/lf_asr/benchmarks/phase1"
    34→WORKER_SWEEP = [4, 8, 12, 16, 24, 32]
    35→TARGET_SR = 16000
    36→
    37→
    38→@dataclass
    39→class SampleResult:
    40→    tar_member_name: str
    41→    duration_s: float
    42→    orig_sr: int
    43→    decode_time_ms: float
    44→    resample_time_ms: float
    45→    total_time_ms: float
    46→    success: bool
    47→    error: str = ""
    48→
    49→
    50→def process_single_sample(args: tuple) -> SampleResult:
    51→    """Process a single sample: tar read → decode → resample → ndarray."""
    52→    audio_tar_path, tar_member_name, expected_duration = args
    53→
    54→    t_start = time.perf_counter()
    55→    try:
    56→        # Read from tar
    57→        with tarfile.open(audio_tar_path, "r") as tf:
    58→            member = tf.getmember(tar_member_name)
    59→            f = tf.extractfile(member)
    60→            audio_bytes = f.read()
    61→
    62→        # Decode FLAC
    63→        t_decode_start = time.perf_counter()
    64→        audio_data, sr = sf.read(io.BytesIO(audio_bytes), dtype="float32")
    65→        t_decode_end = time.perf_counter()
    66→
    67→        # Convert to mono if stereo
    68→        if audio_data.ndim > 1:
    69→            audio_data = audio_data.mean(axis=1)
    70→
    71→        # Resample to 16k if needed
    72→        t_resample_start = time.perf_counter()
    73→        if sr != TARGET_SR:
    74→            if HAS_TORCHAUDIO:
    75→                tensor = torch.from_numpy(audio_data).unsqueeze(0)
    76→                resampler = torchaudio.transforms.Resample(sr, TARGET_SR)
    77→                tensor = resampler(tensor)
    78→                audio_data = tensor.squeeze(0).numpy()
    79→            else:
    80→                # Fallback: simple linear interpolation (not production quality)
    81→                ratio = TARGET_SR / sr
    82→                n_out = int(len(audio_data) * ratio)
    83→                indices = np.linspace(0, len(audio_data) - 1, n_out)
    84→                audio_data = np.interp(indices, np.arange(len(audio_data)), audio_data).astype(np.float32)
    85→        t_resample_end = time.perf_counter()
    86→
    87→        t_end = time.perf_counter()
    88→
    89→        return SampleResult(
    90→            tar_member_name=tar_member_name,
    91→            duration_s=expected_duration,
    92→            orig_sr=sr,
    93→            decode_time_ms=(t_decode_end - t_decode_start) * 1000,
    94→            resample_time_ms=(t_resample_end - t_resample_start) * 1000,
    95→            total_time_ms=(t_end - t_start) * 1000,
    96→            success=True,
    97→        )
    98→
    99→    except Exception as e:
   100→        t_end = time.perf_counter()
   101→        return SampleResult(
   102→            tar_member_name=tar_member_name,
   103→            duration_s=expected_duration,
   104→            orig_sr=0,
   105→            decode_time_ms=0,
   106→            resample_time_ms=0,
   107→            total_time_ms=(t_end - t_start) * 1000,
   108→            success=False,
   109→            error=str(e),
   110→        )
   111→
   112→
   113→def run_benchmark(manifest: pd.DataFrame, num_workers: int) -> dict:
   114→    """Run the benchmark with a given number of workers."""
   115→    tasks = list(zip(
   116→        manifest["audio_tar_path"],
   117→        manifest["tar_member_name"],
   118→        manifest["duration_s"],
   119→    ))
   120→
   121→    cpu_before = psutil.cpu_percent(interval=None, percpu=False)
   122→
   123→    t_wall_start = time.perf_counter()
   124→
   125→    results = []
   126→    with ProcessPoolExecutor(max_workers=num_workers) as executor:
   127→        # Start CPU monitoring
   128→        psutil.cpu_percent(interval=None)
   129→
   130→        futures = {executor.submit(process_single_sample, t): t for t in tasks}
   131→        for future in as_completed(futures):
   132→            results.append(future.result())
   133→
   134→    t_wall_end = time.perf_counter()
   135→    wall_time = t_wall_end - t_wall_start
   136→
   137→    cpu_after = psutil.cpu_percent(interval=None, percpu=False)
   138→
   139→    # Compute metrics
   140→    successes = [r for r in results if r.success]
   141→    failures = [r for r in results if not r.success]
   142→
   143→    total_times = [r.total_time_ms for r in successes]
   144→    total_audio_s = sum(r.duration_s for r in successes)
   145→
   146→    if not successes:
   147→        return {
   148→            "workers": num_workers,
   149→            "total_samples": len(tasks),
   150→            "success_count": 0,
   151→            "failure_count": len(failures),
   152→            "error": "All samples failed",
   153→        }
   154→
   155→    latencies = sorted(total_times)
   156→    p50_idx = int(len(latencies) * 0.50)
   157→    p95_idx = min(int(len(latencies) * 0.95), len(latencies) - 1)
   158→
   159→    # Count how many needed resampling
   160→    resample_count = sum(1 for r in successes if r.orig_sr != TARGET_SR)
   161→
   162→    return {
   163→        "workers": num_workers,
   164→        "wall_time_s": round(wall_time, 2),
   165→        "total_samples": len(tasks),
   166→        "success_count": len(successes),
   167→        "failure_count": len(failures),
   168→        "samples_per_sec": round(len(successes) / wall_time, 2),
   169→        "audio_seconds_per_sec": round(total_audio_s / wall_time, 2),
   170→        "total_audio_hours": round(total_audio_s / 3600, 3),
   171→        "p50_latency_ms": round(latencies[p50_idx], 2),
   172→        "p95_latency_ms": round(latencies[p95_idx], 2),
   173→        "min_latency_ms": round(latencies[0], 2),
   174→        "max_latency_ms": round(latencies[-1], 2),
   175→        "mean_latency_ms": round(np.mean(total_times), 2),
   176→        "cpu_utilization_pct": round(cpu_after, 1),
   177→        "resample_needed_count": resample_count,
   178→        "resample_fraction": round(resample_count / len(successes), 4),
   179→        "mean_decode_ms": round(np.mean([r.decode_time_ms for r in successes]), 2),
   180→        "mean_resample_ms": round(np.mean([r.resample_time_ms for r in successes if r.orig_sr != TARGET_SR]) if resample_count > 0 else 0, 2),
   181→    }
   182→
   183→
   184→def get_environment_info() -> dict:
   185→    """Collect environment information."""
   186→    env = {
   187→        "python_version": sys.version.split()[0],
   188→        "cpu_count": os.cpu_count(),
   189→        "ram_gb": round(psutil.virtual_memory().total / (1024**3), 1),
   190→    }
   191→
   192→    try:
   193→        env["torch_version"] = torch.__version__
   194→    except Exception:
   195→        env["torch_version"] = "not installed"
   196→
   197→    try:
   198→        env["torchaudio_version"] = torchaudio.__version__
   199→    except Exception:
   200→        env["torchaudio_version"] = "not installed"
   201→
   202→    try:
   203→        import transformers
   204→        env["transformers_version"] = transformers.__version__
   205→    except Exception:
   206→        env["transformers_version"] = "not installed"
   207→
   208→    try:
   209→        import subprocess
   210→        result = subprocess.run(
   211→            ["git", "rev-parse", "HEAD"],
   212→            cwd="/root/gemini-asr/lf_asr/vendor/LlamaFactory",
   213→            capture_output=True, text=True,
   214→        )
   215→        env["llamafactory_commit"] = result.stdout.strip()
   216→    except Exception:
   217→        env["llamafactory_commit"] = "unknown"
   218→
   219→    try:
   220→        result = subprocess.run(
   221→            ["nvidia-smi", "--query-gpu=name,memory.total", "--format=csv,noheader"],
   222→            capture_output=True, text=True,
   223→        )
   224→        env["gpus"] = result.stdout.strip().split("\n")
   225→    except Exception:
   226→        env["gpus"] = []
   227→
   228→    env["soundfile_version"] = sf.__version__
   229→
   230→    return env
   231→
   232→
   233→def generate_report(env: dict, sweep_results: list[dict]) -> str:
   234→    """Generate markdown report."""
   235→    lines = []
   236→    lines.append("# Phase 1: Raw Audio Pipeline Benchmark Report\n")
   237→    lines.append(f"**Date:** {time.strftime('%Y-%m-%d %H:%M:%S')}\n")
   238→
   239→    # Environment
   240→    lines.append("## Environment Matrix\n")
   241→    lines.append("| Component | Version |")
   242→    lines.append("|-----------|---------|")
   243→    lines.append(f"| Python | {env['python_version']} |")
   244→    lines.append(f"| PyTorch | {env['torch_version']} |")
   245→    lines.append(f"| torchaudio | {env['torchaudio_version']} |")
   246→    lines.append(f"| transformers | {env['transformers_version']} |")
   247→    lines.append(f"| soundfile | {env['soundfile_version']} |")
   248→    lines.append(f"| LlamaFactory commit | `{env['llamafactory_commit'][:12]}` |")
   249→    lines.append(f"| CPUs | {env['cpu_count']} |")
   250→    lines.append(f"| RAM | {env['ram_gb']} GB |")
   251→    if env.get("gpus"):
   252→        lines.append(f"| GPUs | {', '.join(env['gpus'])} |")
   253→    lines.append("")
   254→
   255→    # Throughput table
   256→    lines.append("## Throughput vs Workers\n")
   257→    lines.append("| Workers | Wall Time (s) | Samples/sec | Audio-sec/sec | p50 Latency (ms) | p95 Latency (ms) | CPU % | Failures |")
   258→    lines.append("|---------|--------------|-------------|---------------|-------------------|-------------------|-------|----------|")
   259→    for r in sweep_results:
   260→        if "error" in r:
   261→            lines.append(f"| {r['workers']} | - | - | - | - | - | - | {r['failure_count']} |")
   262→        else:
   263→            lines.append(
   264→                f"| {r['workers']} | {r['wall_time_s']} | {r['samples_per_sec']} | "
   265→                f"{r['audio_seconds_per_sec']} | {r['p50_latency_ms']} | {r['p95_latency_ms']} | "
   266→                f"{r['cpu_utilization_pct']} | {r['failure_count']} |"
   267→            )
   268→    lines.append("")
   269→
   270→    # Resample analysis
   271→    if sweep_results and "resample_fraction" in sweep_results[0]:
   272→        r0 = sweep_results[0]
   273→        lines.append("## Resample Analysis\n")
   274→        lines.append(f"- Samples needing resample (SR != 16kHz): **{r0['resample_needed_count']}** / {r0['success_count']} ({r0['resample_fraction']*100:.1f}%)")
   275→        lines.append(f"- Mean decode time: **{r0['mean_decode_ms']:.1f} ms**")
   276→        if r0['resample_needed_count'] > 0:
   277→            lines.append(f"- Mean resample time (when needed): **{r0['mean_resample_ms']:.1f} ms**")
   278→        lines.append("")
   279→
   280→    # Go/No-Go
   281→    lines.append("## Go/No-Go: Runtime Decode + Resample\n")
   282→
   283→    # Find best throughput
   284→    best = max(sweep_results, key=lambda r: r.get("audio_seconds_per_sec", 0))
   285→    best_throughput = best.get("audio_seconds_per_sec", 0)
   286→    resample_frac = sweep_results[0].get("resample_fraction", 0) if sweep_results else 0
   287→
   288→    if resample_frac > 0.3:
   289→        lines.append(f"**DECISION: NO-GO for runtime resample.**\n")
   290→        lines.append(f"- {resample_frac*100:.1f}% of samples require resampling from 48kHz → 16kHz.")
   291→        lines.append(f"- Best throughput with {best['workers']} workers: {best_throughput} audio-sec/sec.")
   292→        lines.append(f"- Runtime resampling adds per-sample latency and wastes CPU in the training loop.")
   293→        lines.append(f"- **Recommendation:** Pre-convert all audio to 16kHz mono FLAC in Phase 2.")
   294→    else:
   295→        lines.append(f"**DECISION: GO for runtime decode (resample fraction low: {resample_frac*100:.1f}%).**\n")
   296→        lines.append(f"- Best throughput with {best['workers']} workers: {best_throughput} audio-sec/sec.")
   297→
   298→    lines.append("")
   299→    lines.append("## Recommended Target Ingest Format (Phase 2)\n")
   300→    lines.append("1. **Audio format:** 16 kHz, mono, FLAC (lossless, compact)")
   301→    lines.append("2. **Storage:** Re-tar with consistent naming, one tar per shard")
   302→    lines.append("3. **Index:** Unified manifest parquet with byte offsets for O(1) random access")
   303→    lines.append("4. **Pre-compute:** All resampling done offline; training path is decode-only")
   304→    lines.append("5. **Validation:** SHA256 integrity checks on every converted file")
   305→    lines.append("")
   306→
   307→    return "\n".join(lines)
   308→
   309→
   310→def main():
   311→    print("Loading benchmark manifest...")
   312→    manifest = pd.read_parquet(MANIFEST_PATH)
   313→    print(f"  Samples: {len(manifest)}")
   314→    print(f"  Total audio: {manifest['duration_s'].sum() / 3600:.2f} hours")
   315→
   316→    env = get_environment_info()
   317→    print(f"  Environment: Python {env['python_version']}, torch {env['torch_version']}")
   318→
   319→    sweep_results = []
   320→    for n_workers in WORKER_SWEEP:
   321→        print(f"\n--- Running with {n_workers} workers ---")
   322→        result = run_benchmark(manifest, n_workers)
   323→        sweep_results.append(result)
   324→        if "error" not in result:
   325→            print(f"  Samples/sec: {result['samples_per_sec']}")
   326→            print(f"  Audio-sec/sec: {result['audio_seconds_per_sec']}")
   327→            print(f"  p50: {result['p50_latency_ms']}ms, p95: {result['p95_latency_ms']}ms")
   328→            print(f"  Failures: {result['failure_count']}")
   329→        else:
   330→            print(f"  ERROR: {result['error']}")
   331→
   332→    # Save JSON results
   333→    os.makedirs(OUTPUT_DIR, exist_ok=True)
   334→    json_path = os.path.join(OUTPUT_DIR, "bench_results.json")
   335→    with open(json_path, "w") as f:
   336→        json.dump({"environment": env, "sweep_results": sweep_results}, f, indent=2)
   337→    print(f"\nJSON results saved to {json_path}")
   338→
   339→    # Save markdown report
   340→    report = generate_report(env, sweep_results)
   341→    report_path = os.path.join(OUTPUT_DIR, "report.md")
   342→    with open(report_path, "w") as f:
   343→        f.write(report)
   344→    print(f"Markdown report saved to {report_path}")
   345→
   346→    # Print summary
   347→    print("\n" + "=" * 60)
   348→    print("SUMMARY")
   349→    print("=" * 60)
   350→    best = max(sweep_results, key=lambda r: r.get("samples_per_sec", 0))
   351→    print(f"Best throughput: {best.get('samples_per_sec', 0)} samples/sec at {best['workers']} workers")
   352→    print(f"Best audio throughput: {best.get('audio_seconds_per_sec', 0)} audio-sec/sec")
   353→
   354→
   355→if __name__ == "__main__":
   356→    main()
   357→


     1→#!/usr/bin/env python3
     2→"""Phase 3: URI decode microbenchmark — compare tar:// URI loader vs Phase 2 baseline."""
     3→
     4→import io
     5→import json
     6→import os
     7→import sys
     8→import time
     9→from concurrent.futures import ProcessPoolExecutor, as_completed
    10→
    11→import numpy as np
    12→import pandas as pd
    13→
    14→os.environ["OMP_NUM_THREADS"] = "1"
    15→os.environ["MKL_NUM_THREADS"] = "1"
    16→os.environ["TORCH_NUM_THREADS"] = "1"
    17→
    18→ARTIFACTS_DIR = "/root/gemini-asr/lf_asr/artifacts/phase2"
    19→WORKER_SWEEP = [4, 8, 12, 16, 24, 32]
    20→TARGET_SAMPLES = 2000
    21→SEED = 42
    22→
    23→
    24→def build_uri_manifest(global_manifest_path: str, n_samples: int, seed: int) -> pd.DataFrame:
    25→    rng = np.random.default_rng(seed)
    26→    df = pd.read_parquet(global_manifest_path)
    27→    df["_dbucket"] = pd.cut(df["duration_s"], bins=[0, 5, 10, float("inf")],
    28→                            labels=["short", "medium", "long"], right=True)
    29→    groups = df.groupby(["language", "_dbucket"], observed=True)
    30→    per_group = max(1, n_samples // groups.ngroups)
    31→    sampled = []
    32→    for _, group_df in groups:
    33→        n = min(per_group, len(group_df))
    34→        sampled.append(group_df.sample(n=n, random_state=rng.integers(2**31)))
    35→    result = pd.concat(sampled, ignore_index=True).sample(frac=1, random_state=seed).reset_index(drop=True)
    36→    # Build URI
    37→    result["uri"] = "tar://" + result["tar_path"] + "|" + result["tar_offset_data"].astype(str) + "|" + result["tar_nbytes"].astype(str)
    38→    return result.drop(columns=["_dbucket"])
    39→
    40→
    41→def decode_uri_sample(args):
    42→    """Decode via the patched LlamaFactory loader."""
    43→    uri, expected_duration = args
    44→    t_start = time.perf_counter()
    45→    try:
    46→        # Use the actual patched loader
    47→        sys.path.insert(0, "/root/gemini-asr/lf_asr/vendor/LlamaFactory/src")
    48→        from llamafactory.data.mm_plugin import _load_audio_from_tar_uri
    49→        audio, sr = _load_audio_from_tar_uri(uri)
    50→        t_end = time.perf_counter()
    51→        return {
    52→            "success": True,
    53→            "duration_s": expected_duration,
    54→            "sr": sr,
    55→            "total_ms": (t_end - t_start) * 1000,
    56→        }
    57→    except Exception as e:
    58→        t_end = time.perf_counter()
    59→        return {
    60→            "success": False,
    61→            "duration_s": expected_duration,
    62→            "total_ms": (t_end - t_start) * 1000,
    63→            "error": str(e),
    64→        }
    65→
    66→
    67→def run_sweep(manifest, workers):
    68→    tasks = list(zip(manifest["uri"], manifest["duration_s"]))
    69→    t_start = time.perf_counter()
    70→    results = []
    71→    with ProcessPoolExecutor(max_workers=workers) as executor:
    72→        futures = {executor.submit(decode_uri_sample, t): t for t in tasks}
    73→        for f in as_completed(futures):
    74→            results.append(f.result())
    75→    t_end = time.perf_counter()
    76→    wall = t_end - t_start
    77→    ok = [r for r in results if r["success"]]
    78→    fail = [r for r in results if not r["success"]]
    79→    if not ok:
    80→        return {"workers": workers, "error": "all failed", "failure_count": len(fail)}
    81→    total_audio = sum(r["duration_s"] for r in ok)
    82→    latencies = sorted(r["total_ms"] for r in ok)
    83→    return {
    84→        "workers": workers,
    85→        "wall_time_s": round(wall, 2),
    86→        "samples_per_sec": round(len(ok) / wall, 2),
    87→        "audio_seconds_per_sec": round(total_audio / wall, 2),
    88→        "p50_latency_ms": round(latencies[int(len(latencies) * 0.50)], 2),
    89→        "p95_latency_ms": round(latencies[min(int(len(latencies) * 0.95), len(latencies) - 1)], 2),
    90→        "failure_count": len(fail),
    91→    }
    92→
    93→
    94→def main():
    95→    manifest = build_uri_manifest(os.path.join(ARTIFACTS_DIR, "global_manifest.parquet"), TARGET_SAMPLES, SEED)
    96→    print(f"Samples: {len(manifest)}, Audio: {manifest['duration_s'].sum()/3600:.2f}h")
    97→
    98→    # Phase 2 baseline
    99→    p2_baseline = [
   100→        {"workers": 4, "audio_seconds_per_sec": 24.95},
   101→        {"workers": 8, "audio_seconds_per_sec": 266.07},
   102→        {"workers": 12, "audio_seconds_per_sec": 369.92},
   103→        {"workers": 16, "audio_seconds_per_sec": 450.16},
   104→        {"workers": 24, "audio_seconds_per_sec": 549.1},
   105→        {"workers": 32, "audio_seconds_per_sec": 589.12},
   106→    ]
   107→    p2_map = {r["workers"]: r["audio_seconds_per_sec"] for r in p2_baseline}
   108→
   109→    results = []
   110→    for w in WORKER_SWEEP:
   111→        print(f"\n--- {w} workers ---")
   112→        r = run_sweep(manifest, w)
   113→        results.append(r)
   114→        p2_tput = p2_map.get(w, 0)
   115→        if "error" not in r:
   116→            ratio = r["audio_seconds_per_sec"] / p2_tput * 100 if p2_tput > 0 else 0
   117→            gate = "PASS" if ratio >= 90 else "FAIL"
   118→            print(f"  URI: {r['audio_seconds_per_sec']} audio-sec/sec, "
   119→                  f"P2 baseline: {p2_tput}, ratio: {ratio:.1f}% [{gate}]")
   120→
   121→    # Report
   122→    print(f"\n{'='*60}")
   123→    print("URI DECODE BENCHMARK vs PHASE 2 BASELINE")
   124→    print(f"{'='*60}")
   125→    print(f"{'Workers':>8} {'URI a-s/s':>12} {'P2 a-s/s':>12} {'Ratio':>8} {'Gate':>6}")
   126→    all_pass = True
   127→    for r in results:
   128→        w = r["workers"]
   129→        p2 = p2_map.get(w, 0)
   130→        uri_tput = r.get("audio_seconds_per_sec", 0)
   131→        ratio = uri_tput / p2 * 100 if p2 > 0 else 0
   132→        gate = "PASS" if ratio >= 90 else "FAIL"
   133→        if ratio < 90:
   134→            all_pass = False
   135→        print(f"{w:>8} {uri_tput:>12.1f} {p2:>12.1f} {ratio:>7.1f}% {gate:>6}")
   136→
   137→    print(f"\nOverall: {'ALL PASS' if all_pass else 'SOME FAIL'}")
   138→
   139→    with open(os.path.join(ARTIFACTS_DIR, "uri_bench_results.json"), "w") as f:
   140→        json.dump({"results": results, "phase2_baseline": p2_baseline, "all_pass": all_pass}, f, indent=2)
   141→
   142→
   143→if __name__ == "__main__":
   144→    main()
   145→


Now let me check the LlamaFactory integration and find how models are loaded:


find /home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory -name "*.py" -path "*mm_plugin*" -o -name "*.py" -path "*converter*" | head -10


     1→# Copyright 2025 HuggingFace Inc. and the LlamaFactory team.
     2→#
     3→# This code is inspired by the HuggingFace's Transformers library.
     4→# https://github.com/huggingface/transformers/blob/v4.40.0/src/transformers/models/llava/processing_llava.py
     5→#
     6→# Licensed under the Apache License, Version 2.0 (the "License");
     7→# you may not use this file except in compliance with the License.
     8→# You may obtain a copy of the License at
     9→#
    10→#     http://www.apache.org/licenses/LICENSE-2.0
    11→#
    12→# Unless required by applicable law or agreed to in writing, software
    13→# distributed under the License is distributed on an "AS IS" BASIS,
    14→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    15→# See the License for the specific language governing permissions and
    16→# limitations under the License.
    17→
    18→import inspect
    19→import math
    20→import os
    21→import re
    22→import threading
    23→from collections import OrderedDict
    24→from copy import deepcopy
    25→from dataclasses import dataclass
    26→from io import BytesIO
    27→from typing import TYPE_CHECKING, BinaryIO, Literal, NotRequired, Optional, TypedDict, Union
    28→
    29→import numpy as np
    30→import torch
    31→import torchaudio
    32→from transformers.image_utils import get_image_size, is_valid_image, make_flat_list_of_images, to_numpy_array
    33→
    34→
    35→# ---------------------------------------------------------------------------
    36→# tar:// URI audio loader — per-process LRU FD cache + thread-safe pread
    37→# ---------------------------------------------------------------------------
    38→_TAR_URI_PREFIX = "tar://"
    39→_TAR_FD_CACHE_MAX = 256
    40→
    41→
    42→class _TarFDCache:
    43→    """Per-process LRU cache of open file descriptors for tar files."""
    44→
    45→    def __init__(self, max_size: int = _TAR_FD_CACHE_MAX):
    46→        self._cache: OrderedDict[str, int] = OrderedDict()
    47→        self._max = max_size
    48→        self._lock = threading.Lock()
    49→
    50→    def get_fd(self, tar_path: str) -> int:
    51→        with self._lock:
    52→            if tar_path in self._cache:
    53→                self._cache.move_to_end(tar_path)
    54→                return self._cache[tar_path]
    55→            fd = os.open(tar_path, os.O_RDONLY)
    56→            self._cache[tar_path] = fd
    57→            while len(self._cache) > self._max:
    58→                _, old_fd = self._cache.popitem(last=False)
    59→                try:
    60→                    os.close(old_fd)
    61→                except OSError:
    62→                    pass
    63→            return fd
    64→
    65→
    66→_tar_fd_cache = _TarFDCache()
    67→
    68→
    69→def _parse_tar_uri(uri: str) -> tuple[str, int, int]:
    70→    """Parse 'tar:///abs/path.tar|offset|nbytes' → (path, offset, nbytes)."""
    71→    body = uri[len(_TAR_URI_PREFIX):]
    72→    parts = body.rsplit("|", 2)
    73→    if len(parts) != 3:
    74→        raise ValueError(f"Invalid tar URI (expected tar://<path>|<offset>|<nbytes>): {uri}")
    75→    return parts[0], int(parts[1]), int(parts[2])
    76→
    77→
    78→def _load_audio_from_tar_uri(uri: str) -> tuple["torch.Tensor", int]:
    79→    """Load audio from a tar:// URI using pread for thread-safe random access."""
    80→    import soundfile as sf
    81→
    82→    tar_path, offset, nbytes = _parse_tar_uri(uri)
    83→    fd = _tar_fd_cache.get_fd(tar_path)
    84→    raw = os.pread(fd, nbytes, offset)
    85→    if len(raw) != nbytes:
    86→        raise IOError(f"pread returned {len(raw)} bytes, expected {nbytes} from {uri}")
    87→    audio_np, sr = sf.read(BytesIO(raw), dtype="float32")
    88→    # soundfile returns (samples,) or (samples, channels) — convert to torch [C, T]
    89→    audio_tensor = torch.from_numpy(audio_np)
    90→    if audio_tensor.ndim == 1:
    91→        audio_tensor = audio_tensor.unsqueeze(0)  # [1, T]
    92→    else:
    93→        audio_tensor = audio_tensor.T  # [channels, T]
    94→    return audio_tensor, sr
    95→
    96→
    97→def _get_decode_fallback_num_samples(sampling_rate: float) -> int:
    98→    fallback_seconds = 1.0
    99→    try:
   100→        fallback_seconds = float(os.environ.get("LLAMAFACTORY_AUDIO_DECODE_FALLBACK_SEC", "1.0"))
   101→    except ValueError:
   102→        pass
   103→
   104→    return max(1, int(round(fallback_seconds * float(sampling_rate))))
   105→
   106→
   107→def _should_fallback_on_decode_error() -> bool:
   108→    return os.environ.get("LLAMAFACTORY_AUDIO_DECODE_FALLBACK", "1") == "1"
   109→from transformers.models.mllama.processing_mllama import (
   110→    convert_sparse_cross_attention_mask_to_dense,
   111→    get_cross_attention_token_mask,
   112→)
   113→from transformers.video_utils import make_batched_videos
   114→from typing_extensions import override
   115→
   116→from ..extras import logging
   117→from ..extras.constants import AUDIO_PLACEHOLDER, IGNORE_INDEX, IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER
   118→from ..extras.packages import is_pillow_available, is_pyav_available, is_transformers_version_greater_than
   119→
   120→logger = logging.get_logger(__name__)
   121→
   122→_audio_decode_error_count = 0
   123→_audio_decode_error_lock = threading.Lock()
   124→
   125→
   126→def _log_decode_fallback(audio: "AudioInput", err: Exception, fallback_len: int, sampling_rate: float) -> None:
   127→    global _audio_decode_error_count
   128→
   129→    with _audio_decode_error_lock:
   130→        _audio_decode_error_count += 1
   131→        err_idx = _audio_decode_error_count
   132→
   133→    # Avoid log spam on very large runs.
   134→    if err_idx <= 5 or err_idx % 100 == 0:
   135→        try:
   136→            audio_repr = str(audio)
   137→            if len(audio_repr) > 160:
   138→                audio_repr = audio_repr[:160] + "..."
   139→            err_repr = f"{type(err).__name__}"
   140→            try:
   141→                err_repr += f": {err}"
   142→            except Exception:
   143→                err_repr += " (str() failed)"
   144→        except Exception:
   145→            audio_repr = "<unrepresentable>"
   146→            err_repr = "<unrepresentable>"
   147→
   148→        logger.warning_rank0(
   149→            f"Audio decode failed (count={err_idx}); using zero-audio fallback "
   150→            f"({fallback_len} samples @ {int(sampling_rate)}Hz). "
   151→            f"Source={audio_repr}. Error={err_repr}"
   152→        )
   153→
   154→
   155→if is_pillow_available():
   156→    from PIL import Image
   157→    from PIL.Image import Image as ImageObject
   158→
   159→
   160→if is_pyav_available():
   161→    import av
   162→
   163→
   164→if TYPE_CHECKING:
   165→    from av.stream import Stream
   166→    from numpy.typing import NDArray
   167→    from transformers import PreTrainedTokenizer, ProcessorMixin
   168→    from transformers.feature_extraction_sequence_utils import SequenceFeatureExtractor
   169→    from transformers.image_processing_utils import BaseImageProcessor
   170→    from transformers.video_processing_utils import BaseVideoProcessor
   171→
   172→    class EncodedImage(TypedDict):
   173→        path: str | None
   174→        bytes: bytes | None
   175→
   176→    ImageInput = Union[str, bytes, EncodedImage, BinaryIO, ImageObject]
   177→    VideoInput = Union[str, BinaryIO, list[list[ImageInput]]]
   178→    AudioInput = Union[str, BinaryIO, NDArray]
   179→
   180→    class RegularizedImageOutput(TypedDict):
   181→        images: list[ImageObject]
   182→
   183→    class RegularizedVideoOutput(TypedDict):
   184→        videos: list[list[ImageObject]]
   185→        durations: list[float]
   186→        fps_per_video: NotRequired[list[float]]
   187→
   188→    class RegularizedAudioOutput(TypedDict):
   189→        audios: list[NDArray]
   190→        sampling_rates: list[float]
   191→
   192→    class MMProcessor(ProcessorMixin):
   193→        patch_size: int
   194→        image_seq_length: int
   195→        num_additional_image_tokens: int
   196→        vision_feature_select_strategy: Literal["default", "full"]
   197→
   198→        def _get_number_of_features(self, orig_height: int, orig_width: int, height: int, width: int) -> int:
   199→            pass
   200→


find /home/ubuntu/training/gemma3n-E2B-asr-ft -type f -name "*.json" -path "*artifacts*" | head -10


     1→{
     2→  "asr_train": {
     3→    "file_name": "train.parquet",
     4→    "formatting": "alpaca",
     5→    "columns": {
     6→      "prompt": "instruction",
     7→      "query": "input",
     8→      "response": "output",
     9→      "audios": "audios"
    10→    }
    11→  },
    12→  "asr_dev": {
    13→    "file_name": "dev.parquet",
    14→    "formatting": "alpaca",
    15→    "columns": {
    16→      "prompt": "instruction",
    17→      "query": "input",
    18→      "response": "output",
    19→      "audios": "audios"
    20→    }
    21→  },
    22→  "asr_train_0_4s": {
    23→    "file_name": "train_bucket_0-4s.parquet",
    24→    "formatting": "alpaca",
    25→    "columns": {
    26→      "prompt": "instruction",
    27→      "query": "input",
    28→      "response": "output",
    29→      "audios": "audios"
    30→    }
    31→  },
    32→  "sweep_0_4s": {
    33→    "file_name": "sweep_bucket_0-4s.parquet",
    34→    "formatting": "alpaca",
    35→    "columns": {
    36→      "prompt": "instruction",
    37→      "query": "input",
    38→      "response": "output",
    39→      "audios": "audios"
    40→    }
    41→  },
    42→  "asr_train_4_8s": {
    43→    "file_name": "train_bucket_4-8s.parquet",
    44→    "formatting": "alpaca",
    45→    "columns": {
    46→      "prompt": "instruction",
    47→      "query": "input",
    48→      "response": "output",
    49→      "audios": "audios"
    50→    }
    51→  },
    52→  "sweep_4_8s": {
    53→    "file_name": "sweep_bucket_4-8s.parquet",
    54→    "formatting": "alpaca",
    55→    "columns": {
    56→      "prompt": "instruction",
    57→      "query": "input",
    58→      "response": "output",
    59→      "audios": "audios"
    60→    }
    61→  },
    62→  "asr_train_8_16s": {
    63→    "file_name": "train_bucket_8-16s.parquet",
    64→    "formatting": "alpaca",
    65→    "columns": {
    66→      "prompt": "instruction",
    67→      "query": "input",
    68→      "response": "output",
    69→      "audios": "audios"
    70→    }
    71→  },
    72→  "sweep_8_16s": {
    73→    "file_name": "sweep_bucket_8-16s.parquet",
    74→    "formatting": "alpaca",
    75→    "columns": {
    76→      "prompt": "instruction",
    77→      "query": "input",
    78→      "response": "output",
    79→      "audios": "audios"
    80→    }
    81→  },
    82→  "asr_train_16_32s": {
    83→    "file_name": "train_bucket_16-32s.parquet",
    84→    "formatting": "alpaca",
    85→    "columns": {
    86→      "prompt": "instruction",
    87→      "query": "input",
    88→      "response": "output",
    89→      "audios": "audios"
    90→    }
    91→  },
    92→  "sweep_16_32s": {
    93→    "file_name": "sweep_bucket_16-32s.parquet",
    94→    "formatting": "alpaca",
    95→    "columns": {
    96→      "prompt": "instruction",
    97→      "query": "input",
    98→      "response": "output",
    99→      "audios": "audios"
   100→    }
   101→  },
   102→  "asr_train_gt32s": {
   103→    "file_name": "train_bucket_gt32s.parquet",
   104→    "formatting": "alpaca",
   105→    "columns": {
   106→      "prompt": "instruction",
   107→      "query": "input",
   108→      "response": "output",
   109→      "audios": "audios"
   110→    }
   111→  },
   112→  "sweep_gt32s": {
   113→    "file_name": "sweep_bucket_gt32s.parquet",
   114→    "formatting": "alpaca",
   115→    "columns": {
   116→      "prompt": "instruction",
   117→      "query": "input",
   118→      "response": "output",
   119→      "audios": "audios"
   120→    }
   121→  }
   122→}


Now let me look at some of the phase tools to understand the data processing pipeline:


     1→#!/usr/bin/env python3
     2→"""Phase 4: Build LlamaFactory-ready parquet datasets from Phase 2 manifests.
     3→
     4→Output format: Alpaca with audios column
     5→- instruction: contains exactly one <audio> token
     6→- input: language tag
     7→- output: transcript
     8→- audios: ["tar://<path>|<offset>|<nbytes>"]
     9→- duration_bucket: for analysis
    10→
    11→I/O locality ordering: group by tar_path, 256-row blocks, shuffle blocks not rows.
    12→"""
    13→
    14→import hashlib
    15→import json
    16→import os
    17→import sys
    18→
    19→import numpy as np
    20→import pandas as pd
    21→
    22→ARTIFACTS_DIR = "/root/gemini-asr/lf_asr/artifacts/phase2"
    23→OUTPUT_DIR = "/root/gemini-asr/lf_asr/artifacts/phase4"
    24→BLOCK_SIZE = 256
    25→SEED = 42
    26→
    27→
    28→def build_duration_bucket(duration_s: pd.Series) -> pd.Series:
    29→    """Assign duration buckets."""
    30→    bins = [0, 4, 8, 16, 32, float("inf")]
    31→    labels = ["0-4s", "4-8s", "8-16s", "16-32s", ">32s"]
    32→    return pd.cut(duration_s, bins=bins, labels=labels, right=True)
    33→
    34→
    35→def block_shuffle(df: pd.DataFrame, block_size: int, seed: int) -> pd.DataFrame:
    36→    """Group by tar_path, form blocks of block_size rows, shuffle blocks."""
    37→    rng = np.random.default_rng(seed)
    38→
    39→    # Sort by tar_path for I/O locality within blocks
    40→    df = df.sort_values("tar_path").reset_index(drop=True)
    41→
    42→    # Assign block IDs
    43→    n = len(df)
    44→    block_ids = np.arange(n) // block_size
    45→
    46→    # Shuffle block IDs
    47→    unique_blocks = np.unique(block_ids)
    48→    rng.shuffle(unique_blocks)
    49→    block_order = {old: new for new, old in enumerate(unique_blocks)}
    50→    df["_block_order"] = [block_order[b] for b in block_ids]
    51→
    52→    # Sort by shuffled block order (preserves within-block order)
    53→    df = df.sort_values("_block_order").reset_index(drop=True)
    54→    df = df.drop(columns=["_block_order"])
    55→    return df
    56→
    57→
    58→def build_lf_dataset(split_name: str) -> pd.DataFrame:
    59→    """Build LlamaFactory-format dataset for a split."""
    60→    src_path = os.path.join(ARTIFACTS_DIR, f"{split_name}.parquet")
    61→    df = pd.read_parquet(src_path)
    62→    print(f"  {split_name}: {len(df):,} rows loaded")
    63→
    64→    # Filter to rows with transcripts
    65→    has_tx = df["transcript"].notna() & (df["transcript"].str.strip() != "")
    66→    dropped = (~has_tx).sum()
    67→    df = df[has_tx].copy()
    68→    if dropped > 0:
    69→        print(f"    Dropped {dropped:,} rows with empty transcript")
    70→
    71→    # Build URI
    72→    df["uri"] = ("tar://" + df["tar_path"] + "|" +
    73→                 df["tar_offset_data"].astype(str) + "|" +
    74→                 df["tar_nbytes"].astype(str))
    75→
    76→    # Duration bucket
    77→    df["duration_bucket"] = build_duration_bucket(df["duration_s"])
    78→
    79→    # Build LF format
    80→    lf = pd.DataFrame({
    81→        "instruction": "<audio>\nTranscribe the audio.",
    82→        "input": "Language: " + df["language"].values,
    83→        "output": df["transcript"].values,
    84→        "audios": df["uri"].apply(lambda x: [x]),
    85→        "duration_s": df["duration_s"].values,
    86→        "duration_bucket": df["duration_bucket"].values,
    87→        "language": df["language"].values,
    88→        "tar_path": df["tar_path"].values,
    89→    })
    90→
    91→    # Block shuffle for I/O locality (only for train)
    92→    if split_name == "train":
    93→        print(f"    Block-shuffling {len(lf):,} rows (block_size={BLOCK_SIZE})...")
    94→        lf = block_shuffle(lf, BLOCK_SIZE, SEED)
    95→
    96→    return lf
    97→
    98→
    99→def main():
   100→    os.makedirs(OUTPUT_DIR, exist_ok=True)
   101→
   102→    for split in ["train", "dev", "test"]:
   103→        print(f"\nBuilding {split}...")
   104→        lf = build_lf_dataset(split)
   105→
   106→        # Save parquet
   107→        out_path = os.path.join(OUTPUT_DIR, f"{split}.parquet")
   108→        lf.to_parquet(out_path, index=False)
   109→        print(f"  Saved {out_path}: {len(lf):,} rows")
   110→
   111→        # Stats
   112→        print(f"  Duration buckets:")
   113→        for bucket, group in lf.groupby("duration_bucket", observed=True):
   114→            print(f"    {bucket}: {len(group):,} ({100*len(group)/len(lf):.1f}%)")
   115→        print(f"  Languages: {lf['language'].nunique()}")
   116→        print(f"  Total hours: {lf['duration_s'].sum()/3600:.1f}")
   117→
   118→    # Write dataset_info.json
   119→    dataset_info = {
   120→        "asr_train": {
   121→            "file_name": "train.parquet",
   122→            "formatting": "alpaca",
   123→            "columns": {
   124→                "prompt": "instruction",
   125→                "query": "input",
   126→                "response": "output",
   127→                "audios": "audios"
   128→            }
   129→        },
   130→        "asr_dev": {
   131→            "file_name": "dev.parquet",
   132→            "formatting": "alpaca",
   133→            "columns": {
   134→                "prompt": "instruction",
   135→                "query": "input",
   136→                "response": "output",
   137→                "audios": "audios"
   138→            }
   139→        },
   140→        "asr_test": {
   141→            "file_name": "test.parquet",
   142→            "formatting": "alpaca",
   143→            "columns": {
   144→                "prompt": "instruction",
   145→                "query": "input",
   146→                "response": "output",
   147→                "audios": "audios"
   148→            }
   149→        }
   150→    }


Now let me check what model architecture is expected and look at templates:


find /home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory -name "*template*" -type f | head -5


grep -r "gemma3n" /home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src --include="*.py" -l


grep -A 20 "gemma3n" /home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/data/template.py | head -40


    name="gemma3n",
    format_user=StringFormatter(slots=["<start_of_turn>user\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]),
    format_assistant=StringFormatter(slots=["{{content}}<end_of_turn>\n"]),
    format_system=StringFormatter(slots=["{{content}}\n\n"]),
    format_observation=StringFormatter(
        slots=["<start_of_turn>tool\n{{content}}<end_of_turn>\n<start_of_turn>model\n"]
    ),
    format_prefix=EmptyFormatter(slots=[{"bos_token"}]),
    stop_words=["<end_of_turn>"],
    replace_eos=True,
    mm_plugin=get_mm_plugin("gemma3n", image_token="<image_soft_token>", audio_token="<audio_soft_token>"),
    template_class=Llama2Template,
)

register_template(
    name="glm4",
    format_user=StringFormatter(slots=["<|user|>\n{{content}}<|assistant|>"]),
    format_assistant=StringFormatter(slots=["\n{{content}}"]),
    format_system=StringFormatter(slots=["<|system|>\n{{content}}"]),
    format_function=FunctionFormatter(slots=["{{content}}"], tool_format="glm4"),
    format_observation=StringFormatter(slots=["<|observation|>\n{{content}}<|assistant|>"]),
    format_tools=ToolFormatter(tool_format="glm4"),
    format_prefix=EmptyFormatter(slots=["[gMASK]<sop>"]),
    stop_words=["<|user|>", "<|observation|>"],
    efficient_eos=True,
)

# copied from glm4 template
register_template(


Let me look at how the model is loaded and understand the inference pipeline:


grep -r "def.*inference\|def.*generate\|def.*transcribe" /home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src --include="*.py" -l | head -10


find /home/ubuntu/training/gemma3n-E2B-asr-ft -name "*.py" -path "*chat*" -o -name "*.py" -path "*inference*" | grep -v __pycache__ | head -10


     1→# Copyright 2025 THUDM and the LlamaFactory team.
     2→#
     3→# This code is inspired by the THUDM's ChatGLM implementation.
     4→# https://github.com/THUDM/ChatGLM-6B/blob/main/cli_demo.py
     5→#
     6→# Licensed under the Apache License, Version 2.0 (the "License");
     7→# you may not use this file except in compliance with the License.
     8→# You may obtain a copy of the License at
     9→#
    10→#     http://www.apache.org/licenses/LICENSE-2.0
    11→#
    12→# Unless required by applicable law or agreed to in writing, software
    13→# distributed under the License is distributed on an "AS IS" BASIS,
    14→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    15→# See the License for the specific language governing permissions and
    16→# limitations under the License.
    17→
    18→import asyncio
    19→import os
    20→from collections.abc import AsyncGenerator, Generator
    21→from threading import Thread
    22→from typing import TYPE_CHECKING, Any, Optional
    23→
    24→from ..extras.constants import EngineName
    25→from ..extras.misc import torch_gc
    26→from ..hparams import get_infer_args
    27→
    28→
    29→if TYPE_CHECKING:
    30→    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    31→    from .base_engine import BaseEngine, Response
    32→
    33→
    34→def _start_background_loop(loop: "asyncio.AbstractEventLoop") -> None:
    35→    asyncio.set_event_loop(loop)
    36→    loop.run_forever()
    37→
    38→
    39→class ChatModel:
    40→    r"""General class for chat models. Backed by huggingface or vllm engines.
    41→
    42→    Supports both sync and async methods.
    43→    Sync methods: chat(), stream_chat() and get_scores().
    44→    Async methods: achat(), astream_chat() and aget_scores().
    45→    """
    46→
    47→    def __init__(self, args: Optional[dict[str, Any]] = None) -> None:
    48→        model_args, data_args, finetuning_args, generating_args = get_infer_args(args)
    49→
    50→        if model_args.infer_backend == EngineName.HF:
    51→            from .hf_engine import HuggingfaceEngine
    52→
    53→            self.engine: BaseEngine = HuggingfaceEngine(model_args, data_args, finetuning_args, generating_args)
    54→        elif model_args.infer_backend == EngineName.VLLM:
    55→            try:
    56→                from .vllm_engine import VllmEngine
    57→
    58→                self.engine: BaseEngine = VllmEngine(model_args, data_args, finetuning_args, generating_args)
    59→            except ImportError as e:
    60→                raise ImportError(
    61→                    "vLLM not install, you may need to run `pip install vllm`\n"
    62→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    63→                ) from e
    64→        elif model_args.infer_backend == EngineName.SGLANG:
    65→            try:
    66→                from .sglang_engine import SGLangEngine
    67→
    68→                self.engine: BaseEngine = SGLangEngine(model_args, data_args, finetuning_args, generating_args)
    69→            except ImportError as e:
    70→                raise ImportError(
    71→                    "SGLang not install, you may need to run `pip install sglang[all]`\n"
    72→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    73→                ) from e
    74→        elif model_args.infer_backend == EngineName.KT:
    75→            try:
    76→                from .kt_engine import KTransformersEngine
    77→
    78→                self.engine: BaseEngine = KTransformersEngine(model_args, data_args, finetuning_args, generating_args)
    79→            except ImportError as e:
    80→                raise ImportError(
    81→                    "KTransformers not install, you may need to run `pip install ktransformers`\n"
    82→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    83→                ) from e
    84→        else:
    85→            raise NotImplementedError(f"Unknown backend: {model_args.infer_backend}")
    86→
    87→        self._loop = asyncio.new_event_loop()
    88→        self._thread = Thread(target=_start_background_loop, args=(self._loop,), daemon=True)
    89→        self._thread.start()
    90→
    91→    def chat(
    92→        self,
    93→        messages: list[dict[str, str]],
    94→        system: Optional[str] = None,
    95→        tools: Optional[str] = None,
    96→        images: Optional[list["ImageInput"]] = None,
    97→        videos: Optional[list["VideoInput"]] = None,
    98→        audios: Optional[list["AudioInput"]] = None,
    99→        **input_kwargs,
   100→    ) -> list["Response"]:
   101→        r"""Get a list of responses of the chat model."""
   102→        task = asyncio.run_coroutine_threadsafe(
   103→            self.achat(messages, system, tools, images, videos, audios, **input_kwargs), self._loop
   104→        )
   105→        return task.result()
   106→
   107→    async def achat(
   108→        self,
   109→        messages: list[dict[str, str]],
   110→        system: Optional[str] = None,
   111→        tools: Optional[str] = None,
   112→        images: Optional[list["ImageInput"]] = None,
   113→        videos: Optional[list["VideoInput"]] = None,
   114→        audios: Optional[list["AudioInput"]] = None,
   115→        **input_kwargs,
   116→    ) -> list["Response"]:
   117→        r"""Asynchronously get a list of responses of the chat model."""
   118→        return await self.engine.chat(messages, system, tools, images, videos, audios, **input_kwargs)
   119→
   120→    def stream_chat(
   121→        self,
   122→        messages: list[dict[str, str]],
   123→        system: Optional[str] = None,
   124→        tools: Optional[str] = None,
   125→        images: Optional[list["ImageInput"]] = None,
   126→        videos: Optional[list["VideoInput"]] = None,
   127→        audios: Optional[list["AudioInput"]] = None,
   128→        **input_kwargs,
   129→    ) -> Generator[str, None, None]:
   130→        r"""Get the response token-by-token of the chat model."""
   131→        generator = self.astream_chat(messages, system, tools, images, videos, audios, **input_kwargs)
   132→        while True:
   133→            try:
   134→                task = asyncio.run_coroutine_threadsafe(generator.__anext__(), self._loop)
   135→                yield task.result()
   136→            except StopAsyncIteration:
   137→                break
   138→
   139→    async def astream_chat(
   140→        self,
   141→        messages: list[dict[str, str]],
   142→        system: Optional[str] = None,
   143→        tools: Optional[str] = None,
   144→        images: Optional[list["ImageInput"]] = None,
   145→        videos: Optional[list["VideoInput"]] = None,
   146→        audios: Optional[list["AudioInput"]] = None,
   147→        **input_kwargs,
   148→    ) -> AsyncGenerator[str, None]:
   149→        r"""Asynchronously get the response token-by-token of the chat model."""
   150→        async for new_token in self.engine.stream_chat(
   151→            messages, system, tools, images, videos, audios, **input_kwargs
   152→        ):
   153→            yield new_token
   154→
   155→    def get_scores(
   156→        self,
   157→        batch_input: list[str],
   158→        **input_kwargs,
   159→    ) -> list[float]:
   160→        r"""Get a list of scores of the reward model."""
   161→        task = asyncio.run_coroutine_threadsafe(self.aget_scores(batch_input, **input_kwargs), self._loop)
   162→        return task.result()
   163→
   164→    async def aget_scores(
   165→        self,
   166→        batch_input: list[str],
   167→        **input_kwargs,
   168→    ) -> list[float]:
   169→        r"""Asynchronously get a list of scores of the reward model."""
   170→        return await self.engine.get_scores(batch_input, **input_kwargs)
   171→
   172→
   173→def run_chat() -> None:
   174→    if os.name != "nt":
   175→        try:
   176→            import readline  # noqa: F401
   177→        except ImportError:
   178→            print("Install `readline` for a better experience.")
   179→
   180→    chat_model = ChatModel()
   181→    messages = []
   182→    print("Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.")
   183→
   184→    while True:
   185→        try:
   186→            query = input("\nUser: ")
   187→        except UnicodeDecodeError:
   188→            print("Detected decoding error at the inputs, please set the terminal encoding to utf-8.")
   189→            continue
   190→        except Exception:
   191→            raise
   192→
   193→        if query.strip() == "exit":
   194→            break
   195→
   196→        if query.strip() == "clear":
   197→            messages = []
   198→            torch_gc()
   199→            print("History has been removed.")
   200→            continue


Let me check the requirements and look at the README in the vendor directory:


find /home/ubuntu/training/gemma3n-E2B-asr-ft -name "requirements*.txt" | grep -v vendor


(Bash completed with no output)


1→![# LLaMA Factory](assets/logo.png)
     2→
     3→[![GitHub Repo stars](https://img.shields.io/github/stars/hiyouga/LLaMA-Factory?style=social)](https://github.com/hiyouga/LLaMA-Factory/stargazers)
     4→[![GitHub last commit](https://img.shields.io/github/last-commit/hiyouga/LLaMA-Factory)](https://github.com/hiyouga/LLaMA-Factory/commits/main)
     5→[![GitHub contributors](https://img.shields.io/github/contributors/hiyouga/LLaMA-Factory?color=orange)](https://github.com/hiyouga/LLaMA-Factory/graphs/contributors)
     6→[![GitHub workflow](https://github.com/hiyouga/LLaMA-Factory/actions/workflows/tests.yml/badge.svg)](https://github.com/hiyouga/LLaMA-Factory/actions/workflows/tests.yml)
     7→[![PyPI](https://img.shields.io/pypi/v/llamafactory)](https://pypi.org/project/llamafactory/)
     8→[![Citation](https://img.shields.io/badge/citation-1000+-green)](https://scholar.google.com/scholar?cites=12620864006390196564)
     9→[![Docker Pulls](https://img.shields.io/docker/pulls/hiyouga/llamafactory)](https://hub.docker.com/r/hiyouga/llamafactory/tags)
    10→
    11→[![Twitter](https://img.shields.io/twitter/follow/llamafactory_ai)](https://twitter.com/llamafactory_ai)
    12→[![Discord](assets/thirdparty/discord.svg)](https://discord.gg/rKfvV9r9FK)
    13→[![WeChat](https://img.shields.io/badge/WeChat-User%20Group-blue?logo=wechat)](https://github.com/hiyouga/llamafactory-community)
    14→[![Blog](https://img.shields.io/badge/Hugo-Official%20Blog-blue?logo=hugo)](https://blog.llamafactory.net/en/)
    15→
    16→[![Open in Colab](assets/thirdparty/colab.svg)](https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing)
    17→[![Open in DSW](assets/thirdparty/dsw.svg)](https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory)
    18→[![Open in Lab4ai](assets/thirdparty/lab4ai.svg)](https://www.lab4ai.cn/course/detail?id=7c13e60f6137474eb40f6fd3983c0f46&utm_source=LLaMA-Factory)
    19→[![Open in Online](assets/thirdparty/online.svg)](https://www.llamafactory.com.cn/?utm_source=LLaMA-Factory)
    20→[![Open in Spaces](https://img.shields.io/badge/🤗-Open%20in%20Spaces-blue)](https://huggingface.co/spaces/hiyouga/LLaMA-Board)
    21→[![Open in Studios](https://img.shields.io/badge/ModelScope-Open%20in%20Studios-blue)](https://modelscope.cn/studios/hiyouga/LLaMA-Board)
    22→[![Open in Novita](https://img.shields.io/badge/Novita-Deploy%20Template-blue)](https://novita.ai/templates-library/105981?sharer=88115474-394e-4bda-968e-b88e123d0c47)
    23→
    24→### Used by [Amazon](https://aws.amazon.com/cn/blogs/machine-learning/how-apoidea-group-enhances-visual-information-extraction-from-banking-documents-with-multimodal-models-using-llama-factory-on-amazon-sagemaker-hyperpod/), [NVIDIA](https://developer.nvidia.com/rtx/ai-toolkit), [Aliyun](https://help.aliyun.com/zh/pai/use-cases/fine-tune-a-llama-3-model-with-llama-factory), etc.
    25→
    26→

    27→
    28→### Supporters ❤
    29→
    30→|

Warp, the agentic terminal for developers
Available for MacOS, Linux, & Windows |   |
    31→| ---- | ---- |
    32→
    33→----
    34→
    35→### Easily fine-tune 100+ large language models with zero-code [CLI](#quickstart) and [Web UI](#fine-tuning-with-llama-board-gui-powered-by-gradio)
    36→
    37→![GitHub Trend](https://trendshift.io/api/badge/repositories/4535)
    38→
    39→

    40→
    41→👋 Join our [WeChat](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/main.jpg), [NPU](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/npu.jpg), [Lab4AI](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/lab4ai.jpg), [LLaMA Factory Online](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/online.jpg) user group.
    42→
    43→\[ English | [中文](README_zh.md) \]
    44→
    45→**Fine-tuning a large language model can be easy as...**
    46→
    47→https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e
    48→
    49→Start local training:
    50→- Please refer to [usage](#getting-started)
    51→
    52→Start cloud training:
    53→- **Colab (free)**: https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing
    54→- **PAI-DSW (free trial)**: https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory
    55→- **LLaMA Factory Online**: https://www.llamafactory.com.cn/?utm_source=LLaMA-Factory
    56→- **Alaya NeW (cloud GPU deal)**: https://docs.alayanew.com/docs/documents/useGuide/LLaMAFactory/mutiple/?utm_source=LLaMA-Factory
    57→
    58→Read technical notes:
    59→- **Documentation (WIP)**: https://llamafactory.readthedocs.io/en/latest/
    60→- **Documentation (AMD GPU)**: https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html
    61→- **Official Blog**: https://blog.llamafactory.net/en/
    62→- **Official Course**: https://www.lab4ai.cn/course/detail?id=7c13e60f6137474eb40f6fd3983c0f46&utm_source=LLaMA-Factory
    63→
    64→> [!NOTE]
    65→> Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.
    66→
    67→## Table of Contents
    68→
    69→- [Features](#features)
    70→- [Blogs](#blogs)
    71→- [Changelog](#changelog)
    72→- [Supported Models](#supported-models)
    73→- [Supported Training Approaches](#supported-training-approaches)
    74→- [Provided Datasets](#provided-datasets)
    75→- [Requirement](#requirement)
    76→- [Getting Started](#getting-started)
    77→  - [Installation](#installation)
    78→  - [Data Preparation](#data-preparation)
    79→  - [Quickstart](#quickstart)
    80→  - [Fine-Tuning with LLaMA Board GUI](#fine-tuning-with-llama-board-gui-powered-by-gradio)
    81→  - [LLaMA Factory Online](#llama-factory-online)
    82→  - [Build Docker](#build-docker)
    83→  - [Deploy with OpenAI-style API and vLLM](#deploy-with-openai-style-api-and-vllm)
    84→  - [Download from ModelScope Hub](#download-from-modelscope-hub)
    85→  - [Download from Modelers Hub](#download-from-modelers-hub)
    86→  - [Use W&B Logger](#use-wb-logger)
    87→  - [Use SwanLab Logger](#use-swanlab-logger)
    88→- [Projects using LLaMA Factory](#projects-using-llama-factory)
    89→- [License](#license)
    90→- [Citation](#citation)
    91→- [Acknowledgement](#acknowledgement)
    92→
    93→## Features
    94→
    95→- **Various models**: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc.
    96→- **Integrated methods**: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
    97→- **Scalable resources**: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
    98→- **Advanced algorithms**: [GaLore](https://github.com/jiaweizzhao/GaLore), [BAdam](https://github.com/Ledzy/BAdam), [APOLLO](https://github.com/zhuhanqing/APOLLO), [Adam-mini](https://github.com/zyushun/Adam-mini), [Muon](https://github.com/KellerJordan/Muon), [OFT](https://github.com/huggingface/peft/tree/main/src/peft/tuners/oft), DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.
    99→- **Practical tricks**: [FlashAttention-2](https://github.com/Dao-AILab/flash-attention), [Unsloth](https://github.com/unslothai/unsloth), [Liger Kernel](https://github.com/linkedin/Liger-Kernel), [KTransformers](https://github.com/kvcache-ai/ktransformers/), RoPE scaling, NEFTune and rsLoRA.
   100→- **Wide tasks**: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.


Let me check how the model loads and processes files:


total 172
drwxrwxr-x 2 ubuntu ubuntu   4096 Mar 25 02:16 .
drwxrwxr-x 3 ubuntu ubuntu   4096 Mar 25 02:16 ..
-rw-rw-r-- 1 ubuntu ubuntu 155130 Mar 25 02:16 bench_manifest.parquet
-rw-rw-r-- 1 ubuntu ubuntu   4118 Mar 25 02:16 bench_results.json
-rw-rw-r-- 1 ubuntu ubuntu   2107 Mar 25 02:16 report.md


     1→# Phase 1: Raw Audio Pipeline Benchmark Report
     2→
     3→**Date:** 2026-03-23 00:11:33
     4→
     5→## Environment Matrix
     6→
     7→| Component | Version |
     8→|-----------|---------|
     9→| Python | 3.12.3 |
    10→| PyTorch | 2.10.0+cu128 |
    11→| torchaudio | 2.10.0+cu128 |
    12→| transformers | 5.2.0 |
    13→| soundfile | 0.13.1 |
    14→| LlamaFactory commit | `ffbff33af347` |
    15→| CPUs | 240 |
    16→| RAM | 3023.3 GB |
    17→| GPUs | NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB |
    18→
    19→## Throughput vs Workers
    20→
    21→| Workers | Wall Time (s) | Samples/sec | Audio-sec/sec | p50 Latency (ms) | p95 Latency (ms) | CPU % | Failures |
    22→|---------|--------------|-------------|---------------|-------------------|-------------------|-------|----------|
    23→| 4 | 759.18 | 2.62 | 20.6 | 2116.98 | 2267.2 | 8.5 | 2 |
    24→| 8 | 85.86 | 23.14 | 182.16 | 337.86 | 490.27 | 76.9 | 2 |
    25→| 12 | 76.35 | 26.02 | 204.85 | 453.07 | 664.27 | 89.0 | 2 |
    26→| 16 | 73.98 | 26.86 | 211.43 | 579.32 | 908.85 | 93.7 | 2 |
    27→| 24 | 73.87 | 26.9 | 211.73 | 856.38 | 1384.59 | 95.2 | 2 |
    28→| 32 | 74.26 | 26.76 | 210.63 | 1119.72 | 1979.72 | 94.7 | 2 |
    29→
    30→## Resample Analysis
    31→
    32→- Samples needing resample (SR != 16kHz): **1585** / 1987 (79.8%)
    33→- Mean decode time: **3.7 ms**
    34→- Mean resample time (when needed): **28.6 ms**
    35→
    36→## Go/No-Go: Runtime Decode + Resample
    37→
    38→**DECISION: NO-GO for runtime resample.**
    39→
    40→- 79.8% of samples require resampling from 48kHz → 16kHz.
    41→- Best throughput with 24 workers: 211.73 audio-sec/sec.
    42→- Runtime resampling adds per-sample latency and wastes CPU in the training loop.
    43→- **Recommendation:** Pre-convert all audio to 16kHz mono FLAC in Phase 2.
    44→
    45→## Recommended Target Ingest Format (Phase 2)
    46→
    47→1. **Audio format:** 16 kHz, mono, FLAC (lossless, compact)
    48→2. **Storage:** Re-tar with consistent naming, one tar per shard
    49→3. **Index:** Unified manifest parquet with byte offsets for O(1) random access
    50→4. **Pre-compute:** All resampling done offline; training path is decode-only
    51→5. **Validation:** SHA256 integrity checks on every converted file
    52→


Now let me look at the hf_engine for inference:


     1→# Copyright 2025 the LlamaFactory team.
     2→#
     3→# Licensed under the Apache License, Version 2.0 (the "License");
     4→# you may not use this file except in compliance with the License.
     5→# You may obtain a copy of the License at
     6→#
     7→#     http://www.apache.org/licenses/LICENSE-2.0
     8→#
     9→# Unless required by applicable law or agreed to in writing, software
    10→# distributed under the License is distributed on an "AS IS" BASIS,
    11→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12→# See the License for the specific language governing permissions and
    13→# limitations under the License.
    14→
    15→import asyncio
    16→import os
    17→from collections.abc import AsyncGenerator, Callable
    18→from threading import Thread
    19→from typing import TYPE_CHECKING, Any, Optional, Union
    20→
    21→import torch
    22→from transformers import GenerationConfig, TextIteratorStreamer
    23→from typing_extensions import override
    24→
    25→from ..data import get_template_and_fix_tokenizer
    26→from ..extras import logging
    27→from ..extras.constants import AUDIO_PLACEHOLDER, IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, EngineName
    28→from ..model import load_model, load_tokenizer
    29→from .base_engine import BaseEngine, Response
    30→
    31→
    32→if TYPE_CHECKING:
    33→    from transformers import PreTrainedModel, PreTrainedTokenizer, ProcessorMixin
    34→    from trl import PreTrainedModelWrapper
    35→
    36→    from ..data import Template
    37→    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    38→    from ..hparams import DataArguments, FinetuningArguments, GeneratingArguments, ModelArguments
    39→
    40→
    41→logger = logging.get_logger(__name__)
    42→
    43→
    44→class HuggingfaceEngine(BaseEngine):
    45→    def __init__(
    46→        self,
    47→        model_args: "ModelArguments",
    48→        data_args: "DataArguments",
    49→        finetuning_args: "FinetuningArguments",
    50→        generating_args: "GeneratingArguments",
    51→    ) -> None:
    52→        self.name = EngineName.HF
    53→        self.can_generate = finetuning_args.stage == "sft"
    54→        tokenizer_module = load_tokenizer(model_args)
    55→        self.tokenizer = tokenizer_module["tokenizer"]
    56→        self.processor = tokenizer_module["processor"]
    57→        self.tokenizer.padding_side = "left" if self.can_generate else "right"
    58→        self.template = get_template_and_fix_tokenizer(self.tokenizer, data_args)
    59→        self.model = load_model(
    60→            self.tokenizer, model_args, finetuning_args, is_trainable=False, add_valuehead=(not self.can_generate)
    61→        )  # must after fixing tokenizer to resize vocab
    62→        self.generating_args = generating_args.to_dict()
    63→        try:
    64→            asyncio.get_event_loop()
    65→        except RuntimeError:
    66→            logger.warning_rank0_once("There is no current event loop, creating a new one.")
    67→            loop = asyncio.new_event_loop()
    68→            asyncio.set_event_loop(loop)
    69→
    70→        self.semaphore = asyncio.Semaphore(int(os.getenv("MAX_CONCURRENT", "1")))
    71→
    72→    @staticmethod
    73→    def _process_args(
    74→        model: "PreTrainedModel",
    75→        tokenizer: "PreTrainedTokenizer",
    76→        processor: Optional["ProcessorMixin"],
    77→        template: "Template",
    78→        generating_args: dict[str, Any],
    79→        messages: list[dict[str, str]],
    80→        system: Optional[str] = None,
    81→        tools: Optional[str] = None,
    82→        images: Optional[list["ImageInput"]] = None,
    83→        videos: Optional[list["VideoInput"]] = None,
    84→        audios: Optional[list["AudioInput"]] = None,
    85→        input_kwargs: Optional[dict[str, Any]] = {},
    86→    ) -> tuple[dict[str, Any], int]:
    87→        mm_input_dict = {"images": [], "videos": [], "audios": [], "imglens": [0], "vidlens": [0], "audlens": [0]}
    88→        if images is not None:
    89→            mm_input_dict.update({"images": images, "imglens": [len(images)]})
    90→            if not any(IMAGE_PLACEHOLDER in message["content"] for message in messages):
    91→                messages[0]["content"] = IMAGE_PLACEHOLDER * len(images) + messages[0]["content"]
    92→
    93→        if videos is not None:
    94→            mm_input_dict.update({"videos": videos, "vidlens": [len(videos)]})
    95→            if not any(VIDEO_PLACEHOLDER in message["content"] for message in messages):
    96→                messages[0]["content"] = VIDEO_PLACEHOLDER * len(videos) + messages[0]["content"]
    97→
    98→        if audios is not None:
    99→            mm_input_dict.update({"audios": audios, "audlens": [len(audios)]})
   100→            if not any(AUDIO_PLACEHOLDER in message["content"] for message in messages):
   101→                messages[0]["content"] = AUDIO_PLACEHOLDER * len(audios) + messages[0]["content"]
   102→
   103→        messages = template.mm_plugin.process_messages(
   104→            messages, mm_input_dict["images"], mm_input_dict["videos"], mm_input_dict["audios"], processor
   105→        )
   106→        paired_messages = messages + [{"role": "assistant", "content": ""}]
   107→        prompt_ids, _ = template.encode_oneturn(tokenizer, paired_messages, system, tools)
   108→        prompt_ids, _ = template.mm_plugin.process_token_ids(
   109→            prompt_ids,
   110→            None,
   111→            mm_input_dict["images"],
   112→            mm_input_dict["videos"],
   113→            mm_input_dict["audios"],
   114→            tokenizer,
   115→            processor,
   116→        )
   117→        prompt_length = len(prompt_ids)
   118→        inputs = torch.tensor([prompt_ids], device=model.device)
   119→        attention_mask = torch.ones_like(inputs, dtype=torch.long)
   120→
   121→        do_sample: Optional[bool] = input_kwargs.pop("do_sample", None)
   122→        temperature: Optional[float] = input_kwargs.pop("temperature", None)
   123→        top_p: Optional[float] = input_kwargs.pop("top_p", None)
   124→        top_k: Optional[float] = input_kwargs.pop("top_k", None)
   125→        num_return_sequences: int = input_kwargs.pop("num_return_sequences", 1)
   126→        repetition_penalty: Optional[float] = input_kwargs.pop("repetition_penalty", None)
   127→        length_penalty: Optional[float] = input_kwargs.pop("length_penalty", None)
   128→        skip_special_tokens: Optional[bool] = input_kwargs.pop("skip_special_tokens", None)
   129→        max_length: Optional[int] = input_kwargs.pop("max_length", None)
   130→        max_new_tokens: Optional[int] = input_kwargs.pop("max_new_tokens", None)
   131→        stop: Optional[Union[str, list[str]]] = input_kwargs.pop("stop", None)
   132→
   133→        if stop is not None:
   134→            logger.warning_rank0("Stop parameter is not supported by the huggingface engine yet.")
   135→
   136→        generating_args = generating_args.copy()
   137→        generating_args.update(
   138→            dict(
   139→                do_sample=do_sample if do_sample is not None else generating_args["do_sample"],
   140→                temperature=temperature if temperature is not None else generating_args["temperature"],
   141→                top_p=top_p if top_p is not None else generating_args["top_p"],
   142→                top_k=top_k if top_k is not None else generating_args["top_k"],
   143→                num_return_sequences=num_return_sequences,
   144→                repetition_penalty=repetition_penalty
   145→                if repetition_penalty is not None
   146→                else generating_args["repetition_penalty"],
   147→                length_penalty=length_penalty if length_penalty is not None else generating_args["length_penalty"],
   148→                skip_special_tokens=skip_special_tokens
   149→                if skip_special_tokens is not None
   150→                else generating_args["skip_special_tokens"],
   151→                eos_token_id=template.get_stop_token_ids(tokenizer),
   152→                pad_token_id=tokenizer.pad_token_id,
   153→            )
   154→        )
   155→
   156→        if isinstance(num_return_sequences, int) and num_return_sequences > 1:  # do_sample needs temperature > 0
   157→            generating_args["do_sample"] = True
   158→            generating_args["temperature"] = generating_args["temperature"] or 1.0
   159→
   160→        if not generating_args["temperature"]:
   161→            generating_args["do_sample"] = False
   162→
   163→        if not generating_args["do_sample"]:
   164→            generating_args.pop("temperature", None)
   165→            generating_args.pop("top_p", None)
   166→
   167→        if max_length:
   168→            generating_args.pop("max_new_tokens", None)
   169→            generating_args["max_length"] = max_length
   170→
   171→        if max_new_tokens:
   172→            generating_args.pop("max_length", None)
   173→            generating_args["max_new_tokens"] = max_new_tokens
   174→
   175→        gen_kwargs = dict(
   176→            inputs=inputs,
   177→            attention_mask=attention_mask,
   178→            generation_config=GenerationConfig(**generating_args),
   179→        )
   180→
   181→        mm_inputs = template.mm_plugin.get_mm_inputs(**mm_input_dict, batch_ids=[prompt_ids], processor=processor)
   182→        for key, value in mm_inputs.items():
   183→            if isinstance(value, list) and isinstance(value[0], torch.Tensor):  # for pixtral inputs
   184→                value = torch.stack(value)  # assume they have same sizes
   185→            elif (
   186→                isinstance(value, list) and isinstance(value[0], list) and isinstance(value[0][0], torch.Tensor)
   187→            ):  # for minicpmv inputs
   188→                value = torch.stack([torch.stack(v) for v in value])
   189→            elif not isinstance(value, torch.Tensor):
   190→                value = torch.tensor(value)
   191→
   192→            if torch.is_floating_point(value):  # cast data dtype for paligemma
   193→                value = value.to(model.dtype)
   194→
   195→            if key == "second_per_grid_ts":  # qwen2.5vl special case
   196→                gen_kwargs[key] = value.tolist()
   197→            else:
   198→                gen_kwargs[key] = value.to(model.device)
   199→
   200→        if getattr(model.config, "model_type", None) in ["minicpmv", "minicpmo"]:


Let me look at one of the actual tools to understand the structure better: