find /home/ubuntu/training/gemma3n-E2B-asr-ft -name "requirements*.txt" | grep -v vendor


(Bash completed with no output)


1→![# LLaMA Factory](assets/logo.png)
     2→
     3→[![GitHub Repo stars](https://img.shields.io/github/stars/hiyouga/LLaMA-Factory?style=social)](https://github.com/hiyouga/LLaMA-Factory/stargazers)
     4→[![GitHub last commit](https://img.shields.io/github/last-commit/hiyouga/LLaMA-Factory)](https://github.com/hiyouga/LLaMA-Factory/commits/main)
     5→[![GitHub contributors](https://img.shields.io/github/contributors/hiyouga/LLaMA-Factory?color=orange)](https://github.com/hiyouga/LLaMA-Factory/graphs/contributors)
     6→[![GitHub workflow](https://github.com/hiyouga/LLaMA-Factory/actions/workflows/tests.yml/badge.svg)](https://github.com/hiyouga/LLaMA-Factory/actions/workflows/tests.yml)
     7→[![PyPI](https://img.shields.io/pypi/v/llamafactory)](https://pypi.org/project/llamafactory/)
     8→[![Citation](https://img.shields.io/badge/citation-1000+-green)](https://scholar.google.com/scholar?cites=12620864006390196564)
     9→[![Docker Pulls](https://img.shields.io/docker/pulls/hiyouga/llamafactory)](https://hub.docker.com/r/hiyouga/llamafactory/tags)
    10→
    11→[![Twitter](https://img.shields.io/twitter/follow/llamafactory_ai)](https://twitter.com/llamafactory_ai)
    12→[![Discord](assets/thirdparty/discord.svg)](https://discord.gg/rKfvV9r9FK)
    13→[![WeChat](https://img.shields.io/badge/WeChat-User%20Group-blue?logo=wechat)](https://github.com/hiyouga/llamafactory-community)
    14→[![Blog](https://img.shields.io/badge/Hugo-Official%20Blog-blue?logo=hugo)](https://blog.llamafactory.net/en/)
    15→
    16→[![Open in Colab](assets/thirdparty/colab.svg)](https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing)
    17→[![Open in DSW](assets/thirdparty/dsw.svg)](https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory)
    18→[![Open in Lab4ai](assets/thirdparty/lab4ai.svg)](https://www.lab4ai.cn/course/detail?id=7c13e60f6137474eb40f6fd3983c0f46&utm_source=LLaMA-Factory)
    19→[![Open in Online](assets/thirdparty/online.svg)](https://www.llamafactory.com.cn/?utm_source=LLaMA-Factory)
    20→[![Open in Spaces](https://img.shields.io/badge/🤗-Open%20in%20Spaces-blue)](https://huggingface.co/spaces/hiyouga/LLaMA-Board)
    21→[![Open in Studios](https://img.shields.io/badge/ModelScope-Open%20in%20Studios-blue)](https://modelscope.cn/studios/hiyouga/LLaMA-Board)
    22→[![Open in Novita](https://img.shields.io/badge/Novita-Deploy%20Template-blue)](https://novita.ai/templates-library/105981?sharer=88115474-394e-4bda-968e-b88e123d0c47)
    23→
    24→### Used by [Amazon](https://aws.amazon.com/cn/blogs/machine-learning/how-apoidea-group-enhances-visual-information-extraction-from-banking-documents-with-multimodal-models-using-llama-factory-on-amazon-sagemaker-hyperpod/), [NVIDIA](https://developer.nvidia.com/rtx/ai-toolkit), [Aliyun](https://help.aliyun.com/zh/pai/use-cases/fine-tune-a-llama-3-model-with-llama-factory), etc.
    25→
    26→

    27→
    28→### Supporters ❤
    29→
    30→|

Warp, the agentic terminal for developers
Available for MacOS, Linux, & Windows |   |
    31→| ---- | ---- |
    32→
    33→----
    34→
    35→### Easily fine-tune 100+ large language models with zero-code [CLI](#quickstart) and [Web UI](#fine-tuning-with-llama-board-gui-powered-by-gradio)
    36→
    37→![GitHub Trend](https://trendshift.io/api/badge/repositories/4535)
    38→
    39→

    40→
    41→👋 Join our [WeChat](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/main.jpg), [NPU](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/npu.jpg), [Lab4AI](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/lab4ai.jpg), [LLaMA Factory Online](https://github.com/hiyouga/llamafactory-community/blob/main/wechat/online.jpg) user group.
    42→
    43→\[ English | [中文](README_zh.md) \]
    44→
    45→**Fine-tuning a large language model can be easy as...**
    46→
    47→https://github.com/user-attachments/assets/3991a3a8-4276-4d30-9cab-4cb0c4b9b99e
    48→
    49→Start local training:
    50→- Please refer to [usage](#getting-started)
    51→
    52→Start cloud training:
    53→- **Colab (free)**: https://colab.research.google.com/drive/1eRTPn37ltBbYsISy9Aw2NuI2Aq5CQrD9?usp=sharing
    54→- **PAI-DSW (free trial)**: https://gallery.pai-ml.com/#/preview/deepLearning/nlp/llama_factory
    55→- **LLaMA Factory Online**: https://www.llamafactory.com.cn/?utm_source=LLaMA-Factory
    56→- **Alaya NeW (cloud GPU deal)**: https://docs.alayanew.com/docs/documents/useGuide/LLaMAFactory/mutiple/?utm_source=LLaMA-Factory
    57→
    58→Read technical notes:
    59→- **Documentation (WIP)**: https://llamafactory.readthedocs.io/en/latest/
    60→- **Documentation (AMD GPU)**: https://rocm.docs.amd.com/projects/ai-developer-hub/en/latest/notebooks/fine_tune/llama_factory_llama3.html
    61→- **Official Blog**: https://blog.llamafactory.net/en/
    62→- **Official Course**: https://www.lab4ai.cn/course/detail?id=7c13e60f6137474eb40f6fd3983c0f46&utm_source=LLaMA-Factory
    63→
    64→> [!NOTE]
    65→> Except for the above links, all other websites are unauthorized third-party websites. Please carefully use them.
    66→
    67→## Table of Contents
    68→
    69→- [Features](#features)
    70→- [Blogs](#blogs)
    71→- [Changelog](#changelog)
    72→- [Supported Models](#supported-models)
    73→- [Supported Training Approaches](#supported-training-approaches)
    74→- [Provided Datasets](#provided-datasets)
    75→- [Requirement](#requirement)
    76→- [Getting Started](#getting-started)
    77→  - [Installation](#installation)
    78→  - [Data Preparation](#data-preparation)
    79→  - [Quickstart](#quickstart)
    80→  - [Fine-Tuning with LLaMA Board GUI](#fine-tuning-with-llama-board-gui-powered-by-gradio)
    81→  - [LLaMA Factory Online](#llama-factory-online)
    82→  - [Build Docker](#build-docker)
    83→  - [Deploy with OpenAI-style API and vLLM](#deploy-with-openai-style-api-and-vllm)
    84→  - [Download from ModelScope Hub](#download-from-modelscope-hub)
    85→  - [Download from Modelers Hub](#download-from-modelers-hub)
    86→  - [Use W&B Logger](#use-wb-logger)
    87→  - [Use SwanLab Logger](#use-swanlab-logger)
    88→- [Projects using LLaMA Factory](#projects-using-llama-factory)
    89→- [License](#license)
    90→- [Citation](#citation)
    91→- [Acknowledgement](#acknowledgement)
    92→
    93→## Features
    94→
    95→- **Various models**: LLaMA, LLaVA, Mistral, Mixtral-MoE, Qwen3, Qwen3-VL, DeepSeek, Gemma, GLM, Phi, etc.
    96→- **Integrated methods**: (Continuous) pre-training, (multimodal) supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, etc.
    97→- **Scalable resources**: 16-bit full-tuning, freeze-tuning, LoRA and 2/3/4/5/6/8-bit QLoRA via AQLM/AWQ/GPTQ/LLM.int8/HQQ/EETQ.
    98→- **Advanced algorithms**: [GaLore](https://github.com/jiaweizzhao/GaLore), [BAdam](https://github.com/Ledzy/BAdam), [APOLLO](https://github.com/zhuhanqing/APOLLO), [Adam-mini](https://github.com/zyushun/Adam-mini), [Muon](https://github.com/KellerJordan/Muon), [OFT](https://github.com/huggingface/peft/tree/main/src/peft/tuners/oft), DoRA, LongLoRA, LLaMA Pro, Mixture-of-Depths, LoRA+, LoftQ and PiSSA.
    99→- **Practical tricks**: [FlashAttention-2](https://github.com/Dao-AILab/flash-attention), [Unsloth](https://github.com/unslothai/unsloth), [Liger Kernel](https://github.com/linkedin/Liger-Kernel), [KTransformers](https://github.com/kvcache-ai/ktransformers/), RoPE scaling, NEFTune and rsLoRA.
   100→- **Wide tasks**: Multi-turn dialogue, tool using, image understanding, visual grounding, video recognition, audio understanding, etc.


total 172
drwxrwxr-x 2 ubuntu ubuntu   4096 Mar 25 02:16 .
drwxrwxr-x 3 ubuntu ubuntu   4096 Mar 25 02:16 ..
-rw-rw-r-- 1 ubuntu ubuntu 155130 Mar 25 02:16 bench_manifest.parquet
-rw-rw-r-- 1 ubuntu ubuntu   4118 Mar 25 02:16 bench_results.json
-rw-rw-r-- 1 ubuntu ubuntu   2107 Mar 25 02:16 report.md


     1→# Phase 1: Raw Audio Pipeline Benchmark Report
     2→
     3→**Date:** 2026-03-23 00:11:33
     4→
     5→## Environment Matrix
     6→
     7→| Component | Version |
     8→|-----------|---------|
     9→| Python | 3.12.3 |
    10→| PyTorch | 2.10.0+cu128 |
    11→| torchaudio | 2.10.0+cu128 |
    12→| transformers | 5.2.0 |
    13→| soundfile | 0.13.1 |
    14→| LlamaFactory commit | `ffbff33af347` |
    15→| CPUs | 240 |
    16→| RAM | 3023.3 GB |
    17→| GPUs | NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB, NVIDIA B200, 183359 MiB |
    18→
    19→## Throughput vs Workers
    20→
    21→| Workers | Wall Time (s) | Samples/sec | Audio-sec/sec | p50 Latency (ms) | p95 Latency (ms) | CPU % | Failures |
    22→|---------|--------------|-------------|---------------|-------------------|-------------------|-------|----------|
    23→| 4 | 759.18 | 2.62 | 20.6 | 2116.98 | 2267.2 | 8.5 | 2 |
    24→| 8 | 85.86 | 23.14 | 182.16 | 337.86 | 490.27 | 76.9 | 2 |
    25→| 12 | 76.35 | 26.02 | 204.85 | 453.07 | 664.27 | 89.0 | 2 |
    26→| 16 | 73.98 | 26.86 | 211.43 | 579.32 | 908.85 | 93.7 | 2 |
    27→| 24 | 73.87 | 26.9 | 211.73 | 856.38 | 1384.59 | 95.2 | 2 |
    28→| 32 | 74.26 | 26.76 | 210.63 | 1119.72 | 1979.72 | 94.7 | 2 |
    29→
    30→## Resample Analysis
    31→
    32→- Samples needing resample (SR != 16kHz): **1585** / 1987 (79.8%)
    33→- Mean decode time: **3.7 ms**
    34→- Mean resample time (when needed): **28.6 ms**
    35→
    36→## Go/No-Go: Runtime Decode + Resample
    37→
    38→**DECISION: NO-GO for runtime resample.**
    39→
    40→- 79.8% of samples require resampling from 48kHz → 16kHz.
    41→- Best throughput with 24 workers: 211.73 audio-sec/sec.
    42→- Runtime resampling adds per-sample latency and wastes CPU in the training loop.
    43→- **Recommendation:** Pre-convert all audio to 16kHz mono FLAC in Phase 2.
    44→
    45→## Recommended Target Ingest Format (Phase 2)
    46→
    47→1. **Audio format:** 16 kHz, mono, FLAC (lossless, compact)
    48→2. **Storage:** Re-tar with consistent naming, one tar per shard
    49→3. **Index:** Unified manifest parquet with byte offsets for O(1) random access
    50→4. **Pre-compute:** All resampling done offline; training path is decode-only
    51→5. **Validation:** SHA256 integrity checks on every converted file
    52→


     1→# Copyright 2025 the LlamaFactory team.
     2→#
     3→# Licensed under the Apache License, Version 2.0 (the "License");
     4→# you may not use this file except in compliance with the License.
     5→# You may obtain a copy of the License at
     6→#
     7→#     http://www.apache.org/licenses/LICENSE-2.0
     8→#
     9→# Unless required by applicable law or agreed to in writing, software
    10→# distributed under the License is distributed on an "AS IS" BASIS,
    11→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12→# See the License for the specific language governing permissions and
    13→# limitations under the License.
    14→
    15→import asyncio
    16→import os
    17→from collections.abc import AsyncGenerator, Callable
    18→from threading import Thread
    19→from typing import TYPE_CHECKING, Any, Optional, Union
    20→
    21→import torch
    22→from transformers import GenerationConfig, TextIteratorStreamer
    23→from typing_extensions import override
    24→
    25→from ..data import get_template_and_fix_tokenizer
    26→from ..extras import logging
    27→from ..extras.constants import AUDIO_PLACEHOLDER, IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, EngineName
    28→from ..model import load_model, load_tokenizer
    29→from .base_engine import BaseEngine, Response
    30→
    31→
    32→if TYPE_CHECKING:
    33→    from transformers import PreTrainedModel, PreTrainedTokenizer, ProcessorMixin
    34→    from trl import PreTrainedModelWrapper
    35→
    36→    from ..data import Template
    37→    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    38→    from ..hparams import DataArguments, FinetuningArguments, GeneratingArguments, ModelArguments
    39→
    40→
    41→logger = logging.get_logger(__name__)
    42→
    43→
    44→class HuggingfaceEngine(BaseEngine):
    45→    def __init__(
    46→        self,
    47→        model_args: "ModelArguments",
    48→        data_args: "DataArguments",
    49→        finetuning_args: "FinetuningArguments",
    50→        generating_args: "GeneratingArguments",
    51→    ) -> None:
    52→        self.name = EngineName.HF
    53→        self.can_generate = finetuning_args.stage == "sft"
    54→        tokenizer_module = load_tokenizer(model_args)
    55→        self.tokenizer = tokenizer_module["tokenizer"]
    56→        self.processor = tokenizer_module["processor"]
    57→        self.tokenizer.padding_side = "left" if self.can_generate else "right"
    58→        self.template = get_template_and_fix_tokenizer(self.tokenizer, data_args)
    59→        self.model = load_model(
    60→            self.tokenizer, model_args, finetuning_args, is_trainable=False, add_valuehead=(not self.can_generate)
    61→        )  # must after fixing tokenizer to resize vocab
    62→        self.generating_args = generating_args.to_dict()
    63→        try:
    64→            asyncio.get_event_loop()
    65→        except RuntimeError:
    66→            logger.warning_rank0_once("There is no current event loop, creating a new one.")
    67→            loop = asyncio.new_event_loop()
    68→            asyncio.set_event_loop(loop)
    69→
    70→        self.semaphore = asyncio.Semaphore(int(os.getenv("MAX_CONCURRENT", "1")))
    71→
    72→    @staticmethod
    73→    def _process_args(
    74→        model: "PreTrainedModel",
    75→        tokenizer: "PreTrainedTokenizer",
    76→        processor: Optional["ProcessorMixin"],
    77→        template: "Template",
    78→        generating_args: dict[str, Any],
    79→        messages: list[dict[str, str]],
    80→        system: Optional[str] = None,
    81→        tools: Optional[str] = None,
    82→        images: Optional[list["ImageInput"]] = None,
    83→        videos: Optional[list["VideoInput"]] = None,
    84→        audios: Optional[list["AudioInput"]] = None,
    85→        input_kwargs: Optional[dict[str, Any]] = {},
    86→    ) -> tuple[dict[str, Any], int]:
    87→        mm_input_dict = {"images": [], "videos": [], "audios": [], "imglens": [0], "vidlens": [0], "audlens": [0]}
    88→        if images is not None:
    89→            mm_input_dict.update({"images": images, "imglens": [len(images)]})
    90→            if not any(IMAGE_PLACEHOLDER in message["content"] for message in messages):
    91→                messages[0]["content"] = IMAGE_PLACEHOLDER * len(images) + messages[0]["content"]
    92→
    93→        if videos is not None:
    94→            mm_input_dict.update({"videos": videos, "vidlens": [len(videos)]})
    95→            if not any(VIDEO_PLACEHOLDER in message["content"] for message in messages):
    96→                messages[0]["content"] = VIDEO_PLACEHOLDER * len(videos) + messages[0]["content"]
    97→
    98→        if audios is not None:
    99→            mm_input_dict.update({"audios": audios, "audlens": [len(audios)]})
   100→            if not any(AUDIO_PLACEHOLDER in message["content"] for message in messages):
   101→                messages[0]["content"] = AUDIO_PLACEHOLDER * len(audios) + messages[0]["content"]
   102→
   103→        messages = template.mm_plugin.process_messages(
   104→            messages, mm_input_dict["images"], mm_input_dict["videos"], mm_input_dict["audios"], processor
   105→        )
   106→        paired_messages = messages + [{"role": "assistant", "content": ""}]
   107→        prompt_ids, _ = template.encode_oneturn(tokenizer, paired_messages, system, tools)
   108→        prompt_ids, _ = template.mm_plugin.process_token_ids(
   109→            prompt_ids,
   110→            None,
   111→            mm_input_dict["images"],
   112→            mm_input_dict["videos"],
   113→            mm_input_dict["audios"],
   114→            tokenizer,
   115→            processor,
   116→        )
   117→        prompt_length = len(prompt_ids)
   118→        inputs = torch.tensor([prompt_ids], device=model.device)
   119→        attention_mask = torch.ones_like(inputs, dtype=torch.long)
   120→
   121→        do_sample: Optional[bool] = input_kwargs.pop("do_sample", None)
   122→        temperature: Optional[float] = input_kwargs.pop("temperature", None)
   123→        top_p: Optional[float] = input_kwargs.pop("top_p", None)
   124→        top_k: Optional[float] = input_kwargs.pop("top_k", None)
   125→        num_return_sequences: int = input_kwargs.pop("num_return_sequences", 1)
   126→        repetition_penalty: Optional[float] = input_kwargs.pop("repetition_penalty", None)
   127→        length_penalty: Optional[float] = input_kwargs.pop("length_penalty", None)
   128→        skip_special_tokens: Optional[bool] = input_kwargs.pop("skip_special_tokens", None)
   129→        max_length: Optional[int] = input_kwargs.pop("max_length", None)
   130→        max_new_tokens: Optional[int] = input_kwargs.pop("max_new_tokens", None)
   131→        stop: Optional[Union[str, list[str]]] = input_kwargs.pop("stop", None)
   132→
   133→        if stop is not None:
   134→            logger.warning_rank0("Stop parameter is not supported by the huggingface engine yet.")
   135→
   136→        generating_args = generating_args.copy()
   137→        generating_args.update(
   138→            dict(
   139→                do_sample=do_sample if do_sample is not None else generating_args["do_sample"],
   140→                temperature=temperature if temperature is not None else generating_args["temperature"],
   141→                top_p=top_p if top_p is not None else generating_args["top_p"],
   142→                top_k=top_k if top_k is not None else generating_args["top_k"],
   143→                num_return_sequences=num_return_sequences,
   144→                repetition_penalty=repetition_penalty
   145→                if repetition_penalty is not None
   146→                else generating_args["repetition_penalty"],
   147→                length_penalty=length_penalty if length_penalty is not None else generating_args["length_penalty"],
   148→                skip_special_tokens=skip_special_tokens
   149→                if skip_special_tokens is not None
   150→                else generating_args["skip_special_tokens"],
   151→                eos_token_id=template.get_stop_token_ids(tokenizer),
   152→                pad_token_id=tokenizer.pad_token_id,
   153→            )
   154→        )
   155→
   156→        if isinstance(num_return_sequences, int) and num_return_sequences > 1:  # do_sample needs temperature > 0
   157→            generating_args["do_sample"] = True
   158→            generating_args["temperature"] = generating_args["temperature"] or 1.0
   159→
   160→        if not generating_args["temperature"]:
   161→            generating_args["do_sample"] = False
   162→
   163→        if not generating_args["do_sample"]:
   164→            generating_args.pop("temperature", None)
   165→            generating_args.pop("top_p", None)
   166→
   167→        if max_length:
   168→            generating_args.pop("max_new_tokens", None)
   169→            generating_args["max_length"] = max_length
   170→
   171→        if max_new_tokens:
   172→            generating_args.pop("max_length", None)
   173→            generating_args["max_new_tokens"] = max_new_tokens
   174→
   175→        gen_kwargs = dict(
   176→            inputs=inputs,
   177→            attention_mask=attention_mask,
   178→            generation_config=GenerationConfig(**generating_args),
   179→        )
   180→
   181→        mm_inputs = template.mm_plugin.get_mm_inputs(**mm_input_dict, batch_ids=[prompt_ids], processor=processor)
   182→        for key, value in mm_inputs.items():
   183→            if isinstance(value, list) and isinstance(value[0], torch.Tensor):  # for pixtral inputs
   184→                value = torch.stack(value)  # assume they have same sizes
   185→            elif (
   186→                isinstance(value, list) and isinstance(value[0], list) and isinstance(value[0][0], torch.Tensor)
   187→            ):  # for minicpmv inputs
   188→                value = torch.stack([torch.stack(v) for v in value])
   189→            elif not isinstance(value, torch.Tensor):
   190→                value = torch.tensor(value)
   191→
   192→            if torch.is_floating_point(value):  # cast data dtype for paligemma
   193→                value = value.to(model.dtype)
   194→
   195→            if key == "second_per_grid_ts":  # qwen2.5vl special case
   196→                gen_kwargs[key] = value.tolist()
   197→            else:
   198→                gen_kwargs[key] = value.to(model.device)
   199→
   200→        if getattr(model.config, "model_type", None) in ["minicpmv", "minicpmo"]:


     1→#!/usr/bin/env python3
     2→"""Phase 6: Preflight checks before smoke training.
     3→
     4→1. 10k random URI decode check — zero failures
     5→2. Zero runtime resample events
     6→3. Verify dataset_info.json and parquet are valid
     7→"""
     8→
     9→import json
    10→import os
    11→import sys
    12→import time
    13→from concurrent.futures import ProcessPoolExecutor, as_completed
    14→from pathlib import Path
    15→
    16→import numpy as np
    17→import pyarrow.parquet as pq
    18→
    19→os.environ["OMP_NUM_THREADS"] = "1"
    20→os.environ["MKL_NUM_THREADS"] = "1"
    21→os.environ["TORCH_NUM_THREADS"] = "1"
    22→
    23→DATASET_DIR = "/root/gemini-asr/lf_asr/artifacts/phase4"
    24→N_DECODE_CHECK = 10000
    25→WORKERS = 32
    26→SEED = 42
    27→
    28→
    29→def decode_uri(args):
    30→    """Decode a single URI and verify SR=16000."""
    31→    uri, expected_sr = args
    32→    try:
    33→        sys.path.insert(0, "/root/gemini-asr/lf_asr/vendor/LlamaFactory/src")
    34→        from llamafactory.data.mm_plugin import _load_audio_from_tar_uri
    35→        audio, sr = _load_audio_from_tar_uri(uri)
    36→        if sr != expected_sr:
    37→            return {"success": False, "error": f"SR mismatch: {sr} != {expected_sr}", "uri": uri}
    38→        if audio.shape[-1] == 0:
    39→            return {"success": False, "error": "Empty audio", "uri": uri}
    40→        return {"success": True, "sr": sr, "samples": audio.shape[-1]}
    41→    except Exception as e:
    42→        return {"success": False, "error": str(e), "uri": uri}
    43→
    44→
    45→def decode_uri_with_fallback(uri: str, expected_sr: int = 16000) -> dict:
    46→    """Verify plugin fallback path can recover from decode errors."""
    47→    try:
    48→        sys.path.insert(0, "/root/gemini-asr/lf_asr/vendor/LlamaFactory/src")
    49→        from llamafactory.data.mm_plugin import get_mm_plugin
    50→
    51→        os.environ["LLAMAFACTORY_AUDIO_DECODE_FALLBACK"] = "1"
    52→        os.environ["LLAMAFACTORY_AUDIO_DECODE_FALLBACK_SEC"] = "1.0"
    53→        plugin = get_mm_plugin("gemma3n")
    54→        out = plugin._regularize_audios([uri], sampling_rate=expected_sr)
    55→        audio = out["audios"][0]
    56→        if audio.shape[0] == 0:
    57→            return {"success": False, "error": "fallback returned empty audio", "uri": uri}
    58→        return {"success": True, "samples": int(audio.shape[0]), "uri": uri}
    59→    except Exception as e:
    60→        return {"success": False, "error": str(e), "uri": uri}
    61→
    62→
    63→def main():
    64→    print("Phase 6 Preflight Checks")
    65→    print("=" * 60)
    66→
    67→    # Check 1: dataset_info.json exists
    68→    info_path = os.path.join(DATASET_DIR, "dataset_info.json")
    69→    assert os.path.exists(info_path), f"Missing {info_path}"
    70→    with open(info_path) as f:
    71→        info = json.load(f)
    72→    assert "asr_train" in info
    73→    print("[PASS] dataset_info.json valid")
    74→
    75→    # Check 2: parquet files exist and have expected columns.
    76→    # phase4 may intentionally omit test.parquet (all-data training policy).
    77→    required_splits = ["train", "dev"]
    78→    optional_splits = ["test"]
    79→    split_paths: dict[str, Path] = {}
    80→    expected_cols = {"instruction", "input", "output", "audios"}
    81→
    82→    for split in required_splits + optional_splits:
    83→        path = Path(DATASET_DIR) / f"{split}.parquet"
    84→        if split in required_splits:
    85→            assert path.exists(), f"Missing required {path}"
    86→        elif not path.exists():
    87→            print(f"[INFO] Optional {path.name} not found; continuing.")
    88→            continue
    89→
    90→        pf = pq.ParquetFile(path)
    91→        cols = set(pf.schema_arrow.names)
    92→        missing = expected_cols - cols
    93→        assert not missing, f"{path} missing columns: {sorted(missing)}"
    94→        assert pf.metadata.num_rows > 0, f"{path} is empty"
    95→
    96→        # Verify first row format without loading full file.
    97→        first_rg = pf.read_row_group(0, columns=["instruction", "audios"])
    98→        instruction = first_rg["instruction"][0].as_py()
    99→        audios_val = first_rg["audios"][0].as_py()
   100→        assert isinstance(instruction, str) and instruction.startswith("<audio>")
   101→        assert hasattr(audios_val, "__iter__") and len(audios_val) > 0
   102→        assert str(audios_val[0]).startswith("tar://")
   103→        split_paths[split] = path
   104→        print(f"[PASS] {split}.parquet: {pf.metadata.num_rows:,} rows, columns OK")
   105→
   106→    # Check 3: 10k random URI decode
   107→    print(f"\nDecoding {N_DECODE_CHECK:,} random URIs with {WORKERS} workers...")
   108→    train_pf = pq.ParquetFile(split_paths["train"])
   109→    rng = np.random.default_rng(SEED)
   110→    num_rgs = train_pf.metadata.num_row_groups
   111→    rg_sizes = [train_pf.metadata.row_group(i).num_rows for i in range(num_rgs)]
   112→
   113→    sample_n = min(N_DECODE_CHECK, train_pf.metadata.num_rows)
   114→    picks_by_rg: dict[int, list[int]] = {}
   115→    for _ in range(sample_n):
   116→        rg = int(rng.integers(0, num_rgs))
   117→        row_idx = int(rng.integers(0, rg_sizes[rg]))
   118→        picks_by_rg.setdefault(rg, []).append(row_idx)
   119→
   120→    uris: list[str] = []
   121→    for rg, row_indices in picks_by_rg.items():
   122→        tbl = train_pf.read_row_group(rg, columns=["audios"])
   123→        col = tbl["audios"]
   124→        for row_idx in row_indices:
   125→            audios_val = col[row_idx].as_py()
   126→            uri = audios_val[0] if isinstance(audios_val, list) else audios_val
   127→            uris.append(uri)
   128→
   129→    assert len(uris) == sample_n, f"URI sampling mismatch: {len(uris)} != {sample_n}"
   130→    tasks = [(uri, 16000) for uri in uris]
   131→
   132→    t_start = time.perf_counter()
   133→    results = []
   134→    with ProcessPoolExecutor(max_workers=WORKERS) as executor:
   135→        futures = {executor.submit(decode_uri, t): t for t in tasks}
   136→        for f in as_completed(futures):
   137→            results.append(f.result())
   138→    t_end = time.perf_counter()
   139→
   140→    ok = [r for r in results if r["success"]]
   141→    fail = [r for r in results if not r["success"]]
   142→
   143→    print(f"  Decoded {len(ok):,} OK, {len(fail)} failed in {t_end - t_start:.1f}s")
   144→    if fail:
   145→        print(f"  FAILURES:")
   146→        for f_item in fail[:10]:
   147→            print(f"    {f_item['error']}: {f_item['uri'][:80]}...")
   148→
   149→    # Check for resample (all should be 16000)
   150→    sr_mismatch = [r for r in fail if "SR mismatch" in r.get("error", "")]
   151→    print(f"  SR mismatches (runtime resample events): {len(sr_mismatch)}")
   152→
   153→    fallback_checked = 0
   154→    fallback_failed = 0
   155→    if fail:
   156→        print("  Verifying fallback-path recovery on failed URIs...")
   157→        fallback_results = [decode_uri_with_fallback(r["uri"]) for r in fail]
   158→        fallback_checked = len(fallback_results)
   159→        fallback_failed = sum(1 for r in fallback_results if not r["success"])
   160→        print(f"  Fallback recovery: {fallback_checked - fallback_failed}/{fallback_checked} OK")
   161→
   162→    # Verdict
   163→    print(f"\n{'='*60}")
   164→    all_pass_strict = len(fail) == 0 and len(sr_mismatch) == 0
   165→    all_pass_with_fallback = len(sr_mismatch) == 0 and fallback_failed == 0
   166→    if all_pass_strict:
   167→        print("PREFLIGHT: ALL CHECKS PASSED")
   168→    elif all_pass_with_fallback:
   169→        print("PREFLIGHT: PASS_WITH_FALLBACK (strict decode had recoverable corrupt samples)")
   170→    else:
   171→        print(
   172→            "PREFLIGHT: FAILED "
   173→            f"({len(fail)} strict decode failures, {len(sr_mismatch)} SR mismatches, "
   174→            f"{fallback_failed} fallback failures)"
   175→        )
   176→
   177→    # Save report
   178→    report = {
   179→        "decode_checked": len(results),
   180→        "decode_ok": len(ok),
   181→        "decode_failed": len(fail),
   182→        "sr_mismatches": len(sr_mismatch),
   183→        "fallback_checked": fallback_checked,
   184→        "fallback_failed": fallback_failed,
   185→        "wall_time_s": round(t_end - t_start, 2),
   186→        "all_pass_strict": all_pass_strict,
   187→        "all_pass_with_fallback": all_pass_with_fallback,
   188→    }
   189→    with open(os.path.join(DATASET_DIR, "../phase6_preflight.json"), "w") as f:
   190→        json.dump(report, f, indent=2)
   191→
   192→
   193→if __name__ == "__main__":
   194→    main()
   195→


     1→# Copyright 2025 the LlamaFactory team.
     2→#
     3→# Licensed under the Apache License, Version 2.0 (the "License");
     4→# you may not use this file except in compliance with the License.
     5→# You may obtain a copy of the License at
     6→#
     7→#     http://www.apache.org/licenses/LICENSE-2.0
     8→#
     9→# Unless required by applicable law or agreed to in writing, software
    10→# distributed under the License is distributed on an "AS IS" BASIS,
    11→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12→# See the License for the specific language governing permissions and
    13→# limitations under the License.
    14→
    15→import re
    16→from copy import deepcopy
    17→from dataclasses import dataclass
    18→from typing import TYPE_CHECKING, Optional, Union
    19→
    20→from typing_extensions import override
    21→
    22→from ..extras import logging
    23→from .data_utils import Role
    24→from .formatter import EmptyFormatter, FunctionFormatter, StringFormatter, ToolFormatter
    25→from .mm_plugin import get_mm_plugin
    26→
    27→
    28→if TYPE_CHECKING:
    29→    from transformers import PreTrainedTokenizer
    30→
    31→    from ..hparams import DataArguments
    32→    from .formatter import SLOTS, Formatter
    33→    from .mm_plugin import BasePlugin
    34→    from .tool_utils import FunctionCall
    35→
    36→
    37→logger = logging.get_logger(__name__)
    38→
    39→
    40→@dataclass
    41→class Template:
    42→    format_user: "Formatter"
    43→    format_assistant: "Formatter"
    44→    format_system: "Formatter"
    45→    format_function: "Formatter"
    46→    format_observation: "Formatter"
    47→    format_tools: "Formatter"
    48→    format_prefix: "Formatter"
    49→    default_system: str
    50→    stop_words: list[str]
    51→    thought_words: tuple[str, str]
    52→    tool_call_words: tuple[str, str]
    53→    efficient_eos: bool
    54→    replace_eos: bool
    55→    replace_jinja_template: bool
    56→    enable_thinking: Optional[bool]
    57→    mm_plugin: "BasePlugin"
    58→
    59→    def encode_oneturn(
    60→        self,
    61→        tokenizer: "PreTrainedTokenizer",
    62→        messages: list[dict[str, str]],
    63→        system: Optional[str] = None,
    64→        tools: Optional[str] = None,
    65→    ) -> tuple[list[int], list[int]]:
    66→        r"""Return a single pair of token ids representing prompt and response respectively."""
    67→        encoded_messages = self._encode(tokenizer, messages, system, tools)
    68→        prompt_ids = []
    69→        for encoded_ids in encoded_messages[:-1]:
    70→            prompt_ids += encoded_ids
    71→
    72→        response_ids = encoded_messages[-1]
    73→        return prompt_ids, response_ids
    74→
    75→    def encode_multiturn(
    76→        self,
    77→        tokenizer: "PreTrainedTokenizer",
    78→        messages: list[dict[str, str]],
    79→        system: Optional[str] = None,
    80→        tools: Optional[str] = None,
    81→    ) -> list[tuple[list[int], list[int]]]:
    82→        r"""Return multiple pairs of token ids representing prompts and responses respectively."""
    83→        encoded_messages = self._encode(tokenizer, messages, system, tools)
    84→        return [(encoded_messages[i], encoded_messages[i + 1]) for i in range(0, len(encoded_messages), 2)]
    85→
    86→    def extract_tool(self, content: str) -> Union[str, list["FunctionCall"]]:
    87→        r"""Extract tool message."""
    88→        return self.format_tools.extract(content)
    89→
    90→    def get_stop_token_ids(self, tokenizer: "PreTrainedTokenizer") -> list[int]:
    91→        r"""Return stop token ids."""
    92→        stop_token_ids = {tokenizer.eos_token_id}
    93→        for token in self.stop_words:
    94→            stop_token_ids.add(tokenizer.convert_tokens_to_ids(token))
    95→
    96→        return list(stop_token_ids)
    97→
    98→    def add_thought(self, content: str = "") -> str:
    99→        r"""Add empty thought to assistant message."""
   100→        return f"{self.thought_words[0]}{self.thought_words[1]}" + content


Bash

cd /home/ubuntu/training/gemma3n-E2B-asr-ft && git log --oneline | head -20


d70c21e Gemma3n-E2B ASR full finetune pipeline


Perfect. Now I have comprehensive understanding. Let me create a detailed summary report:

## COMPREHENSIVE GEMMA3N-E2B-ASR-FT REPOSITORY ANALYSIS

### 1. OVERALL STRUCTURE

The repository at `/home/ubuntu/training/gemma3n-E2B-asr-ft/` is an ASR (Automatic Speech Recognition) fine-tuning pipeline built on top of LlamaFactory. Directory structure:

```
gemma3n-E2B-asr-ft/
├── artifacts/              # Pipeline outputs (phase2, phase4, phase6)
├── benchmarks/            # Benchmark manifests and results
│   └── phase1/           # Raw audio pipeline benchmarks
├── configs/              # Training configurations
│   ├── gemma3n_asr_full_ft.yaml      # Full fine-tune (8 batch size)
│   ├── gemma3n_asr_lora_smoke.yaml   # LoRA smoke test (4 batch size)
│   ├── smoke_100step.yaml
│   └── smoke_500step.yaml
├── docs/                 # Documentation and runbooks
│   ├── overnight_plan.md       # Phases 3-6 execution plan
│   ├── phase2_runbook.md       # Audio conversion pipeline
│   └── perf_contract.md        # Performance benchmarks
├── scripts/              # Launch and monitoring scripts
│   ├── launch_smoke.sh
│   ├── launch_full_ft.sh
│   ├── watch_training.sh
│   └── push_ckpt_r2.sh
├── tools/                # Phase-by-phase pipeline tools
├── vendor/LlamaFactory/  # Local copy of LlamaFactory (model training framework)
└── .git/
```

### 2. MODEL ARCHITECTURE

**Base Model:** Gemma-3N (Google Gemma 3 Nano multimodal model)
- **Architecture Type:** Vision-Language Model with Audio Support (multimodal)
- **Framework:** LlamaFactory (fine-tuning framework)
- **Model Location:** `/workspace/gemini-asr/gemma-model`
- **Task:** Audio Transcription (Speech-to-Text)

**Key Multimodal Components:**
- Vision tower: For image/video processing
- Audio encoder: For audio input processing
- Multimodal projector: Fuses audio features with text tokens
- Template: `gemma3n` with audio soft tokens (`<audio_soft_token>`)

**Audio Input Format:**
- Frequency: 16 kHz (pre-converted, no runtime resampling)
- Channels: Mono
- Format: FLAC (lossless compression)
- Input Token: `<audio>` in instruction text

### 3. INFERENCE SUPPORT & MODEL LOADING

**Inference Engine:** HuggingFace Transformers (with LlamaFactory wrapper)

**Key Classes:**
1. **ChatModel** (`/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/chat/chat_model.py`)
   - Main inference interface
   - Supports sync: `chat()`, `stream_chat()`, `get_scores()`
   - Supports async: `achat()`, `astream_chat()`, `aget_scores()`
   - Accepts audio via `audios` parameter

2. **HuggingfaceEngine** (hf_engine.py)
   - Loads tokenizer: `load_tokenizer(model_args)`
   - Loads model: `load_model(tokenizer, model_args, ...)`
   - Processes multimodal inputs (images, videos, audios)
   - Audio is passed as list of audio paths or tensor data

3. **Audio Decoding Plugin** (`mm_plugin.py`)
   - **tar:// URI Support:** `_load_audio_from_tar_uri(uri)` → (torch.Tensor, sample_rate)
   - **URI Format:** `tar:///path/to/audio.tar|offset|nbytes`
   - **Implementation:** Uses `os.pread()` for thread-safe random access with FD caching
   - **Fallback:** Zero-audio tensor (1 second at target SR) on decode errors

**Audio Loading Pipeline:**
```python
# From mm_plugin.py
def _load_audio_from_tar_uri(uri: str) -> tuple[torch.Tensor, int]:
    """Load from tar:// URI → (audio_tensor [C, T], sample_rate)"""
    tar_path, offset, nbytes = _parse_tar_uri(uri)
    fd = _tar_fd_cache.get_fd(tar_path)  # LRU FD cache (max 256)
    raw = os.pread(fd, nbytes, offset)    # Thread-safe pread
    audio_np, sr = sf.read(BytesIO(raw), dtype="float32")
    # Convert (samples,) or (samples, channels) → torch [C, T]
    return torch.Tensor, sr
```

**Inference Messages Format:**
```python
messages = [
    {"role": "user", "content": "<audio>\nTranscribe the audio."}
]
response = chat_model.chat(
    messages=messages,
    audios=["path/to/audio.wav"]  # or tar:// URIs
)
```

### 4. TRAINING DATA PIPELINE

**Preprocessing Phases (6 total):**

1. **Phase 1 - Audio Benchmarking** (Complete)
   - Benchmark: tar read → FLAC decode → resample 16k mono → ndarray
   - Result: 79.8% of samples needed 48k→16k resampling at runtime
   - Decision: Pre-convert all audio offline

2. **Phase 2 - Audio Conversion** (Complete)
   - Input: 78M samples in 48k/16k mixed sample rates
   - Output: 16kHz mono FLAC in tar files
   - Orchestrated conversion: `phase2_orchestrate_conversion.py` (8 workers)
   - Atomic swap strategy for safety
   - Result: 74.9M train, 1.57M dev, 1.57M test rows

3. **Phase 3 - URI Loader Patch** (Complete)
   - tar:// URI support added to LlamaFactory
   - pread + FD cache for efficient random access
   - Validation: 10k random URI decode, 100% success

4. **Phase 4 - LlamaFactory Dataset Build** (Complete)
   - Input: Phase 2 manifest parquets
   - Output: Alpaca-format parquet datasets
   - Format:
     ```
     instruction: "<audio>\nTranscribe the audio."
     input: "Language: <lang>"
     output: "<transcript>"
     audios: ["tar://path|offset|nbytes"]
     duration_bucket: "0-4s" | "4-8s" | "8-16s" | "16-32s" | ">32s"
     ```
   - Block-shuffle for I/O locality (256-row blocks)
   - All buckets included: dispose, golden, redo, finalsftdata, indicvoices

5. **Phase 5 - Config Generation** (Complete)
   - LoRA smoke test: 4 batch size
   - Full fine-tune: 8 batch size
   - 64 preprocessing workers, 8 dataloader workers
   - disable_shuffling: true (uses block-shuffled order)

6. **Phase 6 - Preflight** (Complete)
   - 10k random URI decode validation
   - Zero SR mismatches enforced
   - All checks passed

### 5. BENCHMARK & EVALUATION SCRIPTS

**Location:** `/home/ubuntu/training/gemma3n-E2B-asr-ft/tools/`

**Key Scripts:**

1. **bench_raw_audio_pipeline.py**
   - Tests: tar → FLAC decode → resample → ndarray
   - Worker sweep: 4, 8, 12, 16, 24, 32 workers
   - Metrics: samples/sec, audio-sec/sec, p50/p95 latency, CPU%
   - Output: JSON + Markdown report

2. **phase3_uri_bench.py**
   - Compares tar:// URI loader vs Phase 2 baseline
   - 2000 stratified samples (by language × duration bucket)
   - Gate: URI perf >= 90% of baseline
   - Result: ALL PASS at all worker counts

3. **phase6_preflight.py**
   - Validates dataset_info.json and parquets
   - 10k random URI decode with 32 workers
   - Checks: no SR mismatches, fallback recovery
   - Saves report: `phase6_preflight.json`

**Existing Artifacts:**
- `/home/ubuntu/training/gemma3n-E2B-asr-ft/artifacts/phase2/` - Global manifest, splits
- `/home/ubuntu/training/gemma3n-E2B-asr-ft/artifacts/phase4/` - LF datasets + dataset_info.json
- `/home/ubuntu/training/gemma3n-E2B-asr-ft/benchmarks/phase1/` - Audio pipeline bench

### 6. TRAINING CONFIGURATION

**Two Main Configs:**

**A. LoRA Smoke Test** (`gemma3n_asr_lora_smoke.yaml`)
```yaml
model_name_or_path: /workspace/gemini-asr/gemma-model
trust_remote_code: true
stage: sft
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_target: self_attn.q_proj,self_attn.k_proj,self_attn.v_proj,self_attn.o_proj,mlp.gate_proj,mlp.up_proj,mlp.down_proj
freeze_multi_modal_projector: false

dataset_dir: /root/gemini-asr/lf_asr/artifacts/phase4
dataset: asr_train
eval_dataset: asr_dev
template: gemma3n

per_device_train_batch_size: 4
per_device_eval_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 2.0e-4
num_train_epochs: 1.0
max_steps: 2000
warmup_steps: 100

bf16: true
flash_attn: auto
preprocessing_num_workers: 64
dataloader_num_workers: 8
dataloader_persistent_workers: true
dataloader_prefetch_factor: 4

seed: 42
cutoff_len: 2048
```

**B. Full Fine-Tune** (`gemma3n_asr_full_ft.yaml`)
```yaml
finetuning_type: full
freeze_vision_tower: false
per_device_train_batch_size: 8
gradient_accumulation_steps: 8
learning_rate: 5.0e-5
gradient_checkpointing: true
flash_attn: fa2
```

**Data Loading:**
- `disable_shuffling: true` - Uses pre-ordered block-shuffled data
- `remove_unused_columns: false` - Retains metadata columns
- No runtime resample (LLAMAFACTORY_ALLOW_RESAMPLE=0)
- Fallback enabled: 1 second of silence on decode errors

### 7. DEPENDENCIES & ENVIRONMENT

**Key Dependencies** (from imports and configs):
- **PyTorch:** 2.10.0+cu128
- **torchaudio:** 2.10.0+cu128 (for audio processing)
- **transformers:** 5.2.0 (Hugging Face)
- **pandas:** Data manipulation
- **soundfile:** FLAC decoding
- **numpy:** Numerical operations
- **LlamaFactory:** Vendored in `vendor/LlamaFactory/`

**Environment Setup** (from launch_smoke.sh):
```bash
export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export TORCH_NUM_THREADS=1
export LLAMAFACTORY_ALLOW_RESAMPLE=0
export LLAMAFACTORY_AUDIO_DECODE_FALLBACK=1
export LLAMAFACTORY_AUDIO_DECODE_FALLBACK_SEC=1.0
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
ulimit -n 1048576  # File descriptor limit for tar FD cache
```

**GPU:** 8x NVIDIA B200 (183GB each)
**CPU:** 240 cores
**RAM:** 3TB

### 8. WRITING A BENCHMARK SCRIPT - KEY PATTERNS

To write inference/benchmark scripts for this model:

**Pattern 1: Using ChatModel API**
```python
from llamafactory.chat import ChatModel

chat_model = ChatModel({
    "model_name_or_path": "/workspace/gemini-asr/gemma-model",
    "trust_remote_code": True,
    "template": "gemma3n",
})

messages = [{"role": "user", "content": "<audio>\nTranscribe the audio."}]
responses = chat_model.chat(
    messages=messages,
    audios=["tar://path/to/audio.tar|offset|nbytes"],  # or local paths
    temperature=0.0,
    max_new_tokens=256
)
print(responses[0].response_text)
```

**Pattern 2: Direct URI Decoding**
```python
import sys
sys.path.insert(0, "/root/gemini-asr/lf_asr/vendor/LlamaFactory/src")
from llamafactory.data.mm_plugin import _load_audio_from_tar_uri

uri = "tar:///path/audio.tar|1024|5120"
audio_tensor, sr = _load_audio_from_tar_uri(uri)
# audio_tensor: torch.Tensor [C, T]
# sr: sample rate (16000)
```

**Pattern 3: Batch Processing with Manifest**
```python
import pandas as pd
from concurrent.futures import ProcessPoolExecutor

manifest = pd.read_parquet("artifacts/phase4/train.parquet")
uris = manifest["audios"].apply(lambda x: x[0])  # Extract first audio

def transcribe_batch(uri_batch):
    # Use ChatModel.chat() with batch of audios
    pass

with ProcessPoolExecutor(max_workers=8) as executor:
    results = list(executor.map(transcribe_batch, chunks(uris, 32)))
```

**Key Metrics to Track:**
- Throughput: samples/sec, audio-sec/sec
- Latency: p50, p95 (per-sample decoding + inference)
- GPU utilization: torch.cuda.Event for timing
- SR mismatches: enforce 16kHz only
- Decode failures: count and log with fallback recovery

### 9. FILE PATHS REFERENCE

**Key Code Paths:**
- Model loading: `/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/model/`
- Chat inference: `/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/chat/chat_model.py`
- Audio plugin: `/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/data/mm_plugin.py`
- Templates: `/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/data/template.py`
- Data processing: `/home/ubuntu/training/gemma3n-E2B-asr-ft/vendor/LlamaFactory/src/llamafactory/data/`

**Data Paths:**
- Train data: `/root/gemini-asr/lf_asr/artifacts/phase4/train.parquet` (74.9M rows)
- Dev data: `/root/gemini-asr/lf_asr/artifacts/phase4/dev.parquet` (1.57M rows)
- Dataset info: `/root/gemini-asr/lf_asr/artifacts/phase4/dataset_info.json`
- Audio shards: Various tar files referenced in manifests

**Checkpoint Output:**
- Smoke training: `/root/gemini-asr/lf_asr/outputs/smoke_run/`
- Full training: `/root/gemini-asr/lf_asr/outputs/full_ft_run/`

### 10. PIPELINE DESIGN PRINCIPLES

1. **Zero Runtime Resampling:** All audio pre-converted offline to 16kHz mono FLAC
2. **Efficient Random Access:** tar:// URIs with pread() + FD cache for O(1) access without extraction
3. **I/O Locality:** Block-shuffling (256-row blocks) keeps adjacent samples from same tar file
4. **Reproducibility:** Fixed seed 42, deterministic splits via MD5(video_id)
5. **Data Integrity:** All metadata logged, drop rates tracked, fallback recovery tested
6. **Multiprocessing Safety:** Thread-safe FD cache, per-process state isolation
7. **No Data Filtering:** All quality buckets (dispose, golden, redo, etc.) included

This is a production-grade ASR fine-tuning pipeline with robust audio handling, efficient data I/O, and comprehensive validation at each phase.

agentId: afb3b6cb0cfc03c4e (use SendMessage with to: 'afb3b6cb0cfc03c4e' to continue this agent)
<usage>total_tokens: 69798
tool_uses: 33
duration_ms: 86575</usage>

Prompt:
Thoroughly explore the repo at /home/ubuntu/training/gemma3n-E2B-asr-ft/. I need to understand:

1. The overall structure - list all files and directories
2. The model architecture - what kind of ASR model is this?
3. Inference support - how do you load the model and run transcription? Look for any inference scripts, model loading code, transcribe methods
4. Any existing benchmark or evaluation scripts
5. The training configuration - what languages, what base model
6. Dependencies and requirements

Read all key Python files, configs, and READMEs. I need enough detail to write a benchmark script that loads a checkpoint and runs inference on audio data.


Command running in background with ID: b5dil7b5b. Output is being written to: /tmp/claude-1001/-home-ubuntu-training/5e207b1e-98a7-43a4-9947-9579680cf6bd/tasks/b5dil7b5b.output


total 12
drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 26 00:48 .
drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 23:02 ..
drwxrwxr-x 3 ubuntu ubuntu 4096 Mar 25 23:02 BayAreaBoys___indic-asr-benchmark-6k


Good — dataset is cached, checkpoint is downloading. Let me check the checkpoint download progress and understand the inference API better.


ls -la /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-10000/ 2>&1 | head -30

Check download progress


total 865156
drwxrwxr-x 2 ubuntu ubuntu      4096 Mar 26 00:51 .
drwxrwxr-x 8 ubuntu ubuntu      4096 Mar 26 00:51 ..
-rw-rw-r-- 1 ubuntu ubuntu       613 Mar 25 08:27 chat_template.jinja
-rw-rw-r-- 1 ubuntu ubuntu      4565 Mar 25 08:27 config.json
-rw-rw-r-- 1 ubuntu ubuntu       209 Mar 25 08:27 generation_config.json
-rw-rw-r-- 1 ubuntu ubuntu 889192448 Mar 26 00:51 model.safetensors.5C8B6FA1
-rw-rw-r-- 1 ubuntu ubuntu  25165824 Mar 26 00:51 optimizer.pt.AE1ed226
-rw-rw-r-- 1 ubuntu ubuntu      2262 Mar 25 08:27 processor_config.json
-rw-rw-r-- 1 ubuntu ubuntu     16389 Mar 25 08:27 rng_state_0.pth
-rw-rw-r-- 1 ubuntu ubuntu     16389 Mar 25 08:27 rng_state_1.pth
-rw-rw-r-- 1 ubuntu ubuntu     16389 Mar 25 08:27 rng_state_2.pth
-rw-rw-r-- 1 ubuntu ubuntu     16389 Mar 25 08:27 rng_state_3.pth
-rw-rw-r-- 1 ubuntu ubuntu     16389 Mar 25 08:27 rng_state_4.pth


     1→# Copyright 2025 THUDM and the LlamaFactory team.
     2→#
     3→# This code is inspired by the THUDM's ChatGLM implementation.
     4→# https://github.com/THUDM/ChatGLM-6B/blob/main/cli_demo.py
     5→#
     6→# Licensed under the Apache License, Version 2.0 (the "License");
     7→# you may not use this file except in compliance with the License.
     8→# You may obtain a copy of the License at
     9→#
    10→#     http://www.apache.org/licenses/LICENSE-2.0
    11→#
    12→# Unless required by applicable law or agreed to in writing, software
    13→# distributed under the License is distributed on an "AS IS" BASIS,
    14→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    15→# See the License for the specific language governing permissions and
    16→# limitations under the License.
    17→
    18→import asyncio
    19→import os
    20→from collections.abc import AsyncGenerator, Generator
    21→from threading import Thread
    22→from typing import TYPE_CHECKING, Any, Optional
    23→
    24→from ..extras.constants import EngineName
    25→from ..extras.misc import torch_gc
    26→from ..hparams import get_infer_args
    27→
    28→
    29→if TYPE_CHECKING:
    30→    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    31→    from .base_engine import BaseEngine, Response
    32→
    33→
    34→def _start_background_loop(loop: "asyncio.AbstractEventLoop") -> None:
    35→    asyncio.set_event_loop(loop)
    36→    loop.run_forever()
    37→
    38→
    39→class ChatModel:
    40→    r"""General class for chat models. Backed by huggingface or vllm engines.
    41→
    42→    Supports both sync and async methods.
    43→    Sync methods: chat(), stream_chat() and get_scores().
    44→    Async methods: achat(), astream_chat() and aget_scores().
    45→    """
    46→
    47→    def __init__(self, args: Optional[dict[str, Any]] = None) -> None:
    48→        model_args, data_args, finetuning_args, generating_args = get_infer_args(args)
    49→
    50→        if model_args.infer_backend == EngineName.HF:
    51→            from .hf_engine import HuggingfaceEngine
    52→
    53→            self.engine: BaseEngine = HuggingfaceEngine(model_args, data_args, finetuning_args, generating_args)
    54→        elif model_args.infer_backend == EngineName.VLLM:
    55→            try:
    56→                from .vllm_engine import VllmEngine
    57→
    58→                self.engine: BaseEngine = VllmEngine(model_args, data_args, finetuning_args, generating_args)
    59→            except ImportError as e:
    60→                raise ImportError(
    61→                    "vLLM not install, you may need to run `pip install vllm`\n"
    62→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    63→                ) from e
    64→        elif model_args.infer_backend == EngineName.SGLANG:
    65→            try:
    66→                from .sglang_engine import SGLangEngine
    67→
    68→                self.engine: BaseEngine = SGLangEngine(model_args, data_args, finetuning_args, generating_args)
    69→            except ImportError as e:
    70→                raise ImportError(
    71→                    "SGLang not install, you may need to run `pip install sglang[all]`\n"
    72→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    73→                ) from e
    74→        elif model_args.infer_backend == EngineName.KT:
    75→            try:
    76→                from .kt_engine import KTransformersEngine
    77→
    78→                self.engine: BaseEngine = KTransformersEngine(model_args, data_args, finetuning_args, generating_args)
    79→            except ImportError as e:
    80→                raise ImportError(
    81→                    "KTransformers not install, you may need to run `pip install ktransformers`\n"
    82→                    "or try to use HuggingFace backend: --infer_backend huggingface"
    83→                ) from e
    84→        else:
    85→            raise NotImplementedError(f"Unknown backend: {model_args.infer_backend}")
    86→
    87→        self._loop = asyncio.new_event_loop()
    88→        self._thread = Thread(target=_start_background_loop, args=(self._loop,), daemon=True)
    89→        self._thread.start()
    90→
    91→    def chat(
    92→        self,
    93→        messages: list[dict[str, str]],
    94→        system: Optional[str] = None,
    95→        tools: Optional[str] = None,
    96→        images: Optional[list["ImageInput"]] = None,
    97→        videos: Optional[list["VideoInput"]] = None,
    98→        audios: Optional[list["AudioInput"]] = None,
    99→        **input_kwargs,
   100→    ) -> list["Response"]:
   101→        r"""Get a list of responses of the chat model."""
   102→        task = asyncio.run_coroutine_threadsafe(
   103→            self.achat(messages, system, tools, images, videos, audios, **input_kwargs), self._loop
   104→        )
   105→        return task.result()
   106→
   107→    async def achat(
   108→        self,
   109→        messages: list[dict[str, str]],
   110→        system: Optional[str] = None,
   111→        tools: Optional[str] = None,
   112→        images: Optional[list["ImageInput"]] = None,
   113→        videos: Optional[list["VideoInput"]] = None,
   114→        audios: Optional[list["AudioInput"]] = None,
   115→        **input_kwargs,
   116→    ) -> list["Response"]:
   117→        r"""Asynchronously get a list of responses of the chat model."""
   118→        return await self.engine.chat(messages, system, tools, images, videos, audios, **input_kwargs)
   119→
   120→    def stream_chat(
   121→        self,
   122→        messages: list[dict[str, str]],
   123→        system: Optional[str] = None,
   124→        tools: Optional[str] = None,
   125→        images: Optional[list["ImageInput"]] = None,
   126→        videos: Optional[list["VideoInput"]] = None,
   127→        audios: Optional[list["AudioInput"]] = None,
   128→        **input_kwargs,
   129→    ) -> Generator[str, None, None]:
   130→        r"""Get the response token-by-token of the chat model."""
   131→        generator = self.astream_chat(messages, system, tools, images, videos, audios, **input_kwargs)
   132→        while True:
   133→            try:
   134→                task = asyncio.run_coroutine_threadsafe(generator.__anext__(), self._loop)
   135→                yield task.result()
   136→            except StopAsyncIteration:
   137→                break
   138→
   139→    async def astream_chat(
   140→        self,
   141→        messages: list[dict[str, str]],
   142→        system: Optional[str] = None,
   143→        tools: Optional[str] = None,
   144→        images: Optional[list["ImageInput"]] = None,
   145→        videos: Optional[list["VideoInput"]] = None,
   146→        audios: Optional[list["AudioInput"]] = None,
   147→        **input_kwargs,
   148→    ) -> AsyncGenerator[str, None]:
   149→        r"""Asynchronously get the response token-by-token of the chat model."""
   150→        async for new_token in self.engine.stream_chat(
   151→            messages, system, tools, images, videos, audios, **input_kwargs
   152→        ):
   153→            yield new_token
   154→
   155→    def get_scores(
   156→        self,
   157→        batch_input: list[str],
   158→        **input_kwargs,
   159→    ) -> list[float]:
   160→        r"""Get a list of scores of the reward model."""
   161→        task = asyncio.run_coroutine_threadsafe(self.aget_scores(batch_input, **input_kwargs), self._loop)
   162→        return task.result()
   163→
   164→    async def aget_scores(
   165→        self,
   166→        batch_input: list[str],
   167→        **input_kwargs,
   168→    ) -> list[float]:
   169→        r"""Asynchronously get a list of scores of the reward model."""
   170→        return await self.engine.get_scores(batch_input, **input_kwargs)
   171→
   172→
   173→def run_chat() -> None:
   174→    if os.name != "nt":
   175→        try:
   176→            import readline  # noqa: F401
   177→        except ImportError:
   178→            print("Install `readline` for a better experience.")
   179→
   180→    chat_model = ChatModel()
   181→    messages = []
   182→    print("Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.")
   183→
   184→    while True:
   185→        try:
   186→            query = input("\nUser: ")
   187→        except UnicodeDecodeError:
   188→            print("Detected decoding error at the inputs, please set the terminal encoding to utf-8.")
   189→            continue
   190→        except Exception:
   191→            raise
   192→
   193→        if query.strip() == "exit":
   194→            break
   195→
   196→        if query.strip() == "clear":
   197→            messages = []
   198→            torch_gc()
   199→            print("History has been removed.")
   200→            continue
   201→
   202→        messages.append({"role": "user", "content": query})
   203→        print("Assistant: ", end="", flush=True)
   204→
   205→        response = ""
   206→        for new_text in chat_model.stream_chat(messages):
   207→            print(new_text, end="", flush=True)
   208→            response += new_text
   209→        print()
   210→        messages.append({"role": "assistant", "content": response})
   211→

# Copyright 2025 THUDM and the LlamaFactory team.
#
# This code is inspired by the THUDM's ChatGLM implementation.
# https://github.com/THUDM/ChatGLM-6B/blob/main/cli_demo.py
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import asyncio
import os
from collections.abc import AsyncGenerator, Generator
from threading import Thread
from typing import TYPE_CHECKING, Any, Optional

from ..extras.constants import EngineName
from ..extras.misc import torch_gc
from ..hparams import get_infer_args

if TYPE_CHECKING:
    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    from .base_engine import BaseEngine, Response

def _start_background_loop(loop: "asyncio.AbstractEventLoop") -> None:
    asyncio.set_event_loop(loop)
    loop.run_forever()

class ChatModel:
    r"""General class for chat models. Backed by huggingface or vllm engines.

    Supports both sync and async methods.
    Sync methods: chat(), stream_chat() and get_scores().
    Async methods: achat(), astream_chat() and aget_scores().
    """

    def __init__(self, args: Optional[dict[str, Any]] = None) -> None:
        model_args, data_args, finetuning_args, generating_args = get_infer_args(args)

        if model_args.infer_backend == EngineName.HF:
            from .hf_engine import HuggingfaceEngine

            self.engine: BaseEngine = HuggingfaceEngine(model_args, data_args, finetuning_args, generating_args)
        elif model_args.infer_backend == EngineName.VLLM:
            try:
                from .vllm_engine import VllmEngine

                self.engine: BaseEngine = VllmEngine(model_args, data_args, finetuning_args, generating_args)
            except ImportError as e:
                raise ImportError(
                    "vLLM not install, you may need to run `pip install vllm`\n"
                    "or try to use HuggingFace backend: --infer_backend huggingface"
                ) from e
        elif model_args.infer_backend == EngineName.SGLANG:
            try:
                from .sglang_engine import SGLangEngine

                self.engine: BaseEngine = SGLangEngine(model_args, data_args, finetuning_args, generating_args)
            except ImportError as e:
                raise ImportError(
                    "SGLang not install, you may need to run `pip install sglang[all]`\n"
                    "or try to use HuggingFace backend: --infer_backend huggingface"
                ) from e
        elif model_args.infer_backend == EngineName.KT:
            try:
                from .kt_engine import KTransformersEngine

                self.engine: BaseEngine = KTransformersEngine(model_args, data_args, finetuning_args, generating_args)
            except ImportError as e:
                raise ImportError(
                    "KTransformers not install, you may need to run `pip install ktransformers`\n"
                    "or try to use HuggingFace backend: --infer_backend huggingface"
                ) from e
        else:
            raise NotImplementedError(f"Unknown backend: {model_args.infer_backend}")

        self._loop = asyncio.new_event_loop()
        self._thread = Thread(target=_start_background_loop, args=(self._loop,), daemon=True)
        self._thread.start()

    def chat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> list["Response"]:
        r"""Get a list of responses of the chat model."""
        task = asyncio.run_coroutine_threadsafe(
            self.achat(messages, system, tools, images, videos, audios, **input_kwargs), self._loop
        )
        return task.result()

    async def achat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> list["Response"]:
        r"""Asynchronously get a list of responses of the chat model."""
        return await self.engine.chat(messages, system, tools, images, videos, audios, **input_kwargs)

    def stream_chat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> Generator[str, None, None]:
        r"""Get the response token-by-token of the chat model."""
        generator = self.astream_chat(messages, system, tools, images, videos, audios, **input_kwargs)
        while True:
            try:
                task = asyncio.run_coroutine_threadsafe(generator.__anext__(), self._loop)
                yield task.result()
            except StopAsyncIteration:
                break

    async def astream_chat(
        self,
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        **input_kwargs,
    ) -> AsyncGenerator[str, None]:
        r"""Asynchronously get the response token-by-token of the chat model."""
        async for new_token in self.engine.stream_chat(
            messages, system, tools, images, videos, audios, **input_kwargs
        ):
            yield new_token

    def get_scores(
        self,
        batch_input: list[str],
        **input_kwargs,
    ) -> list[float]:
        r"""Get a list of scores of the reward model."""
        task = asyncio.run_coroutine_threadsafe(self.aget_scores(batch_input, **input_kwargs), self._loop)
        return task.result()

    async def aget_scores(
        self,
        batch_input: list[str],
        **input_kwargs,
    ) -> list[float]:
        r"""Asynchronously get a list of scores of the reward model."""
        return await self.engine.get_scores(batch_input, **input_kwargs)

def run_chat() -> None:
    if os.name != "nt":
        try:
            import readline  # noqa: F401
        except ImportError:
            print("Install `readline` for a better experience.")

    chat_model = ChatModel()
    messages = []
    print("Welcome to the CLI application, use `clear` to remove the history, use `exit` to exit the application.")

    while True:
        try:
            query = input("\nUser: ")
        except UnicodeDecodeError:
            print("Detected decoding error at the inputs, please set the terminal encoding to utf-8.")
            continue
        except Exception:
            raise

        if query.strip() == "exit":
            break

        if query.strip() == "clear":
            messages = []
            torch_gc()
            print("History has been removed.")
            continue

        messages.append({"role": "user", "content": query})
        print("Assistant: ", end="", flush=True)

        response = ""
        for new_text in chat_model.stream_chat(messages):
            print(new_text, end="", flush=True)
            response += new_text
        print()
        messages.append({"role": "assistant", "content": response})


     1→### Gemma3n ASR Full Fine-Tune Config
     2→### Full parameter training (not LoRA)
     3→
     4→### Model
     5→model_name_or_path: /workspace/gemini-asr/gemma-model
     6→trust_remote_code: true
     7→
     8→### Method
     9→stage: sft
    10→do_train: true
    11→do_eval: true
    12→finetuning_type: full
    13→
    14→### Multimodal
    15→freeze_multi_modal_projector: false
    16→freeze_vision_tower: false
    17→
    18→### Dataset
    19→dataset_dir: /root/gemini-asr/lf_asr/artifacts/phase4
    20→dataset: asr_train
    21→eval_dataset: asr_dev
    22→template: gemma3n
    23→
    24→### Data loading
    25→preprocessing_num_workers: 64
    26→dataloader_num_workers: 8
    27→dataloader_persistent_workers: true
    28→dataloader_prefetch_factor: 4
    29→dataloader_pin_memory: true
    30→remove_unused_columns: false
    31→disable_shuffling: true
    32→
    33→### Training
    34→per_device_train_batch_size: 8
    35→per_device_eval_batch_size: 4
    36→gradient_accumulation_steps: 8
    37→learning_rate: 5.0e-5
    38→num_train_epochs: 1.0
    39→lr_scheduler_type: cosine
    40→warmup_ratio: 0.02
    41→
    42→### Memory optimization
    43→gradient_checkpointing: true
    44→
    45→### Precision + Speed
    46→bf16: true
    47→flash_attn: fa2
    48→
    49→### Logging + Checkpointing
    50→logging_steps: 10
    51→save_steps: 5000
    52→eval_strategy: steps
    53→eval_steps: 5000
    54→save_total_limit: 5
    55→
    56→### Output
    57→output_dir: /root/gemini-asr/lf_asr/outputs/full_ft_run
    58→overwrite_output_dir: true
    59→report_to: wandb
    60→run_name: gemma-3n
    61→
    62→### Misc
    63→seed: 42
    64→cutoff_len: 2048
    65→

### Gemma3n ASR Full Fine-Tune Config
### Full parameter training (not LoRA)

### Model
model_name_or_path: /workspace/gemini-asr/gemma-model
trust_remote_code: true

### Method
stage: sft
do_train: true
do_eval: true
finetuning_type: full

### Multimodal
freeze_multi_modal_projector: false
freeze_vision_tower: false

### Dataset
dataset_dir: /root/gemini-asr/lf_asr/artifacts/phase4
dataset: asr_train
eval_dataset: asr_dev
template: gemma3n

### Data loading
preprocessing_num_workers: 64
dataloader_num_workers: 8
dataloader_persistent_workers: true
dataloader_prefetch_factor: 4
dataloader_pin_memory: true
remove_unused_columns: false
disable_shuffling: true

### Training
per_device_train_batch_size: 8
per_device_eval_batch_size: 4
gradient_accumulation_steps: 8
learning_rate: 5.0e-5
num_train_epochs: 1.0
lr_scheduler_type: cosine
warmup_ratio: 0.02

### Memory optimization
gradient_checkpointing: true

### Precision + Speed
bf16: true
flash_attn: fa2

### Logging + Checkpointing
logging_steps: 10
save_steps: 5000
eval_strategy: steps
eval_steps: 5000
save_total_limit: 5

### Output
output_dir: /root/gemini-asr/lf_asr/outputs/full_ft_run
overwrite_output_dir: true
report_to: wandb
run_name: gemma-3n

### Misc
seed: 42
cutoff_len: 2048


     1→{
     2→  "architectures": [
     3→    "Gemma3nForConditionalGeneration"
     4→  ],
     5→  "audio_config": {
     6→    "conf_attention_chunk_size": 12,
     7→    "conf_attention_context_left": 13,
     8→    "conf_attention_context_right": 0,
     9→    "conf_attention_logit_cap": 50.0,
    10→    "conf_conv_kernel_size": 5,
    11→    "conf_num_attention_heads": 8,
    12→    "conf_num_hidden_layers": 12,
    13→    "conf_reduction_factor": 4,
    14→    "conf_residual_weight": 0.5,
    15→    "dtype": "bfloat16",
    16→    "gradient_clipping": 10000000000.0,
    17→    "hidden_size": 1536,
    18→    "input_feat_size": 128,
    19→    "model_type": "gemma3n_audio",
    20→    "rms_norm_eps": 1e-06,
    21→    "sscp_conv_channel_size": [
    22→      128,
    23→      32
    24→    ],
    25→    "sscp_conv_group_norm_eps": 0.001,
    26→    "sscp_conv_kernel_size": [
    27→      [
    28→        3,
    29→        3
    30→      ],
    31→      [
    32→        3,
    33→        3
    34→      ]
    35→    ],
    36→    "sscp_conv_stride_size": [
    37→      [
    38→        2,
    39→        2
    40→      ],
    41→      [
    42→        2,
    43→        2
    44→      ]
    45→    ],
    46→    "vocab_offset": 262272,
    47→    "vocab_size": 128
    48→  },
    49→  "audio_soft_tokens_per_image": 188,
    50→  "audio_token_id": 262273,
    51→  "boa_token_id": 256000,
    52→  "boi_token_id": 255999,
    53→  "bos_token_id": 2,
    54→  "dtype": "float32",
    55→  "eoa_token_id": 262272,
    56→  "eoi_token_id": 262144,
    57→  "eos_token_id": 106,
    58→  "hidden_size": 2048,
    59→  "image_token_id": 262145,
    60→  "initializer_range": 0.02,
    61→  "model_type": "gemma3n",
    62→  "pad_token_id": 0,
    63→  "text_config": {
    64→    "activation_sparsity_pattern": [
    65→      0.95,
    66→      0.95,
    67→      0.95,
    68→      0.95,
    69→      0.95,
    70→      0.95,
    71→      0.95,
    72→      0.95,
    73→      0.95,
    74→      0.95,
    75→      0.0,
    76→      0.0,
    77→      0.0,
    78→      0.0,
    79→      0.0,
    80→      0.0,
    81→      0.0,
    82→      0.0,
    83→      0.0,
    84→      0.0,
    85→      0.0,
    86→      0.0,
    87→      0.0,
    88→      0.0,
    89→      0.0,
    90→      0.0,
    91→      0.0,
    92→      0.0,
    93→      0.0,
    94→      0.0
    95→    ],
    96→    "altup_active_idx": 0,
    97→    "altup_coef_clip": 120.0,
    98→    "altup_correct_scale": true,
    99→    "altup_num_inputs": 4,
   100→    "attention_bias": false,
   101→    "attention_dropout": 0.0,
   102→    "bos_token_id": 2,
   103→    "dtype": "bfloat16",
   104→    "eos_token_id": 1,
   105→    "final_logit_softcapping": 30.0,
   106→    "head_dim": 256,
   107→    "hidden_activation": "gelu_pytorch_tanh",
   108→    "hidden_size": 2048,
   109→    "hidden_size_per_layer_input": 256,
   110→    "initializer_range": 0.02,
   111→    "intermediate_size": [
   112→      8192,
   113→      8192,
   114→      8192,
   115→      8192,
   116→      8192,
   117→      8192,
   118→      8192,
   119→      8192,
   120→      8192,
   121→      8192,
   122→      8192,
   123→      8192,
   124→      8192,
   125→      8192,
   126→      8192,
   127→      8192,
   128→      8192,
   129→      8192,
   130→      8192,
   131→      8192,
   132→      8192,
   133→      8192,
   134→      8192,
   135→      8192,
   136→      8192,
   137→      8192,
   138→      8192,
   139→      8192,
   140→      8192,
   141→      8192
   142→    ],
   143→    "laurel_rank": 64,
   144→    "layer_types": [
   145→      "sliding_attention",
   146→      "sliding_attention",
   147→      "sliding_attention",
   148→      "sliding_attention",
   149→      "full_attention",
   150→      "sliding_attention",
   151→      "sliding_attention",
   152→      "sliding_attention",
   153→      "sliding_attention",
   154→      "full_attention",
   155→      "sliding_attention",
   156→      "sliding_attention",
   157→      "sliding_attention",
   158→      "sliding_attention",
   159→      "full_attention",
   160→      "sliding_attention",
   161→      "sliding_attention",
   162→      "sliding_attention",
   163→      "sliding_attention",
   164→      "full_attention",
   165→      "sliding_attention",
   166→      "sliding_attention",
   167→      "sliding_attention",
   168→      "sliding_attention",
   169→      "full_attention",
   170→      "sliding_attention",
   171→      "sliding_attention",
   172→      "sliding_attention",
   173→      "sliding_attention",
   174→      "full_attention"
   175→    ],
   176→    "max_position_embeddings": 32768,
   177→    "model_type": "gemma3n_text",
   178→    "num_attention_heads": 8,
   179→    "num_hidden_layers": 30,
   180→    "num_key_value_heads": 2,
   181→    "num_kv_shared_layers": 10,
   182→    "pad_token_id": 0,
   183→    "rms_norm_eps": 1e-06,
   184→    "rope_parameters": {
   185→      "full_attention": {
   186→        "rope_theta": 1000000.0,
   187→        "rope_type": "default"
   188→      },
   189→      "sliding_attention": {
   190→        "rope_theta": 10000.0,
   191→        "rope_type": "default"
   192→      }
   193→    },
   194→    "sliding_window": 512,
   195→    "tie_word_embeddings": true,
   196→    "use_cache": false,
   197→    "vocab_size": 262400,
   198→    "vocab_size_per_layer_input": 262144
   199→  },
   200→  "tie_word_embeddings": true,
   201→  "transformers_version": "5.2.0",
   202→  "use_cache": false,
   203→  "vision_config": {
   204→    "architecture": "mobilenetv5_300m_enc",
   205→    "do_pooling": false,
   206→    "dtype": "bfloat16",
   207→    "hidden_size": 2048,
   208→    "initializer_range": 0.02,
   209→    "label_names": [
   210→      "LABEL_0",
   211→      "LABEL_1"
   212→    ],
   213→    "model_args": null,
   214→    "model_type": "gemma3n_vision",
   215→    "num_classes": 2,
   216→    "rms_norm_eps": 1e-06,
   217→    "vocab_offset": 262144,
   218→    "vocab_size": 128
   219→  },
   220→  "vision_soft_tokens_per_image": 256
   221→}
   222→


     1→{
     2→  "audio_seq_length": 188,
     3→  "feature_extractor": {
     4→    "crop_size": null,
     5→    "data_format": "channels_first",
     6→    "default_to_square": false,
     7→    "device": null,
     8→    "disable_grouping": null,
     9→    "dither": 0.0,
    10→    "do_center_crop": null,
    11→    "do_convert_rgb": null,
    12→    "do_normalize": false,
    13→    "do_rescale": true,
    14→    "do_resize": true,
    15→    "feature_extractor_type": "Gemma3nAudioFeatureExtractor",
    16→    "feature_size": 128,
    17→    "fft_length": 1024,
    18→    "fft_overdrive": true,
    19→    "frame_length": 512,
    20→    "hop_length": 160,
    21→    "image_mean": [
    22→      0.5,
    23→      0.5,
    24→      0.5
    25→    ],
    26→    "image_processor_type": "SiglipImageProcessorFast",
    27→    "image_seq_length": 256,
    28→    "image_std": [
    29→      0.5,
    30→      0.5,
    31→      0.5
    32→    ],
    33→    "input_data_format": null,
    34→    "input_scale_factor": 1.0,
    35→    "max_frequency": 7600.0,
    36→    "mel_floor": 1e-05,
    37→    "min_frequency": 125.0,
    38→    "padding_side": "right",
    39→    "padding_value": 0.0,
    40→    "per_bin_mean": null,
    41→    "per_bin_stddev": null,
    42→    "preemphasis": 0.97,
    43→    "preemphasis_htk_flavor": true,
    44→    "resample": 2,
    45→    "rescale_factor": 0.00392156862745098,
    46→    "return_attention_mask": true,
    47→    "return_tensors": null,
    48→    "sampling_rate": 16000,
    49→    "size": {
    50→      "height": 768,
    51→      "width": 768
    52→    }
    53→  },
    54→  "image_processor": {
    55→    "data_format": "channels_first",
    56→    "default_to_square": false,
    57→    "dither": 0.0,
    58→    "do_normalize": false,
    59→    "do_rescale": true,
    60→    "do_resize": true,
    61→    "feature_size": 128,
    62→    "fft_length": 1024,
    63→    "fft_overdrive": true,
    64→    "frame_length": 512,
    65→    "hop_length": 160,
    66→    "image_mean": [
    67→      0.5,
    68→      0.5,
    69→      0.5
    70→    ],
    71→    "image_processor_type": "SiglipImageProcessorFast",
    72→    "image_seq_length": 256,
    73→    "image_std": [
    74→      0.5,
    75→      0.5,
    76→      0.5
    77→    ],
    78→    "input_scale_factor": 1.0,
    79→    "max_frequency": 7600.0,
    80→    "mel_floor": 1e-05,
    81→    "min_frequency": 125.0,
    82→    "padding_side": "right",
    83→    "padding_value": 0.0,
    84→    "preemphasis": 0.97,
    85→    "preemphasis_htk_flavor": true,
    86→    "resample": 2,
    87→    "rescale_factor": 0.00392156862745098,
    88→    "return_attention_mask": true,
    89→    "sampling_rate": 16000,
    90→    "size": {
    91→      "height": 768,
    92→      "width": 768
    93→    }
    94→  },
    95→  "image_seq_length": 256,
    96→  "processor_class": "Gemma3nProcessor"
    97→}
    98→


du -sh /home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-10000/ 2>&1

Check current download size


1.8G	/home/ubuntu/training/checkpoints/gemma3n-e2b-ckpt-10000/


Checkpoint is downloading (~1.8G so far of ~20.3GB model). Let me check what's needed for inference and verify the environment supports Gemma3n.


python3 -c "import transformers; print(transformers.__version__)" 2>&1

Check transformers version


python3 -c "from transformers import AutoModelForImageTextToText, AutoProcessor; print('Gemma3n imports OK')" 2>&1

Check if Gemma3n model class is available


Gemma3n imports OK


     1→# Copyright 2025 the LlamaFactory team.
     2→#
     3→# Licensed under the Apache License, Version 2.0 (the "License");
     4→# you may not use this file except in compliance with the License.
     5→# You may obtain a copy of the License at
     6→#
     7→#     http://www.apache.org/licenses/LICENSE-2.0
     8→#
     9→# Unless required by applicable law or agreed to in writing, software
    10→# distributed under the License is distributed on an "AS IS" BASIS,
    11→# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    12→# See the License for the specific language governing permissions and
    13→# limitations under the License.
    14→
    15→import asyncio
    16→import os
    17→from collections.abc import AsyncGenerator, Callable
    18→from threading import Thread
    19→from typing import TYPE_CHECKING, Any, Optional, Union
    20→
    21→import torch
    22→from transformers import GenerationConfig, TextIteratorStreamer
    23→from typing_extensions import override
    24→
    25→from ..data import get_template_and_fix_tokenizer
    26→from ..extras import logging
    27→from ..extras.constants import AUDIO_PLACEHOLDER, IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, EngineName
    28→from ..model import load_model, load_tokenizer
    29→from .base_engine import BaseEngine, Response
    30→
    31→
    32→if TYPE_CHECKING:
    33→    from transformers import PreTrainedModel, PreTrainedTokenizer, ProcessorMixin
    34→    from trl import PreTrainedModelWrapper
    35→
    36→    from ..data import Template
    37→    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    38→    from ..hparams import DataArguments, FinetuningArguments, GeneratingArguments, ModelArguments
    39→
    40→
    41→logger = logging.get_logger(__name__)
    42→
    43→
    44→class HuggingfaceEngine(BaseEngine):
    45→    def __init__(
    46→        self,
    47→        model_args: "ModelArguments",
    48→        data_args: "DataArguments",
    49→        finetuning_args: "FinetuningArguments",
    50→        generating_args: "GeneratingArguments",
    51→    ) -> None:
    52→        self.name = EngineName.HF
    53→        self.can_generate = finetuning_args.stage == "sft"
    54→        tokenizer_module = load_tokenizer(model_args)
    55→        self.tokenizer = tokenizer_module["tokenizer"]
    56→        self.processor = tokenizer_module["processor"]
    57→        self.tokenizer.padding_side = "left" if self.can_generate else "right"
    58→        self.template = get_template_and_fix_tokenizer(self.tokenizer, data_args)
    59→        self.model = load_model(
    60→            self.tokenizer, model_args, finetuning_args, is_trainable=False, add_valuehead=(not self.can_generate)
    61→        )  # must after fixing tokenizer to resize vocab
    62→        self.generating_args = generating_args.to_dict()
    63→        try:
    64→            asyncio.get_event_loop()
    65→        except RuntimeError:
    66→            logger.warning_rank0_once("There is no current event loop, creating a new one.")
    67→            loop = asyncio.new_event_loop()
    68→            asyncio.set_event_loop(loop)
    69→
    70→        self.semaphore = asyncio.Semaphore(int(os.getenv("MAX_CONCURRENT", "1")))
    71→
    72→    @staticmethod
    73→    def _process_args(
    74→        model: "PreTrainedModel",
    75→        tokenizer: "PreTrainedTokenizer",
    76→        processor: Optional["ProcessorMixin"],
    77→        template: "Template",
    78→        generating_args: dict[str, Any],
    79→        messages: list[dict[str, str]],
    80→        system: Optional[str] = None,
    81→        tools: Optional[str] = None,
    82→        images: Optional[list["ImageInput"]] = None,
    83→        videos: Optional[list["VideoInput"]] = None,
    84→        audios: Optional[list["AudioInput"]] = None,
    85→        input_kwargs: Optional[dict[str, Any]] = {},
    86→    ) -> tuple[dict[str, Any], int]:
    87→        mm_input_dict = {"images": [], "videos": [], "audios": [], "imglens": [0], "vidlens": [0], "audlens": [0]}
    88→        if images is not None:
    89→            mm_input_dict.update({"images": images, "imglens": [len(images)]})
    90→            if not any(IMAGE_PLACEHOLDER in message["content"] for message in messages):
    91→                messages[0]["content"] = IMAGE_PLACEHOLDER * len(images) + messages[0]["content"]
    92→
    93→        if videos is not None:
    94→            mm_input_dict.update({"videos": videos, "vidlens": [len(videos)]})
    95→            if not any(VIDEO_PLACEHOLDER in message["content"] for message in messages):
    96→                messages[0]["content"] = VIDEO_PLACEHOLDER * len(videos) + messages[0]["content"]
    97→
    98→        if audios is not None:
    99→            mm_input_dict.update({"audios": audios, "audlens": [len(audios)]})
   100→            if not any(AUDIO_PLACEHOLDER in message["content"] for message in messages):

# Copyright 2025 the LlamaFactory team.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import asyncio
import os
from collections.abc import AsyncGenerator, Callable
from threading import Thread
from typing import TYPE_CHECKING, Any, Optional, Union

import torch
from transformers import GenerationConfig, TextIteratorStreamer
from typing_extensions import override

from ..data import get_template_and_fix_tokenizer
from ..extras import logging
from ..extras.constants import AUDIO_PLACEHOLDER, IMAGE_PLACEHOLDER, VIDEO_PLACEHOLDER, EngineName
from ..model import load_model, load_tokenizer
from .base_engine import BaseEngine, Response

if TYPE_CHECKING:
    from transformers import PreTrainedModel, PreTrainedTokenizer, ProcessorMixin
    from trl import PreTrainedModelWrapper

    from ..data import Template
    from ..data.mm_plugin import AudioInput, ImageInput, VideoInput
    from ..hparams import DataArguments, FinetuningArguments, GeneratingArguments, ModelArguments

logger = logging.get_logger(__name__)

class HuggingfaceEngine(BaseEngine):
    def __init__(
        self,
        model_args: "ModelArguments",
        data_args: "DataArguments",
        finetuning_args: "FinetuningArguments",
        generating_args: "GeneratingArguments",
    ) -> None:
        self.name = EngineName.HF
        self.can_generate = finetuning_args.stage == "sft"
        tokenizer_module = load_tokenizer(model_args)
        self.tokenizer = tokenizer_module["tokenizer"]
        self.processor = tokenizer_module["processor"]
        self.tokenizer.padding_side = "left" if self.can_generate else "right"
        self.template = get_template_and_fix_tokenizer(self.tokenizer, data_args)
        self.model = load_model(
            self.tokenizer, model_args, finetuning_args, is_trainable=False, add_valuehead=(not self.can_generate)
        )  # must after fixing tokenizer to resize vocab
        self.generating_args = generating_args.to_dict()
        try:
            asyncio.get_event_loop()
        except RuntimeError:
            logger.warning_rank0_once("There is no current event loop, creating a new one.")
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)

        self.semaphore = asyncio.Semaphore(int(os.getenv("MAX_CONCURRENT", "1")))

    @staticmethod
    def _process_args(
        model: "PreTrainedModel",
        tokenizer: "PreTrainedTokenizer",
        processor: Optional["ProcessorMixin"],
        template: "Template",
        generating_args: dict[str, Any],
        messages: list[dict[str, str]],
        system: Optional[str] = None,
        tools: Optional[str] = None,
        images: Optional[list["ImageInput"]] = None,
        videos: Optional[list["VideoInput"]] = None,
        audios: Optional[list["AudioInput"]] = None,
        input_kwargs: Optional[dict[str, Any]] = {},
    ) -> tuple[dict[str, Any], int]:
        mm_input_dict = {"images": [], "videos": [], "audios": [], "imglens": [0], "vidlens": [0], "audlens": [0]}
        if images is not None:
            mm_input_dict.update({"images": images, "imglens": [len(images)]})
            if not any(IMAGE_PLACEHOLDER in message["content"] for message in messages):
                messages[0]["content"] = IMAGE_PLACEHOLDER * len(images) + messages[0]["content"]

        if videos is not None:
            mm_input_dict.update({"videos": videos, "vidlens": [len(videos)]})
            if not any(VIDEO_PLACEHOLDER in message["content"] for message in messages):
                messages[0]["content"] = VIDEO_PLACEHOLDER * len(videos) + messages[0]["content"]

        if audios is not None:
            mm_input_dict.update({"audios": audios, "audlens": [len(audios)]})
            if not any(AUDIO_PLACEHOLDER in message["content"] for message in messages):


Transformers version mismatch — checkpoint was trained with 5.2.0 but we have 4.57.6. Let me check compatibility.


python3 -c "from transformers import Gemma3nForConditionalGeneration; print('OK')" 2>&1

Check if Gemma3n model class exists


python3 -c "from transformers import Gemma3nProcessor; print('OK')" 2>&1

Check if Gemma3n processor exists


OK


Good — `Gemma3nForConditionalGeneration` and `Gemma3nProcessor` are available. Now let me check the HF engine's audio processing and the template to understand the exact prompt format.