Output:
Exit code: 0
Wall time: 0 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_next.py:40:    GemmaRMSNorm as Qwen3NextRMSNorm,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:9:from transformers import BatchFeature, Gemma3Config, Gemma3Processor
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:10:from transformers.models.gemma3.processing_gemma3 import Gemma3ProcessorKwargs
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:15:from vllm.model_executor.layers.layernorm import GemmaRMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:56:class Gemma3ImagePixelInputs(TensorSchema):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:74:Gemma3ImageInputs = Gemma3ImagePixelInputs
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:77:class Gemma3ProcessingInfo(BaseProcessingInfo):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:79:        return self.ctx.get_hf_config(Gemma3Config)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:82:        return self.ctx.get_hf_processor(Gemma3Processor, **kwargs)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:89:        processor: Gemma3Processor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:94:            Gemma3ProcessorKwargs,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:114:        processor: Gemma3Processor | None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:144:        # Based on Gemma3ImageProcessor.pan_and_scan
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:183:        processor: Gemma3Processor | None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:218:        processor: Gemma3Processor | None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:245:class Gemma3DummyInputsBuilder(BaseDummyInputsBuilder[Gemma3ProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:276:class Gemma3MultiModalProcessor(BaseMultiModalProcessor[Gemma3ProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:435:class Gemma3MultiModalProjector(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:436:    def __init__(self, config: Gemma3Config):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:445:        self.mm_soft_emb_norm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:480:    Gemma3MultiModalProcessor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:481:    info=Gemma3ProcessingInfo,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:482:    dummy_inputs=Gemma3DummyInputsBuilder,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:484:class Gemma3ForConditionalGeneration(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:530:        self.multi_modal_projector = Gemma3MultiModalProjector(config)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:536:            architectures=["Gemma3ForCausalLM"],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:555:    ) -> Gemma3ImageInputs | None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:559:        assert image_embeds is None, "Gemma3 does not support image_embeds."
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:565:        return Gemma3ImagePixelInputs(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3_mm.py:580:        image_input: Gemma3ImageInputs,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:103:    "GemmaForCausalLM": ("gemma", "GemmaForCausalLM"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:104:    "Gemma2ForCausalLM": ("gemma2", "Gemma2ForCausalLM"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:105:    "Gemma3ForCausalLM": ("gemma3", "Gemma3ForCausalLM"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:106:    "Gemma3nForCausalLM": ("gemma3n", "Gemma3nForCausalLM"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:196:    "Gemma2Model": ("gemma2", "Gemma2ForCausalLM"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:197:    "Gemma3TextModel": ("gemma3", "Gemma3Model"),
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:295:    "Gemma3ForConditionalGeneration": ("gemma3_mm", "Gemma3ForConditionalGeneration"),  # noqa: E501
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:296:    "Gemma3nForConditionalGeneration": (
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:298:        "Gemma3nForConditionalGeneration",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:377:    "PaliGemmaForConditionalGeneration": (
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/registry.py:379:        "PaliGemmaForConditionalGeneration",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/adapters.py:516:    #   - GemmaForCausalLM
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:24:from transformers import Gemma2Config
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:32:from vllm.model_executor.layers.layernorm import GemmaRMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:61:class Gemma2MLP(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:88:                "Gemma2 uses `gelu_pytorch_tanh` as the hidden activation "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:101:class Gemma2Attention(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:104:        config: Gemma2Config,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:189:class Gemma2DecoderLayer(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:192:        config: Gemma2Config,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:199:        self.self_attn = Gemma2Attention(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:212:        self.mlp = Gemma2MLP(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:220:        self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:221:        self.post_attention_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:224:        self.pre_feedforward_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:227:        self.post_feedforward_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:257:class Gemma2Model(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:272:            lambda prefix: Gemma2DecoderLayer(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:277:        self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:376:class Gemma2ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:395:        # currently all existing Gemma models have `tie_word_embeddings` enabled
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma2.py:398:        self.model = Gemma2Model(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:12:    Gemma3nAudioConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:13:    Gemma3nAudioFeatureExtractor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:14:    Gemma3nConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:15:    Gemma3nProcessor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:16:    Gemma3nTextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:17:    Gemma3nVisionConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:28:from vllm.model_executor.models.gemma3n import Gemma3nForCausalLM
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:72:class Gemma3nImagePixelInputs(TensorSchema):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:85:class Gemma3nAudioInputs(TensorSchema):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:98:Gemma3nImageInputs = Gemma3nImagePixelInputs
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:101:class Gemma3nProcessingInfo(BaseProcessingInfo):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:103:        return self.ctx.get_hf_config(Gemma3nConfig)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:106:        return self.ctx.get_hf_processor(Gemma3nProcessor, **kwargs)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:121:        processor: Optional[Gemma3nProcessor],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:126:        For Gemma3n, this should return the full_image_sequence which includes
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:139:        processor: Optional[Gemma3nProcessor],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:144:        For Gemma3n, this should return the full_audio_sequence which includes
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:156:class Gemma3nDummyInputsBuilder(BaseDummyInputsBuilder[Gemma3nProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:176:        audio_feature_extractor: Gemma3nAudioFeatureExtractor = (
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:200:class Gemma3nMultiModalProcessor(BaseMultiModalProcessor[Gemma3nProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:384:class Gemma3nMultimodalEmbedder(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:390:        multimodal_config: Union[Gemma3nAudioConfig, Gemma3nVisionConfig],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:391:        text_config: Gemma3nTextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:459:    Gemma3nMultiModalProcessor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:460:    info=Gemma3nProcessingInfo,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:461:    dummy_inputs=Gemma3nDummyInputsBuilder,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:463:class Gemma3nForConditionalGeneration(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:506:        self.embed_vision = Gemma3nMultimodalEmbedder(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:509:        self.embed_audio = Gemma3nMultimodalEmbedder(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:517:            architectures=["Gemma3nForCausalLM"],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:519:        self.language_model = cast(Gemma3nForCausalLM, self.language_model)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:532:    ) -> Optional[Gemma3nImageInputs]:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:536:        assert image_embeds is None, "Gemma3n does not support image_embeds."
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:540:        return Gemma3nImagePixelInputs(pixel_values=pixel_values)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:544:    ) -> Optional[Gemma3nAudioInputs]:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:553:        return Gemma3nAudioInputs(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:582:        image_input: Gemma3nImageInputs,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:608:        audio_input: Gemma3nAudioInputs,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:620:        # The Gemma3nProcessor expects all audio will be 30s in length and inserts 188 audio soft tokens into the
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py:770:        Gemma3n supports "free-form" transcription.
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:18:"""Inference-only Gemma model compatible with HuggingFace weights."""
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:27:from transformers import GemmaConfig
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:35:from vllm.model_executor.layers.layernorm import GemmaRMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:68:                "Gemma's activation function was incorrectly set to exact GeLU "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:86:            f"Activation function {hidden_act} is not supported for Gemma models."
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:90:class GemmaMLP(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:124:class GemmaAttention(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:204:class GemmaDecoderLayer(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:207:        config: GemmaConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:214:        self.self_attn = GemmaAttention(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:225:        self.mlp = GemmaMLP(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:233:        self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:234:        self.post_attention_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:262:class GemmaModel(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:278:            lambda prefix: GemmaDecoderLayer(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:283:        self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:367:class GemmaForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:386:        # currently all existing Gemma models have `tie_word_embeddings` enabled
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma.py:390:        self.model = GemmaModel(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:8:from transformers import BatchFeature, PaliGemmaConfig
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:51:class PaliGemmaImagePixelInputs(TensorSchema):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:64:class PaliGemmaImageEmbeddingInputs(TensorSchema):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:76:PaliGemmaImageInputs: TypeAlias = (
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:77:    PaliGemmaImagePixelInputs | PaliGemmaImageEmbeddingInputs
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:81:class PaliGemmaMultiModalProjector(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:92:class PaliGemmaProcessingInfo(BaseProcessingInfo):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:94:        return self.ctx.get_hf_config(PaliGemmaConfig)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:116:class PaliGemmaDummyInputsBuilder(BaseDummyInputsBuilder[PaliGemmaProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:144:class PaliGemmaMultiModalProcessor(BaseMultiModalProcessor[PaliGemmaProcessingInfo]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:249:    PaliGemmaMultiModalProcessor,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:250:    info=PaliGemmaProcessingInfo,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:251:    dummy_inputs=PaliGemmaDummyInputsBuilder,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:253:class PaliGemmaForConditionalGeneration(nn.Module, SupportsMultiModal, SupportsPP):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:296:        self.multi_modal_projector = PaliGemmaMultiModalProjector(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:304:            config.text_config.architectures = ["GemmaForCausalLM"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:306:            config.text_config.architectures = ["Gemma2ForCausalLM"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:321:    ) -> PaliGemmaImageInputs | None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:331:            return PaliGemmaImagePixelInputs(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:338:            return PaliGemmaImageEmbeddingInputs(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/paligemma.py:357:        image_input: PaliGemmaImageInputs,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/transformers/utils.py:29:from vllm.model_executor.layers.layernorm import GemmaRMSNorm, RMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/transformers/utils.py:145:    - `with_scale` indicates whether the layer has a weight (Gemma3n only).
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/transformers/utils.py:155:    # Check if weight is all zeros, which indicates GemmaRMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/transformers/utils.py:167:        return GemmaRMSNorm(**kwargs)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/config.py:27:class Gemma3TextModelConfig:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/config.py:517:    "Gemma3TextModel": Gemma3TextModelConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/transformers/multimodal.py:304:        # Gemma3 and PaliGemma needs `token_type_ids` to work correctly
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:23:from transformers import Gemma3TextConfig
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:32:from vllm.model_executor.layers.layernorm import GemmaRMSNorm
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:62:class Gemma3MLP(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:88:                "Gemma3 uses `gelu_pytorch_tanh` as the hidden activation "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:101:class Gemma3Attention(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:104:        config: Gemma3TextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:154:        self.q_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:155:        self.k_norm = GemmaRMSNorm(self.head_dim, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:229:class Gemma3DecoderLayer(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:232:        config: Gemma3TextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:239:        self.self_attn = Gemma3Attention(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:252:        self.mlp = Gemma3MLP(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:259:        self.input_layernorm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:260:        self.post_attention_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:263:        self.pre_feedforward_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:266:        self.post_feedforward_layernorm = GemmaRMSNorm(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:298:class Gemma3Model(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:315:            lambda prefix: Gemma3DecoderLayer(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:320:        self.norm = GemmaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:447:class Gemma3ForCausalLM(nn.Module, SupportsLoRA, SupportsPP):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:466:        # currently all existing Gemma models have `tie_word_embeddings` enabled
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3.py:469:        self.model = Gemma3Model(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:22:from transformers.models.gemma3n.configuration_gemma3n import Gemma3nTextConfig
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:68:class Gemma3nAltUp(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:189:class Gemma3nLaurelBlock(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:231:class Gemma3nMLP(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:258:                "Gemma3 uses `gelu_pytorch_tanh` as the hidden activation "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:278:class Gemma3nAttention(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:281:        config: Gemma3nTextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:366:                #   - Gemma3nForCausalLM → parameters live under "model.layers"
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:367:                #   - Gemma3nForConditionalGeneration →
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:377:                        "Unexpected prefix format for Gemma3nAttention: "
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:430:class Gemma3nDecoderLayer(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:433:        config: Gemma3nTextConfig,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:439:        assert isinstance(config, Gemma3nTextConfig)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:443:        self.altup = Gemma3nAltUp(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:452:        self.self_attn = Gemma3nAttention(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:463:        self.mlp = Gemma3nMLP(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:474:        self.laurel = Gemma3nLaurelBlock(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:577:class Gemma3nSelfDecoder(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:587:        decoder_layers: list[Gemma3nDecoderLayer],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:746:class Gemma3nCrossDecoder(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:756:        decoder_layers: list[Gemma3nDecoderLayer],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:790:class Gemma3nTextModel(nn.Module, SupportsQuant):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:817:            lambda prefix: Gemma3nDecoderLayer(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:833:            self.self_decoder = Gemma3nSelfDecoder(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:841:            self.cross_decoder = Gemma3nCrossDecoder(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:1103:class Gemma3nForCausalLM(nn.Module):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n.py:1122:        self.model = Gemma3nTextModel(


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Reviewing model methods**

I'm opening the gemma3n_mm model file to examine get_generation_prompt and supports_transcription methods for their implementation details.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
759:    def get_generation_prompt(
770:        Gemma3n supports "free-form" transcription.
771:        We fix its prompt here to standardize transcriptions/translations
774:        # Transcribe this audio [into <>] | for transcription
800:    def get_speech_to_text_config(


Summary: auto


sed -n '730,840p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        self,
        hidden_states: torch.Tensor,
    ) -> Optional[torch.Tensor]:
        return self.language_model.compute_logits(hidden_states)

    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
        loader = AutoWeightsLoader(self)
        return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)

    def get_mm_mapping(self) -> MultiModelKeys:
        """
        Get the module prefix in multimodal models
        """
        return MultiModelKeys.from_string_field(
            language_model="language_model",
            connector="multi_modal_projector",
            tower_model="vision_tower",
        )

    @classmethod
    def get_placeholder_str(cls, modality: str, i: int) -> Optional[str]:
        if modality == "image":
            return "<image_soft_token>"
        elif modality == "audio":
            return "<audio_soft_token>"
        else:
            raise ValueError(f"Unsupported modality: {modality}")

    @classmethod
    def get_generation_prompt(
        cls,
        audio: np.ndarray,
        stt_config: SpeechToTextConfig,
        model_config: ModelConfig,
        language: Optional[str],
        task_type: Literal["transcribe", "translate"],
        request_prompt: str,
        to_language: Optional[str],
    ) -> PromptType:
        """
        Gemma3n supports "free-form" transcription.
        We fix its prompt here to standardize transcriptions/translations
        requests.
        """
        # Transcribe this audio [into <>] | for transcription
        # Translate this audio [from <> into <>] | for translation
        prompt = "<start_of_turn>user\n"
        prompt += "Transcribe" if task_type == "transcribe" else "Translate"
        prompt += " this audio"

        # We assume the language is a valid ISO 639-1 code.
        full_lang_name = cls.supported_languages.get(language, "")
        # Translation only for now
        full_lang_name_to = cls.supported_languages.get(to_language, "")

        if task_type == "transcribe" and full_lang_name:
            prompt += f" into {full_lang_name}"
        elif task_type == "translate":
            if full_lang_name:
                prompt += f" from {full_lang_name}"
            if full_lang_name_to:
                prompt += f" into {full_lang_name_to}"

        prompt += ": <audio_soft_token><end_of_turn>\n<start_of_turn>model\n"

        audio = (audio, stt_config.sample_rate)
        prompts_dict = {"multi_modal_data": {"audio": audio}, "prompt": prompt}
        return cast(PromptType, prompts_dict)

    @classmethod
    def get_speech_to_text_config(
        cls, model_config: ModelConfig, task_type: str
    ) -> SpeechToTextConfig:
        return SpeechToTextConfig(
            # Let's set this to 30 as suggested in the docs for now, although
            # the model is only limited by its context length.
            max_audio_clip_s=30,
            sample_rate=16000,
            # TODO enable chunking after more thorough testing.
            min_energy_split_window_size=None,
        )


**Inspecting audio decoding for input format**

I'm checking decode_pcm16 to confirm if it outputs int16 or float audio since the model inputs likely require float arrays; this could explain transcription inaccuracies.


rg -n "def decode_pcm16|decode" -S maya_transcribe/audio.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
66:def decode_pcm16(path: Path, sample_rate: int = 16000, mono: bool = True) -> tuple[np.ndarray, int]:
68:    Decode audio with ffmpeg into signed 16-bit PCM (optionally mono) at `sample_rate`.
85:    wav, sr = decode_pcm16(path, sample_rate=sample_rate, mono=True)


sed -n '60,100p' maya_transcribe/audio.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        except (TypeError, ValueError):
            channels = None

    return AudioInfo(path=str(path), duration_sec=duration_sec, sample_rate=sample_rate, channels=channels)

def decode_pcm16(path: Path, sample_rate: int = 16000, mono: bool = True) -> tuple[np.ndarray, int]:
    """
    Decode audio with ffmpeg into signed 16-bit PCM (optionally mono) at `sample_rate`.
    Returns (float32 waveform in [-1, 1], sample_rate).
    """
    cmd = ["ffmpeg", "-v", "error", "-i", str(path), "-f", "s16le", "-ar", str(sample_rate)]
    if mono:
        cmd += ["-ac", "1"]
    cmd += ["-"]

    proc = subprocess.run(cmd, check=True, capture_output=True)
    pcm = np.frombuffer(proc.stdout, dtype=np.int16)
    if pcm.size == 0:
        return np.zeros((0,), dtype=np.float32), sample_rate
    wav = pcm.astype(np.float32) / 32768.0
    return wav, sample_rate

def compute_qc(path: Path, sample_rate: int = 16000) -> AudioQCMetrics:
    wav, sr = decode_pcm16(path, sample_rate=sample_rate, mono=True)
    duration_sec = float(wav.shape[0]) / float(sr) if sr else 0.0
    if wav.size == 0:
        return AudioQCMetrics(duration_sec=0.0, rms_dbfs=float("-inf"), peak_dbfs=float("-inf"), clipping_fraction=0.0)

    peak = float(np.max(np.abs(wav)))
    rms = float(np.sqrt(np.mean(np.square(wav))))

    peak_dbfs = 20.0 * float(np.log10(max(peak, 1e-12)))
    rms_dbfs = 20.0 * float(np.log10(max(rms, 1e-12)))

    clipping_threshold = 0.999
    clipping_fraction = float(np.mean(np.abs(wav) >= clipping_threshold))

    return AudioQCMetrics(
        duration_sec=duration_sec,


**Confirming audio input length expectation**

The gemma3n processor expects all audio inputs to be exactly 30 seconds, inserting 188 audio soft tokens accordingly. I'll check if padding or truncation is needed when passing shorter segments to avoid unexpected behavior.


sed -n '600,700p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        )
        # Normalize and embed the soft tokens into language model space.
        vision_outputs *= self.config.vision_config.hidden_size**0.5
        # Return a list of embeddings instead of a batched tensor
        return self.embed_vision(inputs_embeds=vision_outputs).unbind(0)

    def _process_audio_input(
        self,
        audio_input: Gemma3nAudioInputs,
    ) -> list[torch.Tensor]:
        assert self.audio_tower is not None
        # Run on padded features to enable batching
        input_features = audio_input["input_features_padded"].squeeze(1)
        input_features_mask = audio_input["input_features_mask"].squeeze(1)
        audio_outputs, audio_mask = self.audio_tower(
            input_features, ~input_features_mask
        )
        audio_features = self.embed_audio(inputs_embeds=audio_outputs)

        # ruff: noqa
        # The Gemma3nProcessor expects all audio will be 30s in length and inserts 188 audio soft tokens into the
        # text to account for this. However, the audio preprocessing and encoder do not guarantee they will
        # produce 188 soft tokens; they will produce at most that many tokens, but they may produce fewer tokens
        # depending on the length of the longest audio input in the batch. When we encounter this situation, we pad
        # the audio feature out to 188 soft tokens with the embedding of the last token in the embed_audio vocab.
        # TODO precompute and cache padding
        audio_padding_toks = torch.tensor(
            [[self.vocab_size - 1]], dtype=torch.long, device=audio_features.device
        )
        audio_padding_embs = self.embed_audio(input_ids=audio_padding_toks)
        audio_features = torch.where(
            audio_mask.unsqueeze(-1), audio_padding_embs, audio_features
        )

        audio_batch_size, audio_seq_len, audio_embed_dim = audio_features.shape
        extra_padding_tokens = self.config.audio_soft_tokens_per_image - audio_seq_len  # noqa: E501
        extra_padding_features = audio_padding_embs.expand(
            audio_batch_size, extra_padding_tokens, audio_embed_dim
        )

        audio_features = torch.cat((audio_features, extra_padding_features), dim=1)
        # Return a list of embeddings instead of a batched tensor
        return audio_features.unbind(0)

    def get_language_model(self) -> torch.nn.Module:
        return self.language_model

    def embed_multimodal(self, **kwargs: object) -> MultiModalEmbeddings:
        mm_input_by_modality = self._parse_and_validate_multimodal_inputs(**kwargs)
        if mm_input_by_modality is None:
            return []

        multimodal_embeddings: list[torch.Tensor] = []

        # NOTE: It is important to iterate over the keys in this dictionary
        # to preserve the order of the modalities.
        for modality in mm_input_by_modality:
            multimodal_input = mm_input_by_modality[modality]
            if modality == "image":
                vision_embeddings = self._process_image_input(multimodal_input)
                multimodal_embeddings.extend(vision_embeddings)
            if modality == "audio":
                audio_embeddings = self._process_audio_input(multimodal_input)
                multimodal_embeddings.extend(audio_embeddings)
        return multimodal_embeddings

    def embed_input_ids(
        self,
        input_ids: torch.Tensor,
        multimodal_embeddings: Optional[MultiModalEmbeddings] = None,
        *,
        is_multimodal: Optional[torch.Tensor] = None,
        handle_oov_mm_token: bool = False,
    ) -> torch.Tensor:
        # NOTE (NickLucche) Each pass needs tokens to compute PLE so we cache
        # them here, as the model  forward has only access to the input_embeds.
        if input_ids is not None:
            per_layer_inputs = self.language_model.model.get_per_layer_input_embeddings(
                input_ids
            )
            per_layer_inputs = per_layer_inputs.reshape(
                -1,
                self.config.text_config.num_hidden_layers,
                self.config.text_config.hidden_size_per_layer_input,
            )
            self.per_layer_embeddings[: per_layer_inputs.shape[0]].copy_(
                per_layer_inputs
            )

        # This is to satisfy the type checker for each overload
        if multimodal_embeddings is None or is_multimodal is None:
            return super().embed_input_ids(input_ids)

        return super().embed_input_ids(
            input_ids,
            multimodal_embeddings=multimodal_embeddings,
            is_multimodal=is_multimodal,
            handle_oov_mm_token=handle_oov_mm_token,
        )

    def forward(


**Assessing model suitability for ASR**

I'm evaluating whether gemma3n supports ASR or is better for multimodal tasks by comparing audio lengths, outputs, and WER values, planning to sample segments and analyze high WER cases for misalignment or model mismatch.


**Verifying correct transcription API usage**

I'm investigating whether vLLM requires using a dedicated transcription method instead of the general generate call for audio tasks, planning to inspect the LLM interface and method signatures to ensure proper task handling.


python3 - <<'PY'
import inspect
from vllm import LLM
sig = inspect.signature(LLM.generate)
print(sig)
PY


Output:
Exit code: 124
Wall time: 10.1 seconds
Output:
command timed out after 10095 milliseconds
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-15 00:00:35.311046: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(self, prompts: str | vllm.inputs.data.TextPrompt | vllm.inputs.data.TokensPrompt | vllm.inputs.data.EmbedsPrompt | vllm.inputs.data.ExplicitEncoderDecoderPrompt | collections.abc.Sequence[str | vllm.inputs.data.TextPrompt | vllm.inputs.data.TokensPrompt | vllm.inputs.data.EmbedsPrompt | vllm.inputs.data.ExplicitEncoderDecoderPrompt], sampling_params: vllm.sampling_params.SamplingParams | collections.abc.Sequence[vllm.sampling_params.SamplingParams] | None = None, *, use_tqdm: bool | collections.abc.Callable[..., tqdm.asyncio.tqdm_asyncio] = True, lora_request: list[vllm.lora.request.LoRARequest] | vllm.lora.request.LoRARequest | None = None, priority: list[int] | None = None) -> list[vllm.outputs.RequestOutput]


**Noting generate output type**


python3 - <<'PY'
import inspect
from vllm import LLM
print('transcribe' in dir(LLM))
if 'transcribe' in dir(LLM):
    print(inspect.signature(LLM.transcribe))
PY


Output:
Exit code: 124
Wall time: 10.1 seconds
Output:
command timed out after 10099 milliseconds
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-15 00:00:48.932768: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
False


**Checking language support keys**

I'm verifying if the gemma3n model's supported_languages uses the "hi" code or a different key to ensure the Hindi language is properly recognized and handled during transcription.


python3 - <<'PY'
from vllm.model_executor.models.gemma3n_mm import Gemma3nForConditionalGeneration
print('hi' in Gemma3nForConditionalGeneration.supported_languages)
print(Gemma3nForConditionalGeneration.supported_languages.get('hi'))
print('hin' in Gemma3nForConditionalGeneration.supported_languages)
PY


Output:
Exit code: 124
Wall time: 10.1 seconds
Output:
command timed out after 10103 milliseconds
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-15 00:01:04.579494: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
True
Hindi
False


**Analyzing output length versus reference**

I'm exploring if the high WER is due to output length exceeding references significantly, possibly over 100% longer, which would inflate insertion errors and WER; next, I'll compute word count distributions and ratios to confirm.


python3 - <<'PY'
import json
from statistics import mean

def load_refs(path):
    refs = {}
    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            obj = json.loads(line)
            refs[obj['segment_id']] = obj['text']
    return refs

refs = load_refs('data/bench/maya_hf_hin_v2/refs.jsonl')

hyp_lens = []
ref_lens = []
extra = []
missing = []

with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj = json.loads(line)
        seg = obj['segment_id']
        hyp = obj['final']['text']
        ref = refs.get(seg)
        if ref is None:
            missing.append(seg)
            continue
        hyp_lens.append(len(hyp.split()))
        ref_lens.append(len(ref.split()))
        extra.append(len(hyp.split()) - len(ref.split()))

print('pairs', len(hyp_lens), 'missing', len(missing))
print('avg hyp len', mean(hyp_lens), 'avg ref len', mean(ref_lens))
print('avg delta', mean(extra))
print('median hyp len', sorted(hyp_lens)[len(hyp_lens)//2])
print('median ref len', sorted(ref_lens)[len(ref_lens)//2])
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
pairs 200 missing 0
avg hyp len 23.77 avg ref len 11.46
avg delta 12.31
median hyp len 12
median ref len 12


**Analyzing output length distribution**

The mean hypothesis length is much higher than the reference mean despite identical medians, indicating wide output length variation with extreme outliers causing a high WER due to many inserted words. I'm preparing to identify and inspect the top 10 longest outputs.


python3 - <<'PY'
import json
refs = {}
with open('data/bench/maya_hf_hin_v2/refs.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        obj = json.loads(line)
        refs[obj['segment_id']] = obj['text']

items = []
with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj = json.loads(line)
        seg = obj['segment_id']
        hyp = obj['final']['text']
        ref = refs.get(seg, '')
        items.append((len(hyp.split()), len(ref.split()), seg, hyp, ref))

items.sort(reverse=True)
for hl, rl, seg, hyp, ref in items[:5]:
    print('---', seg)
    print('hyp_len', hl, 'ref_len', rl)
    print('ref:', ref)
    print('hyp:', hyp)
    print()
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
--- Kathbath__hindi__valid__2909__844424933549595-1179-m.m4a-14a42d93e8430a34
hyp_len 1022 ref_len 9
ref: हज़रत शेख़ जीलानी के पिता सय्यद वंशावली से थे
hyp: हाय हजरात से एक जिला नी के पिता सैयद वंशवाली से चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार चार

--- Kathbath__hindi__valid__509__844424931069960-326-f.m4a-2a91d13074192a5e
hyp_len 516 ref_len 12
ref: आप हर स्वाद के लिए विभिन्न संगीत की एक विशाल श्रेणी होगा
hyp: आप हर स्वाद के लिए विभिन्न संगीत की एक विशाल श्रेणी चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण

--- Kathbath__hindi__valid__1731__844424932913788-888-f.m4a-9b2c9188acd14a54
hyp_len 349 ref_len 14
ref: शरीर पर बढ़ा हुआ फैट कई तरह की समस्याओं का कारण बन सकता है
hyp: शरीर में बढ़ा हुआ फैट कई तरह की समस्याओं का कारण बन सकता है चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी चर्बी

--- Kathbath__hindi__valid__3034__844424933485998-38-m.m4a-d9605c733d4d50c8
hyp_len 347 ref_len 9
ref: द्वितीय चरण में सेना का आकार तथा उसकी बनावट
hyp: द्वितीय चरण में सेना का आकार तथा उसकी बनावट चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण चण च

--- Kathbath__hindi__valid__442__844424930984291-252-f.m4a-5d16e1505e5debec
hyp_len 262 ref_len 9
ref: मोईन अली के बल्ले से निकला एक बेहतरीन चौका
hyp: मोइन अली के बल्ले से निकला एक बेहतरीन चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का चक्का


Summary: auto


**Identifying missing stop tokens causing infinite repetition**

The output repeats tokens indefinitely likely because stop tokens or end-of-sequence detection aren't properly configured, causing the model to keep generating without halting. Next, I'll verify and set appropriate stop sequences or shorter max token limits to prevent looping.


**Planning to limit max tokens and add stop sequences**

Setting a lower max token limit (128–256) and specifying stop sequences like "<end_of_turn>" may prevent repeated output loops and reduce WER. I'll test these tweaks on a smaller subset before running the full benchmark to confirm improvement.


python3 - <<'PY'
import json
lens=[]
with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj=json.loads(line)
        lens.append(len(obj['final']['text'].split()))

for thr in [50,100,200,500]:
    print('>=',thr, sum(1 for l in lens if l>=thr))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
>= 50 5
>= 100 5
>= 200 5
>= 500 2


**Assessing WER distribution**

I've noted that the overall WER of 137% likely means many segments have errors, not just the few very long outputs. I'll check the eval.json results to analyze WER distribution and median values for clearer error insight.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 1.3760242396492393,
  "cer_mean": 1.0522220838587564,
  "wer_p50": 0.1111111111111111,
  "wer_p90": 0.375,
  "wer_p95": 0.5333333333333333,
  "cer_p50": 0.0425531914893617,
  "cer_p90": 0.16666666666666666,
  "cer_p95": 0.22727272727272727,
  "segment_error_rate": 0.72,
  "pct_wer_gt_5pct": 0.72,
  "pct_wer_gt_10pct": 0.525,
  "wer_by_language_mean": {
    "hin": 1.3760242396492393
  },
  "wer_by_decision_mean": {
    "accept": 1.651370648815905,
    "human": 0.777255064159826
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.176010101010101,
    "10-30s": 0.2831959706959707,
    "3-10s": 1.4480597859587216
  },
  "cer_by_language_mean": {
    "hin": 1.0522220838587564
  },
  "cer_by_decision_mean": {
    "accept": 1.3648066814614563,
    "human": 0.3724746255798699
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.10565015479876161,
    "10-30s": 0.0722371483484663,
    "3-10s": 1.1140633987540878
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 1.3760242396492393,
    "selection_gap_wer_mean": 0.0,
    "selection_gap_wer_p90": 0.0
  }
}


**Analyzing WER to adjust max tokens**

The median WER looks reasonable, but the mean is skewed by extreme outliers, especially in mid-length segments causing overrun and hallucination loops. Setting a max_tokens cap around 128 may help prevent degenerate repeated outputs and reduce WER without needing a repetition penalty, which conflicts with ASR determinism. Next, I'll explore lowering max_tokens in the transcription ASR code for the gemma model.


**Planning model-specific max_tokens and stop adjustment**

Deciding to reduce max_tokens to 128 or 256 specifically for gemma-3n in VllmTranscriptionASR and add a stop sequence to limit output. Next, I'll verify if SamplingParams supports a stop parameter by inspecting its definition in the vllm sampling_params module.


rg -n "class SamplingParams" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/sampling_params.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
111:class SamplingParams(


sed -n '111,220p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/sampling_params.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
class SamplingParams(
    PydanticMsgspecMixin,
    msgspec.Struct,
    omit_defaults=True,  # type: ignore[call-arg]
    # required for @cached_property.
    dict=True,
):  # type: ignore[call-arg]
    """Sampling parameters for text generation.

    Overall, we follow the sampling parameters from the OpenAI text completion
    API (https://platform.openai.com/docs/api-reference/completions/create).
    In addition, we support beam search, which is not supported by OpenAI.
    """

    n: int = 1
    """Number of outputs to return for the given prompt request.

    NOTE:
        `AsyncLLM` streams outputs by default. When `n > 1`, all `n` outputs
        are generated and streamed cumulatively per request. To see all `n`
        outputs upon completion, use `output_kind=RequestOutputKind.FINAL_ONLY`
        in `SamplingParams`."""
    presence_penalty: float = 0.0
    """Penalizes new tokens based on whether they appear in the generated text
    so far. Values > 0 encourage the model to use new tokens, while values < 0
    encourage the model to repeat tokens."""
    frequency_penalty: float = 0.0
    """Penalizes new tokens based on their frequency in the generated text so
    far. Values > 0 encourage the model to use new tokens, while values < 0
    encourage the model to repeat tokens."""
    repetition_penalty: float = 1.0
    """Penalizes new tokens based on whether they appear in the prompt and the
    generated text so far. Values > 1 encourage the model to use new tokens,
    while values < 1 encourage the model to repeat tokens."""
    temperature: float = 1.0
    """Controls the randomness of the sampling. Lower values make the model
    more deterministic, while higher values make the model more random. Zero
    means greedy sampling."""
    top_p: float = 1.0
    """Controls the cumulative probability of the top tokens to consider. Must
    be in (0, 1]. Set to 1 to consider all tokens."""
    top_k: int = 0
    """Controls the number of top tokens to consider. Set to 0 (or -1) to
    consider all tokens."""
    min_p: float = 0.0
    """Represents the minimum probability for a token to be considered,
    relative to the probability of the most likely token. Must be in [0, 1].
    Set to 0 to disable this."""
    seed: int | None = None
    """Random seed to use for the generation."""
    stop: str | list[str] | None = None
    """String(s) that stop the generation when they are generated. The returned
    output will not contain the stop strings."""
    stop_token_ids: list[int] | None = None
    """Token IDs that stop the generation when they are generated. The returned
    output will contain the stop tokens unless the stop tokens are special
    tokens."""
    ignore_eos: bool = False
    """Whether to ignore the EOS token and continue generating
    tokens after the EOS token is generated."""
    max_tokens: int | None = 16
    """Maximum number of tokens to generate per output sequence."""
    min_tokens: int = 0
    """Minimum number of tokens to generate per output sequence before EOS or
    `stop_token_ids` can be generated"""
    logprobs: int | None = None
    """Number of log probabilities to return per output token. When set to
    `None`, no probability is returned. If set to a non-`None` value, the
    result includes the log probabilities of the specified number of most
    likely tokens, as well as the chosen tokens. Note that the implementation
    follows the OpenAI API: The API will always return the log probability of
    the sampled token, so there may be up to `logprobs+1` elements in the
    response. When set to -1, return all `vocab_size` log probabilities."""
    prompt_logprobs: int | None = None
    """Number of log probabilities to return per prompt token.
    When set to -1, return all `vocab_size` log probabilities."""
    flat_logprobs: bool = False
    """Whether to return logprobs in flatten format (i.e. FlatLogprob)
    for better performance.
    NOTE: GC costs of FlatLogprobs is significantly smaller than
    list[dict[int, Logprob]]. After enabled, PromptLogprobs and
    SampleLogprobs would populated as FlatLogprobs."""
    # NOTE: This parameter is only exposed at the engine level for now.
    # It is not exposed in the OpenAI API server, as the OpenAI API does
    # not support returning only a list of token IDs.
    detokenize: bool = True
    """Whether to detokenize the output."""
    skip_special_tokens: bool = True
    """Whether to skip special tokens in the output."""
    spaces_between_special_tokens: bool = True
    """Whether to add spaces between special tokens in the output."""
    # `list[LogitsProcessor] | None` type. We use Any here because
    # `list[LogitsProcessor] | None` type is not supported by msgspec.
    logits_processors: Any | None = None
    """Functions that modify logits based on previously generated tokens, and
    optionally prompt tokens as a first argument."""
    include_stop_str_in_output: bool = False
    """Whether to include the stop strings in output text."""
    truncate_prompt_tokens: Annotated[int, msgspec.Meta(ge=-1)] | None = None
    """If set to -1, will use the truncation size supported by the model. If
    set to an integer k, will use only the last k tokens from the prompt
    (i.e., left truncation). If set to `None`, truncation is disabled."""
    output_kind: RequestOutputKind = RequestOutputKind.CUMULATIVE

    # The below fields are not supposed to be used as an input.
    # They are set in post_init.
    output_text_buffer_length: int = 0
    _all_stop_token_ids: set[int] = msgspec.field(default_factory=set)

    # Fields used to construct logits processors


**Considering stop strings and penalties to prevent loops**

Setting stop strings to "<end_of_turn>" and using repetition or frequency penalties could reduce degenerative loops in generation. Lowering max tokens to around 256 may also help avoid overly long, looping outputs without harming accuracy since transcripts are usually short.


**Planning WER analysis with truncated outputs**

I'm preparing to compute WER by truncating outputs to 50–64 tokens to see if shorter outputs reduce errors, which will help isolate if long-generation loops drive high WER. Next, I’ll inspect existing evaluation scripts to integrate this truncated WER calculation efficiently.


rg -n "wer|cer|jiwer|edit" -S maya_transcribe


Output:
Exit code: 0
Wall time: 0 seconds
Output:
maya_transcribe/asr.py:254:                    # Whisper's newer generate() prefers forced ids via generation_config.
maya_transcribe/asr.py:305:        self.model_map = {str(k).strip().lower(): str(v).strip() for k, v in (model_map or {}).items() if str(v).strip()}
maya_transcribe/asr.py:314:            key = language.strip().lower()
maya_transcribe/asr.py:356:    vLLM-powered speech-to-text backend for multimodal audio models (e.g., Voxtral, Gemma3n).
maya_transcribe/asr.py:401:        if limit_mm_per_prompt is None and "gemma-3n" in model_id.lower():
maya_transcribe/asr.py:497:    code = lang.strip().lower()
maya_transcribe/asr.py:781:        self.model_map = {str(k).strip().lower(): str(v).strip() for k, v in (model_map or self.DEFAULT_MODEL_MAP).items() if str(v).strip()}
maya_transcribe/asr.py:789:            key = language.strip().lower()
maya_transcribe/audio.py:125:        # https://ffmpeg.org/ffmpeg-filters.html#silenceremove
maya_transcribe/audio.py:128:            f"silenceremove=start_periods=1:start_duration={min_silence_sec}:start_threshold={thr}:"
maya_transcribe/bench.py:206:    target_langs = [l.strip().lower() for l in target_langs if l.strip()]
maya_transcribe/pipeline.py:54:    parts = [p.strip().lower() for p in (relpath or "").split("/") if p.strip()]
maya_transcribe/ctc_fit.py:33:    code = lang.strip().lower()
maya_transcribe/align.py:15:    Returns edit distance and an alignment as (ref_token|None, hyp_token|None) pairs.
maya_transcribe/align.py:74:def edit_distance(a: list[str], b: list[str]) -> int:
maya_transcribe/cli.py:88:    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
maya_transcribe/cli.py:246:    trim_silence: bool = typer.Option(True, help="Trim leading/trailing silence (ffmpeg silenceremove)."),
maya_transcribe/cli.py:326:    Compute WER/CER for a hypothesis against a reference JSONL keyed by `segment_id`.
maya_transcribe/cli.py:408:        False, help="Trim leading/trailing silence (ffmpeg silenceremove). Off by default for benchmarking."
maya_transcribe/rover.py:6:from maya_transcribe.align import align_tokens, edit_distance
maya_transcribe/rover.py:30:    return float(edit_distance(tokens_a, tokens_b)) / float(denom)
maya_transcribe/eval.py:8:from jiwer import cer, wer
maya_transcribe/eval.py:23:    wer: float
maya_transcribe/eval.py:24:    cer: float
maya_transcribe/eval.py:25:    oracle_best_wer: float | None = None
maya_transcribe/eval.py:26:    oracle_best_cer: float | None = None
maya_transcribe/eval.py:28:    selection_gap_wer: float | None = None
maya_transcribe/eval.py:68:        decision_filter_set = {str(d).strip().lower() for d in decision_filter if str(d).strip()}
maya_transcribe/eval.py:105:            dec_norm = str(decision or "").strip().lower()
maya_transcribe/eval.py:120:        w = float(wer(ref_norm, hyp_norm))
maya_transcribe/eval.py:121:        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))
maya_transcribe/eval.py:123:        oracle_best_wer = None
maya_transcribe/eval.py:124:        oracle_best_cer = None
maya_transcribe/eval.py:126:        selection_gap_wer = None
maya_transcribe/eval.py:129:            cand_wer: dict[str, float] = {}
maya_transcribe/eval.py:130:            cand_cer: dict[str, float] = {}
maya_transcribe/eval.py:134:                cand_wer[key] = float(wer(ref_n, hyp_n))
maya_transcribe/eval.py:135:                cand_cer[key] = float(cer(ref_n.replace(" ", ""), hyp_n.replace(" ", "")))
maya_transcribe/eval.py:137:            if cand_wer:
maya_transcribe/eval.py:138:                oracle_best_key, oracle_best_wer = min(cand_wer.items(), key=lambda kv: kv[1])
maya_transcribe/eval.py:139:                oracle_best_cer = cand_cer.get(oracle_best_key)
maya_transcribe/eval.py:140:                selection_gap_wer = float(w) - float(oracle_best_wer)
maya_transcribe/eval.py:151:                wer=w,
maya_transcribe/eval.py:152:                cer=c,
maya_transcribe/eval.py:153:                oracle_best_wer=oracle_best_wer,
maya_transcribe/eval.py:154:                oracle_best_cer=oracle_best_cer,
maya_transcribe/eval.py:156:                selection_gap_wer=selection_gap_wer,
maya_transcribe/eval.py:163:    by_lang_cer: dict[str, list[float]] = {}
maya_transcribe/eval.py:164:    by_dec_cer: dict[str, list[float]] = {}
maya_transcribe/eval.py:165:    by_bucket_cer: dict[str, list[float]] = {}
maya_transcribe/eval.py:167:    oracle_best_wers: list[float] = []
maya_transcribe/eval.py:170:            by_lang.setdefault(row.language, []).append(row.wer)
maya_transcribe/eval.py:171:            by_lang_cer.setdefault(row.language, []).append(row.cer)
maya_transcribe/eval.py:173:            by_dec.setdefault(row.decision, []).append(row.wer)
maya_transcribe/eval.py:174:            by_dec_cer.setdefault(row.decision, []).append(row.cer)
maya_transcribe/eval.py:176:            by_bucket.setdefault(row.duration_bucket, []).append(row.wer)
maya_transcribe/eval.py:177:            by_bucket_cer.setdefault(row.duration_bucket, []).append(row.cer)
maya_transcribe/eval.py:178:        if row.selection_gap_wer is not None:
maya_transcribe/eval.py:179:            selection_gaps.append(float(row.selection_gap_wer))
maya_transcribe/eval.py:180:        if row.oracle_best_wer is not None:
maya_transcribe/eval.py:181:            oracle_best_wers.append(float(row.oracle_best_wer))
maya_transcribe/eval.py:186:        "wer_mean": (sum(r.wer for r in rows) / len(rows)) if rows else None,
maya_transcribe/eval.py:187:        "cer_mean": (sum(r.cer for r in rows) / len(rows)) if rows else None,
maya_transcribe/eval.py:188:        "wer_p50": _percentile([r.wer for r in rows], 0.50),
maya_transcribe/eval.py:189:        "wer_p90": _percentile([r.wer for r in rows], 0.90),
maya_transcribe/eval.py:190:        "wer_p95": _percentile([r.wer for r in rows], 0.95),
maya_transcribe/eval.py:191:        "cer_p50": _percentile([r.cer for r in rows], 0.50),
maya_transcribe/eval.py:192:        "cer_p90": _percentile([r.cer for r in rows], 0.90),
maya_transcribe/eval.py:193:        "cer_p95": _percentile([r.cer for r in rows], 0.95),
maya_transcribe/eval.py:194:        "segment_error_rate": (sum(1 for r in rows if r.wer > 0.0) / len(rows)) if rows else None,
maya_transcribe/eval.py:195:        "pct_wer_gt_5pct": (sum(1 for r in rows if r.wer > 0.05) / len(rows)) if rows else None,
maya_transcribe/eval.py:196:        "pct_wer_gt_10pct": (sum(1 for r in rows if r.wer > 0.10) / len(rows)) if rows else None,
maya_transcribe/eval.py:197:        "wer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang.items())},
maya_transcribe/eval.py:198:        "wer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec.items())},
maya_transcribe/eval.py:199:        "wer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket.items())},
maya_transcribe/eval.py:200:        "cer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang_cer.items())},
maya_transcribe/eval.py:201:        "cer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec_cer.items())},
maya_transcribe/eval.py:202:        "cer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket_cer.items())},
maya_transcribe/eval.py:205:            "oracle_best_wer_mean": (sum(oracle_best_wers) / len(oracle_best_wers)) if oracle_best_wers else None,
maya_transcribe/eval.py:206:            "selection_gap_wer_mean": (sum(selection_gaps) / len(selection_gaps)) if selection_gaps else None,
maya_transcribe/eval.py:207:            "selection_gap_wer_p90": _percentile(selection_gaps, 0.90),
maya_transcribe/verify.py:81:            reasons.append("lid_uncertain")
maya_transcribe/gemini_asr.py:29:    ext = path.suffix.lower().lstrip(".")
maya_transcribe/gemini_asr.py:116:    lowered = "\n".join(lines).lower()
maya_transcribe/gemini_asr.py:117:    extra_outputs_present = ("output 2" in lowered) or ("output 3" in lowered) or ("output 4" in lowered)
maya_transcribe/gemini_asr.py:123:        return line.lower().startswith("output ")
maya_transcribe/gemini_asr.py:127:        if ln.lower().replace(" ", "") == "output1:transcript":
maya_transcribe/textnorm.py:21:    - Lowercase for English only
maya_transcribe/textnorm.py:53:    if (lang or "").lower() in {"en", "eng"}:
maya_transcribe/textnorm.py:54:        norm = norm.lower()
maya_transcribe/language.py:103:    return LANG_CODE_TO_SCRIPT.get(lang_code.strip().lower())
maya_transcribe/language.py:161:    lang_code = lang_code.strip().lower()
maya_transcribe/language.py:170:    lang_code = lang_code.strip().lower()
maya_transcribe/language.py:180:    lang_code_norm = lang_code.strip().lower()


**Planning quick WER evaluation script**

I'm preparing to implement a script that truncates hypothesis outputs (e.g., at 50 or 128 tokens) and computes WER with jiwer to assess the impact of decoding length on errors.


python3 - <<'PY'
import json
from jiwer import wer
from maya_transcribe.textnorm import normalize_text

refs = {}
with open('data/bench/maya_hf_hin_v2/refs.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        obj = json.loads(line)
        refs[obj['segment_id']] = obj['text']

scores = []
for max_words in [32, 64, 128]:
    scores.clear()
    with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
        for line in f:
            obj = json.loads(line)
            seg = obj['segment_id']
            ref = refs.get(seg)
            hyp = obj['final']['text']
            if ref is None:
                continue
            hyp_words = hyp.split()
            if len(hyp_words) > max_words:
                hyp = ' '.join(hyp_words[:max_words])
            ref_n = normalize_text(ref, language='hin')
            hyp_n = normalize_text(hyp, language='hin')
            scores.append(float(wer(ref_n, hyp_n)))
    print('max_words', max_words, 'wer_mean', sum(scores)/len(scores))
PY


Output:
Exit code: 1
Wall time: 0.1 seconds
Output:
Traceback (most recent call last):
  File "<stdin>", line 25, in <module>
TypeError: normalize_text() got an unexpected keyword argument 'language'


**Checking text normalization signature**


python3 - <<'PY'
import inspect
from maya_transcribe.textnorm import normalize_text
print(inspect.signature(normalize_text))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
(text: 'str', lang: 'str | None' = None) -> 'str'


**Adjusting to use lang parameter**


python3 - <<'PY'
import json
from jiwer import wer
from maya_transcribe.textnorm import normalize_text

refs = {}
with open('data/bench/maya_hf_hin_v2/refs.jsonl', 'r', encoding='utf-8') as f:
    for line in f:
        obj = json.loads(line)
        refs[obj['segment_id']] = obj['text']

for max_words in [32, 64, 128]:
    scores = []
    with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
        for line in f:
            obj = json.loads(line)
            seg = obj['segment_id']
            ref = refs.get(seg)
            hyp = obj['final']['text']
            if ref is None:
                continue
            hyp_words = hyp.split()
            if len(hyp_words) > max_words:
                hyp = ' '.join(hyp_words[:max_words])
            ref_n = normalize_text(ref, lang='hin')
            hyp_n = normalize_text(hyp, lang='hin')
            scores.append(float(wer(ref_n, hyp_n)))
    print('max_words', max_words, 'wer_mean', sum(scores)/len(scores))
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
max_words 32 wer_mean 0.20836550949050953
max_words 64 wer_mean 0.2864607475857477
max_words 128 wer_mean 0.4426512237762239


Summary: auto


**Analyzing decoding and input configuration issues**

Truncating input reduces WER drastically, revealing looping as a major error source, but WER remains high (21%), indicating Gemma still trails Voxtral. Possible causes include mismatch in expected audio length or sampling parameters, or improper prompt handling in generation that might exclude audio tokens. Next, I’ll focus on fixing decoding to prevent loops and verify prompt and audio input configurations.


**Planning decoding parameter tweaks for gemma**

Identified degenerative loops likely cause large WER spikes for gemma, so I'll add stop tokens like "<end_of_turn>", lower max_tokens to 128, and possibly increase repetition penalty specifically for gemma to reduce output runaway. Next, I'll rerun benchmarks with these sampling adjustments and record results alongside Voxtral baselines in a results file for clear comparison.


python3 - <<'PY'
import json
from statistics import mean
refs = {}
with open('data/bench/maya_hf_hin_v2/refs.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj=json.loads(line)
        refs[obj['segment_id']] = obj

long = []
with open('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj=json.loads(line)
        hyp_len = len(obj['final']['text'].split())
        if hyp_len >= 100:
            seg=obj['segment_id']
            ref = refs.get(seg, {})
            long.append((seg, hyp_len, ref.get('duration_sec')))

print('count', len(long))
print('durations', [d for _,_,d in long])
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
count 5
durations [4.1099375, 5.3870625, 3.784875, 3.575875, 4.4118125]


**Planning dynamic max_tokens per segment**

Long outputs suggest possible model looping, so I'll compute max_tokens based on audio duration (e.g., 6 tokens/sec) and pass it dynamically via per-call SamplingParams to better limit output length and reduce repetition.


**Analyzing token-rate heuristics for dynamic max_tokens**

I’m refining a heuristic to set dynamic max_tokens based on audio duration, targeting about 6–12 tokens per second capped around 512 tokens. This aims to prevent Gemma’s looping issues and high WER by clamping output length, while keeping flexibility to adjust per model without degrading Voxtral.


python3 - <<'PY'
import json
import statistics

wps=[]
with open('data/bench/maya_hf_hin_v2/refs.jsonl','r',encoding='utf-8') as f:
    for line in f:
        obj=json.loads(line)
        words=len(obj['text'].split())
        dur=obj['duration_sec']
        if dur and dur>0:
            wps.append(words/dur)

print('wps mean', statistics.mean(wps))
print('wps p90', statistics.quantiles(wps, n=10)[8])
print('wps max', max(wps))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
wps mean 2.1920800264136524
wps p90 3.0470101768013977
wps max 4.400958780305709


**Planning dynamic token limit based on audio duration**

I'm deciding to modify the ASR class to support dynamic max tokens computed per audio length, defaulting to a safe estimate but allowing an explicit override. I'll rename the parameter to clarify its purpose and handle dynamic calculation when it's unset.


**Designing model-specific dynamic token limits and stop tokens**

I'm planning to store an optional max tokens attribute that, if unset, computes tokens dynamically based on audio duration with a safe tokens-per-second rate. For the Gemma model, I’ll apply specialized constraints: a max token cap, a stop sequence, and optionally a repetition penalty to reduce looping while preserving accuracy. Implementing stop token IDs from the tokenizer seems necessary for reliable stopping, so I’ll inspect how to access tokenizer and token IDs in the vLLM wrapper.


rg -n "class LLM" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
92:class LLM:


sed -n '92,200p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
class LLM:
    """An LLM for generating texts from given prompts and sampling parameters.

    This class includes a tokenizer, a language model (possibly distributed
    across multiple GPUs), and GPU memory space allocated for intermediate
    states (aka KV cache). Given a batch of prompts and sampling parameters,
    this class generates texts from the model, using an intelligent batching
    mechanism and efficient memory management.

    Args:
        model: The name or path of a HuggingFace Transformers model.
        tokenizer: The name or path of a HuggingFace Transformers tokenizer.
        tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer
            if available, and "slow" will always use the slow tokenizer.
        skip_tokenizer_init: If true, skip initialization of tokenizer and
            detokenizer. Expect valid prompt_token_ids and None for prompt
            from the input.
        trust_remote_code: Trust remote code (e.g., from HuggingFace) when
            downloading the model and tokenizer.
        allowed_local_media_path: Allowing API requests to read local images
            or videos from directories specified by the server file system.
            This is a security risk. Should only be enabled in trusted
            environments.
        allowed_media_domains: If set, only media URLs that belong to this
            domain can be used for multi-modal inputs.
        tensor_parallel_size: The number of GPUs to use for distributed
            execution with tensor parallelism.
        dtype: The data type for the model weights and activations. Currently,
            we support `float32`, `float16`, and `bfloat16`. If `auto`, we use
            the `dtype` attribute of the Transformers model's config. However,
            if the `dtype` in the config is `float32`, we will use `float16` instead.
        quantization: The method used to quantize the model weights. Currently,
            we support "awq", "gptq", and "fp8" (experimental).
            If None, we first check the `quantization_config` attribute in the
            model config file. If that is None, we assume the model weights are
            not quantized and use `dtype` to determine the data type of
            the weights.
        revision: The specific model version to use. It can be a branch name,
            a tag name, or a commit id.
        tokenizer_revision: The specific tokenizer version to use. It can be a
            branch name, a tag name, or a commit id.
        seed: The seed to initialize the random number generator for sampling.
        gpu_memory_utilization: The ratio (between 0 and 1) of GPU memory to
            reserve for the model weights, activations, and KV cache. Higher
            values will increase the KV cache size and thus improve the model's
            throughput. However, if the value is too high, it may cause out-of-
            memory (OOM) errors.
        kv_cache_memory_bytes: Size of KV Cache per GPU in bytes. By default,
            this is set to None and vllm can automatically infer the kv cache
            size based on gpu_memory_utilization. However, users may want to
            manually specify the kv cache memory size. kv_cache_memory_bytes
            allows more fine-grain control of how much memory gets used when
            compared with using gpu_memory_utilization. Note that
            kv_cache_memory_bytes (when not-None) ignores
            gpu_memory_utilization
        swap_space: The size (GiB) of CPU memory per GPU to use as swap space.
            This can be used for temporarily storing the states of the requests
            when their `best_of` sampling parameters are larger than 1. If all
            requests will have `best_of=1`, you can safely set this to 0.
            Noting that `best_of` is only supported in V0. Otherwise, too small
            values may cause out-of-memory (OOM) errors.
        cpu_offload_gb: The size (GiB) of CPU memory to use for offloading
            the model weights. This virtually increases the GPU memory space
            you can use to hold the model weights, at the cost of CPU-GPU data
            transfer for every forward pass.
        enforce_eager: Whether to enforce eager execution. If True, we will
            disable CUDA graph and always execute the model in eager mode.
            If False, we will use CUDA graph and eager execution in hybrid.
        disable_custom_all_reduce: See
            [ParallelConfig][vllm.config.ParallelConfig].
        hf_token: The token to use as HTTP bearer authorization for remote files
            . If `True`, will use the token generated when running
            `huggingface-cli login` (stored in `~/.huggingface`).
        hf_overrides: If a dictionary, contains arguments to be forwarded to the
            HuggingFace config. If a callable, it is called to update the
            HuggingFace config.
        mm_processor_kwargs: Arguments to be forwarded to the model's processor
            for multi-modal data, e.g., image processor. Overrides for the
            multi-modal processor obtained from `AutoProcessor.from_pretrained`.
            The available overrides depend on the model that is being run.
            For example, for Phi-3-Vision: `{"num_crops": 4}`.
        pooler_config: Initialize non-default pooling config for the pooling
            model. e.g. `PoolerConfig(pooling_type="mean", normalize=False)`.
        compilation_config: Either an integer or a dictionary. If it is an
            integer, it is used as the mode of compilation optimization. If it
            is a dictionary, it can specify the full compilation configuration.
        **kwargs: Arguments for [`EngineArgs`][vllm.EngineArgs].

    Note:
        This class is intended to be used for offline inference. For online
        serving, use the [AsyncLLMEngine][vllm.AsyncLLMEngine] class instead.
    """

    def __init__(
        self,
        model: str,
        *,
        runner: RunnerOption = "auto",
        convert: ConvertOption = "auto",
        tokenizer: str | None = None,
        tokenizer_mode: TokenizerMode | str = "auto",
        skip_tokenizer_init: bool = False,
        trust_remote_code: bool = False,
        allowed_local_media_path: str = "",
        allowed_media_domains: list[str] | None = None,
        tensor_parallel_size: int = 1,
        dtype: ModelDType = "auto",
        quantization: QuantizationMethods | None = None,
        revision: str | None = None,


rg -n "tokenizer" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/llm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
75:from vllm.tokenizers import TokenizerLike
76:from vllm.tokenizers.mistral import MistralTokenizer
95:    This class includes a tokenizer, a language model (possibly distributed
103:        tokenizer: The name or path of a HuggingFace Transformers tokenizer.
104:        tokenizer_mode: The tokenizer mode. "auto" will use the fast tokenizer
105:            if available, and "slow" will always use the slow tokenizer.
106:        skip_tokenizer_init: If true, skip initialization of tokenizer and
107:            detokenizer. Expect valid prompt_token_ids and None for prompt
110:            downloading the model and tokenizer.
131:        tokenizer_revision: The specific tokenizer version to use. It can be a
191:        tokenizer: str | None = None,
192:        tokenizer_mode: TokenizerMode | str = "auto",
193:        skip_tokenizer_init: bool = False,
201:        tokenizer_revision: str | None = None,
320:            tokenizer=tokenizer,
321:            tokenizer_mode=tokenizer_mode,
322:            skip_tokenizer_init=skip_tokenizer_init,
330:            tokenizer_revision=tokenizer_revision,
367:    def get_tokenizer(self) -> TokenizerLike:
368:        return self.llm_engine.get_tokenizer()
632:        tokenizer = self.get_tokenizer()
634:            tokenizer.eos_token_id,
677:                prompt_tokens = tokenizer.encode(prompt["prompt"])
760:                                    token_id == tokenizer.eos_token_id
780:                beam.text = tokenizer.decode(beam.tokens)
817:        tokenizer = self.get_tokenizer()
823:            tokenizer,
847:            if isinstance(tokenizer, MistralTokenizer):
849:                    tokenizer,
855:                    tokenizer=tokenizer,
861:                # should not be added by the tokenizer in this case.
862:                prompt_token_ids = tokenizer.encode(
900:        tokenizer and calls the [generate][vllm.LLM.generate] method to generate
1257:        tokenizer: TokenizerLike,
1283:            tokenizer=tokenizer, embed_1=encoded_output_1, embed_2=encoded_output_2
1291:        tokenizer: TokenizerLike,
1302:        if isinstance(tokenizer, MistralTokenizer):
1303:            raise ValueError("Score API is not supported for Mistral tokenizer")
1330:                tokenizer=tokenizer,
1425:        # the tokenizer for models such as
1428:        tokenizer = self.get_tokenizer()
1453:                        prompt = tokenizer.decode(
1487:                tokenizer,
1497:                tokenizer,


**Identifying stop token IDs for gemma-3n**

Noting that the gemma-3n tokenizer likely supports converting "<end_of_turn>" to token ID via `convert_tokens_to_ids` or `encode`, but multi-token outputs could complicate stop token handling; planning to implement cautious conversion with fallbacks to handle these cases safely.


**Considering stop string approach with special token handling**

Evaluating using stop strings like "<end_of_turn>" with `skip_special_tokens=False` to ensure the stop sequence is detectable, since special tokens might be removed otherwise; next step is to verify how vLLM applies stop strings relative to special token skipping.


rg -n "stop_token_ids|stop\s*=|include_stop_str_in_output" /home/ubuntu/.local/lib/python3.10/site-packages/vllm