Output:
Exit code: 0
Wall time: 10.6 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:31:29.369534: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
def get_decoder_dummy_data(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int] | None = None,
        mm_options: Mapping[str, BaseDummyOptions] | None = None,
    ) -> DummyDecoderData:
        mm_inputs = self._get_dummy_mm_inputs(seq_len, mm_counts, mm_options)

        prompt_token_ids = mm_inputs["prompt_token_ids"]
        total_len = len(prompt_token_ids)

        if total_len < seq_len:
            prompt_token_ids.extend([0] * (seq_len - total_len))

        return DummyDecoderData(
            prompt_token_ids=prompt_token_ids,
            multi_modal_data=mm_inputs["mm_kwargs"].require_data(),
            multi_modal_placeholders=mm_inputs["mm_placeholders"],
        )

    def get_mm_max_tokens(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int] | None = None,
    ) -> Mapping[str, int]:
        """
        Returns the maximum number of embeddings per item of each modality, excluding
        any break/text tokens in-between multimodal embeddings/encoder outputs.
        """
        if mm_counts is None:
            mm_counts = self.get_mm_limits()

        max_tokens_per_item = self.processing_info.get_mm_max_tokens_per_item(


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


Summary: auto


**Inspecting function definition**

I'm looking earlier in the file to find the definition of the `get_mm_limits` function to understand its behavior and usage.


python3 - <<'PY'
import inspect
from vllm.multimodal import profiling
src = inspect.getsource(profiling)
idx = src.find('def get_mm_limits')
print(src[idx:idx+800])
PY


Output:
Exit code: 0
Wall time: 11.6 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:31:51.992926: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
def get_mm_limits(self) -> Mapping[str, int]:
        return self.processor.allowed_mm_limits

    def _get_dummy_mm_inputs(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int] | None = None,
        mm_options: Mapping[str, BaseDummyOptions] | None = None,
    ) -> MultiModalInputs:
        if mm_counts is None:
            mm_counts = self.get_mm_limits()

        factory = self.dummy_inputs
        processor_inputs = factory.get_dummy_processor_inputs(
            seq_len, mm_counts, mm_options
        )

        return self.processor.apply(
            prompt=processor_inputs.prompt,
            mm_data=processor_inputs.mm_data,
            hf_processor_mm_kwargs=processor_inputs.hf_processor_mm_kwargs,
            tokenization_kwargs=processor_inputs.tokenization_


**Assessing multi-modal limit overriding**

I'm analyzing how to set image count to zero for gemma3n by overriding `allowed_mm_limits`, considering patching `Gemma3nProcessingInfo` or passing `mm_processor_kwargs` to LLM; due to complexity, modifying `gemma3n_mm.py` to force image modality off or using an environment variable might be simpler next steps.


**Considering audio-only Gemma3n via config**

I'm exploring if Gemma3n can run audio-only by passing `mm_processor_kwargs={"max_images":0}` without using transformers, and will check `allowed_mm_limits` in `processing.py` to verify if this is supported.


rg -n "allowed_mm_limits" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py -n


Output:
Exit code: 0
Wall time: 0 seconds
Output:
1200:    def get_allowed_mm_limits(self) -> Mapping[str, int]:
1303:        self._allowed_mm_limits = self.info.get_allowed_mm_limits()
1310:    def allowed_mm_limits(self):
1311:        return self._allowed_mm_limits
1341:        allowed_limit = self.allowed_mm_limits.get(modality, 0)


sed -n '1180,1235p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:

    def get_hf_processor(self, **kwargs: object) -> ProcessorMixin:
        """
        Subclasses can override this method to handle
        specific kwargs from model config or user inputs.
        """
        return self.ctx.get_hf_processor(**kwargs)

    @abstractmethod
    def get_supported_mm_limits(self) -> Mapping[str, int | None]:
        """
        Return the maximum supported number of items for each modality.

        A value of `None` means unlimited number of items.

        Omitting a modality from the returned dictionary means that
        it is not supported at all.
        """
        raise NotImplementedError

    def get_allowed_mm_limits(self) -> Mapping[str, int]:
        """Return the maximum allowed number of items for each modality."""
        supported_mm_limits = self.get_supported_mm_limits()
        mm_config = self.ctx.get_mm_config()

        allowed_limits = dict[str, int]()
        for modality, supported_limit in supported_mm_limits.items():
            user_limit = mm_config.get_limit_per_prompt(modality)

            allowed_limits[modality] = (
                user_limit
                if supported_limit is None
                else min(user_limit, supported_limit)
            )

        return allowed_limits

    def get_mm_max_tokens_per_item(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int],
    ) -> Mapping[str, int] | None:
        """
        Return the maximum number of tokens per item of for each modality.

        When `None` (the default) is returned, vLLM will generate dummy inputs
        (images/videos) at maximum possible sizes and process them to determine
        the maximum token count per modality.

        This approach works but can be very slow for certain models (e.g.,
        Qwen2.5-VL), leading to very long startup time. For better performance,
        each model can override this method to return pre-computed maximum token
        counts, avoiding the need for dummy input generation and processing.

        Note:
            The maximum number of tokens per item of each modality returned


**Inspecting multimodal config for prompt limits**

I'm checking the multimodal config module to find if environment variables or parameters exist that control per-prompt limits, focusing on mm_config and how it derives limits.


sed -n '1,200p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from collections.abc import Mapping
from typing import Any, Literal, TypeAlias

from pydantic import ConfigDict, Field, field_validator, model_validator
from pydantic.dataclasses import dataclass

from vllm.attention.backends.registry import AttentionBackendEnum
from vllm.config.utils import config
from vllm.utils.hashing import safe_hash

@dataclass
class BaseDummyOptions:
    """Base options for generating dummy data during profiling."""

    count: int = Field(999, ge=0)

@dataclass(config=ConfigDict(extra="forbid"))
class VideoDummyOptions(BaseDummyOptions):
    """Options for generating dummy video data during profiling."""

    num_frames: int | None = Field(None, gt=0)
    width: int | None = Field(None, gt=0)
    height: int | None = Field(None, gt=0)

@dataclass(config=ConfigDict(extra="forbid"))
class ImageDummyOptions(BaseDummyOptions):
    """Options for generating dummy image data during profiling."""

    width: int | None = Field(None, gt=0)
    height: int | None = Field(None, gt=0)

@dataclass(config=ConfigDict(extra="forbid"))
class AudioDummyOptions(BaseDummyOptions):
    """Options for generating dummy audio data during profiling."""

    length: int | None = Field(None, gt=0)

MMEncoderTPMode = Literal["weights", "data"]
MMCacheType = Literal["shm", "lru"]
DummyOptions: TypeAlias = (
    BaseDummyOptions | VideoDummyOptions | ImageDummyOptions | AudioDummyOptions
)

@config
@dataclass
class MultiModalConfig:
    """Controls the behavior of multimodal models."""

    limit_per_prompt: dict[str, DummyOptions] = Field(default_factory=dict)
    """The maximum number of input items and options allowed per
        prompt for each modality.
    Defaults to 999 for each modality.

    Legacy format (count only):
        {"image": 16, "video": 2}

    Configurable format (with options):
        {"video": {"count": 1, "num_frames": 32, "width": 512, "height": 512},
        "image": {"count": 5, "width": 512, "height": 512}}

    Mixed format (combining both):
        {"image": 16, "video": {"count": 1, "num_frames": 32, "width": 512,
        "height": 512}}
    """
    enable_mm_embeds: bool = False
    """If `True`, enables passing multimodal embeddings:
    for `LLM` class, this refers to tensor inputs under `multi_modal_data`;
    for the OpenAI-compatible server, this refers to chat messages with content
    `"type": "*_embeds"`.

    WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed.
    Only enable this flag for trusted users!"""
    media_io_kwargs: dict[str, dict[str, Any]] = Field(default_factory=dict)
    """Additional args passed to process media inputs, keyed by modalities.
    For example, to set num_frames for video, set
    `--media-io-kwargs '{"video": {"num_frames": 40} }'`"""
    mm_processor_kwargs: dict[str, object] | None = None
    """Arguments to be forwarded to the model's processor for multi-modal data,
    e.g., image processor. Overrides for the multi-modal processor obtained
    from `transformers.AutoProcessor.from_pretrained`.

    The available overrides depend on the model that is being run.

    For example, for Phi-3-Vision:
    `{"num_crops": 4}`."""
    mm_processor_cache_gb: float = Field(default=4, ge=0)
    """The size (in GiB) of the multi-modal processor cache, which is used to
    avoid re-processing past multi-modal inputs.

    This cache is duplicated for each API process and engine core process,
    resulting in a total memory usage of
    `mm_processor_cache_gb * (api_server_count + data_parallel_size)`.

    Set to `0` to disable this cache completely (not recommended)."""
    mm_processor_cache_type: MMCacheType = "lru"
    """Type of cache to use for the multi-modal preprocessor/mapper. If `shm`,
    use shared memory FIFO cache. If `lru`, use mirrored LRU cache."""
    mm_shm_cache_max_object_size_mb: int = Field(default=128, ge=0)
    """Size limit (in MiB) for each object stored in the multi-modal processor
    shared memory cache. Only effective when `mm_processor_cache_type` is
    `"shm"`."""
    mm_encoder_tp_mode: MMEncoderTPMode = "weights"
    """Indicates how to optimize multi-modal encoder inference using tensor
    parallelism (TP).

    - `"weights"`: Within the same vLLM engine, split the weights of
        each layer across TP ranks. (default TP behavior)\n
    - `"data"`: Within the same vLLM engine, split the batched input data
        across TP ranks to process the data in parallel, while hosting
        the full weights on each TP rank.
        This batch-level DP is not to be confused with API request-level
        DP (which is controlled by `--data-parallel-size`).
        This is only supported on a per-model basis and falls back to
        `"weights"` if the encoder does not support DP."""
    mm_encoder_attn_backend: AttentionBackendEnum | None = None
    """Optional override for the multi-modal encoder attention backend when
    using vision transformers. Accepts any value from
    `vllm.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`)."""
    interleave_mm_strings: bool = False
    """Enable fully interleaved support for multimodal prompts, while using
    --chat-template-content-format=string."""
    skip_mm_profiling: bool = False
    """When enabled, skips multimodal memory profiling and only profiles with
    language backbone model during engine initialization.

    This reduces engine startup time but shifts the responsibility to users for
    estimating the peak memory usage of the activation of multimodal encoder and
    embedding cache."""
    video_pruning_rate: float | None = Field(default=None, ge=0.0, lt=1.0)
    """Sets pruning rate for video pruning via Efficient Video Sampling.
    Value sits in range [0;1) and determines fraction of media tokens
    from each video to be pruned.
    """

    @field_validator("limit_per_prompt", mode="before")
    @classmethod
    def _validate_limit_per_prompt(
        cls, value: dict[str, int | dict[str, int]]
    ) -> dict[str, DummyOptions]:
        for k, v in value.items():
            # Handle legacy format where only count is specified
            if isinstance(v, int):
                v = {"count": v}
            # Convert to the appropriate DummyOptions subclass
            if k == "video":
                value[k] = VideoDummyOptions(**v)
            elif k == "image":
                value[k] = ImageDummyOptions(**v)
            elif k == "audio":
                value[k] = AudioDummyOptions(**v)
            else:
                value[k] = BaseDummyOptions(**v)
        return value

    @field_validator("mm_encoder_attn_backend", mode="before")
    @classmethod
    def _validate_mm_encoder_attn_backend(
        cls, value: str | AttentionBackendEnum | None
    ) -> AttentionBackendEnum | None:
        if isinstance(value, str) and value.upper() == "XFORMERS":
            raise ValueError(
                "Attention backend 'XFORMERS' has been removed (See PR #29262 for "
                "details). Please select a supported attention backend."
            )

        if value is None or isinstance(value, AttentionBackendEnum):
            return value

        assert isinstance(value, str), (
            "mm_encoder_attn_backend must be a string or an AttentionBackendEnum."
        )
        return AttentionBackendEnum[value.upper()]

    @model_validator(mode="after")
    def _validate_multimodal_config(self):
        if self.mm_processor_cache_type != "shm" and (
            self.mm_shm_cache_max_object_size_mb
            != MultiModalConfig.mm_shm_cache_max_object_size_mb
        ):
            raise ValueError(
                "'mm_shm_cache_max_object_size_mb' should only be set when "
                "'mm_processor_cache_type' is 'shm'."
            )
        return self

    def compute_hash(self) -> str:
        """
        WARNING: Whenever a new field is added to this config,
        ensure that it is included in the factors list if
        it affects the computation graph.


**Inspecting config options for multimodal limit**

I'm clarifying that the LLM constructor accepts `mm_processor_kwargs` but not directly `limit_per_prompt` for multimodal input limits, so next I'll check engine argument definitions and config files for parameters controlling multimodal limits like `limit_per_prompt` or similarly named flags.


rg -n "limit_per_prompt" /home/ubuntu/.local/lib/python3.10/site-packages/vllm -S


Output:
Exit code: 0
Wall time: 0 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py:1207:            user_limit = mm_config.get_limit_per_prompt(modality)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/engine/arg_utils.py:455:        MultiModalConfig, "limit_per_prompt"
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/engine/arg_utils.py:937:            "--limit-mm-per-prompt", **multimodal_kwargs["limit_per_prompt"]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:111:            for m in model_config.multimodal_config.limit_per_prompt
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:134:            mm_config.get_limit_per_prompt(modality) == 0
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/model.py:560:                limit_per_prompt=limit_mm_per_prompt,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:58:    limit_per_prompt: dict[str, DummyOptions] = Field(default_factory=dict)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:144:    @field_validator("limit_per_prompt", mode="before")
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:146:    def _validate_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:215:    def get_limit_per_prompt(self, modality: str) -> int:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:220:        limit_data = self.limit_per_prompt.get(modality)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/multimodal.py:233:        return self.limit_per_prompt.get(modality)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_vl.py:1274:        if multimodal_config.get_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_vl.py:1276:        ) or multimodal_config.get_limit_per_prompt("video"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/mistral3.py:472:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/step3_vl.py:945:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/hunyuan_vision.py:880:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/pixtral.py:399:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_vl.py:1279:        if not multimodal_config.get_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_vl.py:1281:        ) and not multimodal_config.get_limit_per_prompt("video"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_omni_thinker.py:835:        if multimodal_config.get_limit_per_prompt("audio"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_omni_thinker.py:840:        if multimodal_config.get_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_omni_thinker.py:842:        ) or multimodal_config.get_limit_per_prompt("video"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_vl_moe.py:427:        if not multimodal_config.get_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen3_vl_moe.py:429:        ) and not multimodal_config.get_limit_per_prompt("video"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/mllama4.py:768:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/opencua.py:242:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/keye.py:1121:        max_images = mm_config.get_limit_per_prompt("image")
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/keye.py:1122:        max_videos = mm_config.get_limit_per_prompt("video")
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/llava.py:555:        if multimodal_config.get_limit_per_prompt("image"):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py:1184:        if multimodal_config.get_limit_per_prompt(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/qwen2_5_vl.py:1186:        ) or multimodal_config.get_limit_per_prompt("video"):


**Confirming parameter usage for limit_mm_per_prompt**

I'm verifying that `limit_mm_per_prompt` is the correct parameter name used in ModelConfig and can be passed via LLM initialization kwargs by inspecting `vllm/config/model.py` and related usage in `vllm/engine/arg_utils.py`.


rg -n "limit_mm_per_prompt" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/config/model.py -n


Output:
Exit code: 0
Wall time: 0 seconds
Output:
287:    limit_mm_per_prompt: InitVar[dict[str, int | dict[str, int]] | None] = None
337:            "limit_mm_per_prompt",
401:        limit_mm_per_prompt: dict[str, int | dict[str, int]] | None,
560:                limit_per_prompt=limit_mm_per_prompt,


**Planning memory limit param**

I'm designing a `limit_mm_per_prompt` dict config to control memory use by modality in the LLM, adding it as an optional init param to `VllmTranscriptionASR` and passing it via `llm_kwargs`. The CLI will get a new option accepting a JSON file path for this config to easily disable image inputs and reduce OOM risk.


**Adding heuristic default for gemma3n**

I'm updating `VllmTranscriptionASR` to accept an optional `limit_mm_per_prompt` parameter that defaults to `{"image": 0, "audio": 1}` when the model ID includes "gemma-3n", disabling image tokens automatically without changing the CLI or pipeline. This keeps the change minimal while reducing memory for audio-only gemma usage.


Input:
*** Begin Patch
*** Update File: maya_transcribe/asr.py
@@
         tensor_parallel_size: int = 1,
         trust_remote_code: bool = True,
         max_tokens: int | None = None,
         gpu_memory_utilization: float | None = None,
         max_model_len: int | None = None,
+        limit_mm_per_prompt: dict[str, int | dict[str, int]] | None = None,
     ) -> None:
@@
-        llm_kwargs = {
+        if limit_mm_per_prompt is None and "gemma-3n" in model_id.lower():
+            limit_mm_per_prompt = {"image": 0, "audio": 1}
+
+        llm_kwargs = {
             "model": model_id,
             "tensor_parallel_size": self.tensor_parallel_size,
             "trust_remote_code": self.trust_remote_code,
         }
         llm_kwargs["dtype"] = dtype if dtype is not None else "auto"
         if gpu_memory_utilization is not None:
             llm_kwargs["gpu_memory_utilization"] = float(gpu_memory_utilization)
         if max_model_len is not None:
             llm_kwargs["max_model_len"] = int(max_model_len)
+        if limit_mm_per_prompt is not None:
+            llm_kwargs["limit_mm_per_prompt"] = limit_mm_per_prompt
         self._llm = LLM(**llm_kwargs)
*** End Patch


Output: {"output":"Success. Updated the following files:\nM maya_transcribe/asr.py\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


**Adjusting memory limits for rerun**

I'm setting `limit_mm_per_prompt` to reduce dummy image processing by assigning 0 to image and 1 to audio, then rerunning gemma E2B to address ongoing memory consumption by the root process.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.15 --vllm-max-model-len 4096


Output:
Wall time: 135.8 seconds
aborted by user


Summary: auto


i've cleared the other task which is using GPU. you now have 80GB GPU. use more utilization for GPU.


**Adjusting GPU utilization for rerun**

I'm updating Gemma3n's GPU memory utilization to 0.9 per user request while keeping the image memory limit to avoid OOM, then rerunning the benchmark. I'll also consider running Voxtral-Small if GPU memory allows.


**Confirming GPU memory and prioritizing runs**

I'm verifying GPU memory is fully freed before proceeding and planning to run Gemma3n E2B for timely results, while considering Voxtral-Small if resources allow. I'll avoid longer Gemma3n variants to keep runtime reasonable.


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
Wed Jan 14 23:37:49 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   32C    P0             74W /  400W |   14611MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A   1863213      C   VLLM::EngineCore                            14602MiB |
+-----------------------------------------------------------------------------------------+


**Killing own lingering process to free memory**


Output:
Exit code: 0
Wall time: 0 seconds
Output:


**Rechecking GPU status**


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
Wed Jan 14 23:38:02 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   30C    P0             51W /  400W |       1MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+


**Preparing to rerun gemma3n with higher GPU utilization**

GPU is now free, so I'm setting gemma3n to run with 0.9 GPU memory utilization and max model length 4096, disabling images to save memory. After gemma3n, I'll consider running the larger Voxtral-Small 24B model at the same utilization, given available GPU memory.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096


Output:
Exit code: 0
Wall time: 280.4 seconds
Total output lines: 494
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:38:22.156149: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-14 23:38:28 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'disable_log_stats': True, 'limit_mm_per_prompt': {'image': 0, 'audio': 1}, 'model': 'google/gemma-3n-E2B-it'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-14 23:38:29 [model.py:514] Resolved architecture: Gemma3nForConditionalGeneration
INFO 01-14 23:38:29 [model.py:1661] Using max model len 4096
INFO 01-14 23:38:29 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 01-14 23:38:31 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:38:36.918713: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=1865745) INFO 01-14 23:38:40 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='google/gemma-3n-E2B-it', speculative_config=None, tokenizer='google/gemma-3n-E2B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=google/gemma-3n-E2B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1865745) INFO 01-14 23:38:44 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:37017 backend=nccl
(EngineCore_DP0 pid=1865745) INFO 01-14 23:38:44 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=1865745) INFO 01-14 23:38:58 [gpu_model_runner.py:3562] Starting to load model google/gemma-3n-E2B-it...
(EngineCore_DP0 pid=1865745) INFO 01-14 23:38:59 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=1865745)
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
(EngineCore_DP0 pid=1865745)
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:00,  2.13it/s]
(EngineCore_DP0 pid=1865745)
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.40it/s]
(EngineCore_DP0 pid=1865745)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.39it/s]
(EngineCore_DP0 pid=1865745)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.45it/s]
(EngineCore_DP0 pid=1865745)
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:02 [default_loader.py:308] Loading weights took 2.20 seconds
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:03 [gpu_model_runner.py:3659] Model loading took 10.2839 GiB memory and 3.059246 seconds
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:03 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 43 audio items of the maximum feature size.
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:17 [backends.py:643] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/1a387ba34f/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:17 [backends.py:703] Dynamo bytecode transform time: 12.96 s
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:32 [backends.py:226] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 7.866 s
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:32 [monitor.py:34] torch.compile takes 20.82 s in total
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:33 [gpu_worker.py:375] Available KV cache memory: 58.55 GiB
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:34 [kv_cache_utils.py:1291] GPU KV cache size: 1,534,896 tokens
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:34 [kv_cache_utils.py:1296] Maximum concurrency for 4,096 tokens per request: 373.56x
(EngineCore_DP0 pid=1865745)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   4%|▍         | 2/51 [00:00<00:03, 14.66it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   8%|▊         | 4/51 [00:00<00:03, 14.87it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  12%|█▏        | 6/51 [00:00<00:03, 14.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 8/51 [00:00<00:02, 15.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|█▉        | 10/51 [00:00<00:02, 15.00it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  24%|██▎       | 12/51 [00:00<00:02, 14.93it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 14/51 [00:00<00:02, 14.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 16/51 [00:01<00:02, 14.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  35%|███▌      | 18/51 [00:01<00:02, 14.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  39%|███▉      | 20/51 [00:01<00:02, 14.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 22/51 [00:01<00:01, 14.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  47%|████▋     | 24/51 [00:01<00:01, 14.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  51%|█████     | 26/51 [00:01<00:01, 14.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 28/51 [00:01<00:01, 14.71it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  59%|█████▉    | 30/51 [00:02<00:01, 14.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  63%|██████▎   | 32/51 [00:02<00:01, 14.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:01, 14.50it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████   | 36/51 [00:02<00:01, 14.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  75%|███████▍  | 38/51 [00:02<00:00, 14.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  78%|███████▊  | 40/51 [00:02<00:00, 14.62it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 42/51 [00:02<00:00, 14.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  86%|████████▋ | 44/51 [00:02<00:00, 14.57it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  90%|█████████ | 46/51 [00:03<00:00, 14.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  94%|█████████▍| 48/51 [00:03<00:00, 14.40it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  98%|█████████▊| 50/51 [00:03<00:00, 14.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 14.58it/s]
(EngineCore_DP0 pid=1865745)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   6%|▌         | 2/35 [00:00<00:02, 12.88it/s]
Capturing CUDA graphs (decode, FULL):  11%|█▏        | 4/35 [00:00<00:02, 14.09it/s]
Capturing CUDA graphs (decode, FULL):  17%|█▋        | 6/35 [00:00<00:01, 14.55it/s]
Capturing CUDA graphs (decode, FULL):  23%|██▎       | 8/35 [00:00<00:01, 14.65it/s]
Capturing CUDA graphs (decode, FULL):  29%|██▊       | 10/35 [00:00<00:01, 14.78it/s]
Capturing CUDA graphs (decode, FULL):  34%|███▍      | 12/35 [00:00<00:01, 14.87it/s]
Capturing CUDA graphs (decode, FULL):  40%|████      | 14/35 [00:00<00:01, 14.93it/s]
Capturing CUDA graphs (decode, FULL):  46%|████▌     | 16/35 [00:01<00:01, 14.95it/s]
Capturing CUDA graphs (decode, FULL):  51%|█████▏    | 18/35 [00:01<00:01, 15.02it/s]
Capturing CUDA graphs (decode, FULL):  57%|█████▋    | 20/35 [00:01<00:00, 15.08it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 22/35 [00:01<00:00, 14.92it/s]
Capturing CUDA graphs (decode, FULL):  69%|██████▊   | 24/35 [00:01<00:00, 14.98it/s]
Capturing CUDA graphs (decode, FULL):  74%|███████▍  | 26/35 [00:01<00:00, 15.03it/s]
Capturing CUDA graphs (decode, FULL):  80%|████████  | 28/35 [00:01<00:00, 15.08it/s]
Capturing CUDA graphs (decode, FULL):  86%|████████▌ | 30/35 [00:02<00:00, 15.13it/s]
Capturing CUDA graphs (decode, FULL):  91%|█████████▏| 32/35 [00:02<00:00, 15.09it/s]
Capturing CUDA graphs (decode, FULL):  97%|█████████▋| 34/35 [00:02<00:00, 15.08it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 35/35 [00:02<00:00, 14.92it/s]
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:41 [gpu_model_runner.py:4587] Graph capturing finished in 7 secs, took 1.05 GiB
(EngineCore_DP0 pid=1865745) INFO 01-14 23:39:41 [core.py:259] init engine (profile, create kv cache, warmup model) took 38.62 seconds
INFO 01-14 23:39:43 [llm.py:360] Supported tasks: ['generate', 'transcription']

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:16<00:00, 16.04s/it]
Adding requests: 100%|██████████| 1/1 [00:16<00:00, 16.04s/it]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.50it/s, est. speed input: 1149.44 toks/s, output: 60.48 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.50it/s, est. speed input: 1149.44 toks/s, output: 60.48 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.49it/s, est. speed input: 1149.44 toks/s, output: 60.48 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 13.62it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s, est. speed input: 993.46 toks/s, output: 80.79 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.75it/s, est. speed input: 993.46 toks/s, output: 80.79 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.74it/s, est. speed input: 993.46 toks/s, output: 80.79 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.59it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.80it/s, est. speed input: 584.51 toks/s, output: 103.47 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.80it/s, est. speed input: 584.51 toks/s, output: 103.47 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.79it/s, est. speed input: 584.51 toks/s, output: 103.47 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00,  9.91it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00,  9.89it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.55it/s, est. speed input: 1159.39 toks/s, output: 88.74 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.55it/s, est. speed input: 1159.39 toks/s, output: 88.74 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.54it/s, est. speed input: 1159.39 toks/s, output: 88.74 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 27.54it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.61it/s, est. speed input: 1381.79 toks/s, output: 72.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.61it/s, est. speed input: 1381.79 toks/s, output: 72.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.59it/s, est. speed input: 1381.79 toks/s, output: 72.71 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 16.03it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.00it/s, est. speed input: 1045.52 toks/s, output: 75.02 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.00it/s, est. speed input: 1045.52 toks/s, output: 75.02 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.99it/s, est. speed input: 1045.52 toks/s, output: 75.02 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 19.52it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.72it/s, est. speed input: 986.59 toks/s, output: 75.52 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.72it/s, est. speed input: 986.59 toks/s, output: 75.52 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.72it/s, est. speed input: 986.59 toks/s, output: 75.52 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 11.39it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.12it/s, est. speed input: 1070.15 toks/s, output: 87.04 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.12it/s, est. speed input: 1070.15 toks/s, output: 87.04 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.11it/s, est. speed input: 1070.15 toks/s, output: 87.04 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 10.61it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.34it/s, est. speed input: 1115.50 to…27571 tokens truncated…1.90 toks/s, output: 61.67 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 13.68it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:08<00:00,  8.37s/it, est. speed input: 24.97 toks/s, output: 122.34 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:08<00:00,  8.37s/it, est. speed input: 24.97 toks/s, output: 122.34 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:08<00:00,  8.37s/it, est. speed input: 24.97 toks/s, output: 122.34 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 12.47it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.48it/s, est. speed input: 936.12 toks/s, output: 71.65 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.48it/s, est. speed input: 936.12 toks/s, output: 71.65 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.47it/s, est. speed input: 936.12 toks/s, output: 71.65 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 11.26it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.35it/s, est. speed input: 1117.49 toks/s, output: 85.54 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.35it/s, est. speed input: 1117.49 toks/s, output: 85.54 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.34it/s, est. speed input: 1117.49 toks/s, output: 85.54 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 12.97it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.52it/s, est. speed input: 1154.62 toks/s, output: 77.33 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.52it/s, est. speed input: 1154.62 toks/s, output: 77.33 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.51it/s, est. speed input: 1154.62 toks/s, output: 77.33 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 17.91it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.99it/s, est. speed input: 1252.99 toks/s, output: 83.92 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.99it/s, est. speed input: 1252.99 toks/s, output: 83.92 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.98it/s, est. speed input: 1252.99 toks/s, output: 83.92 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 14.09it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.02it/s, est. speed input: 1049.29 toks/s, output: 90.36 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.02it/s, est. speed input: 1049.29 toks/s, output: 90.36 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.02it/s, est. speed input: 1049.29 toks/s, output: 90.36 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 17.43it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.39it/s, est. speed input: 1126.79 toks/s, output: 80.86 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.39it/s, est. speed input: 1126.79 toks/s, output: 80.86 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.38it/s, est. speed input: 1126.79 toks/s, output: 80.86 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 13.55it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.20it/s, est. speed input: 877.94 toks/s, output: 79.81 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.20it/s, est. speed input: 877.94 toks/s, output: 79.81 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.20it/s, est. speed input: 877.94 toks/s, output: 79.81 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 15.40it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.86it/s, est. speed input: 1015.16 toks/s, output: 97.13 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.86it/s, est. speed input: 1015.16 toks/s, output: 97.13 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.85it/s, est. speed input: 1015.16 toks/s, output: 97.13 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.26it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.97it/s, est. speed input: 830.47 toks/s, output: 95.36 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.97it/s, est. speed input: 830.47 toks/s, output: 95.36 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.97it/s, est. speed input: 830.47 toks/s, output: 95.36 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.46it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.77it/s, est. speed input: 996.12 toks/s, output: 90.54 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.77it/s, est. speed input: 996.12 toks/s, output: 90.54 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.76it/s, est. speed input: 996.12 toks/s, output: 90.54 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 17.36it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.77it/s, est. speed input: 1206.04 toks/s, output: 75.01 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.77it/s, est. speed input: 1206.04 toks/s, output: 75.01 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.76it/s, est. speed input: 1206.04 toks/s, output: 75.01 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 13.49it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.98it/s, est. speed input: 1041.12 toks/s, output: 94.63 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.98it/s, est. speed input: 1041.12 toks/s, output: 94.63 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.97it/s, est. speed input: 1041.12 toks/s, output: 94.63 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.58it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.28it/s, est. speed input: 1313.56 toks/s, output: 75.41 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.28it/s, est. speed input: 1313.56 toks/s, output: 75.41 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.27it/s, est. speed input: 1313.56 toks/s, output: 75.41 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 21.92it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.61it/s, est. speed input: 1382.99 toks/s, output: 72.77 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.61it/s, est. speed input: 1382.99 toks/s, output: 72.77 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.60it/s, est. speed input: 1382.99 toks/s, output: 72.77 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 20.78it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.79it/s, est. speed input: 1210.95 toks/s, output: 81.11 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.79it/s, est. speed input: 1210.95 toks/s, output: 81.11 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.79it/s, est. speed input: 1210.95 toks/s, output: 81.11 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00,  9.54it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00,  9.52it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.87it/s, est. speed input: 1017.11 toks/s, output: 92.46 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.87it/s, est. speed input: 1017.11 toks/s, output: 92.46 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.86it/s, est. speed input: 1017.11 toks/s, output: 92.46 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 35.09it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.07it/s, est. speed input: 1268.35 toks/s, output: 84.95 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.07it/s, est. speed input: 1268.35 toks/s, output: 84.95 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.06it/s, est. speed input: 1268.35 toks/s, output: 84.95 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 20.22it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.84it/s, est. speed input: 1220.81 toks/s, output: 81.77 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.84it/s, est. speed input: 1220.81 toks/s, output: 81.77 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.83it/s, est. speed input: 1220.81 toks/s, output: 81.77 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.12it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.95it/s, est. speed input: 1244.34 toks/s, output: 77.39 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.95it/s, est. speed input: 1244.34 toks/s, output: 77.39 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.95it/s, est. speed input: 1244.34 toks/s, output: 77.39 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 19.13it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.01it/s, est. speed input: 1255.38 toks/s, output: 78.08 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.01it/s, est. speed input: 1255.38 toks/s, output: 78.08 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  6.00it/s, est. speed input: 1255.38 toks/s, output: 78.08 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 13.10it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.41it/s, est. speed input: 922.23 toks/s, output: 79.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.41it/s, est. speed input: 922.23 toks/s, output: 79.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.41it/s, est. speed input: 922.23 toks/s, output: 79.42 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.12it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.05it/s, est. speed input: 1055.92 toks/s, output: 85.88 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.05it/s, est. speed input: 1055.92 toks/s, output: 85.88 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.04it/s, est. speed input: 1055.92 toks/s, output: 85.88 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.90it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.68it/s, est. speed input: 978.79 toks/s, output: 74.92 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.68it/s, est. speed input: 978.79 toks/s, output: 74.92 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.67it/s, est. speed input: 978.79 toks/s, output: 74.92 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 18.15it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.35it/s, est. speed input: 910.06 toks/s, output: 82.72 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.35it/s, est. speed input: 910.06 toks/s, output: 82.72 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.35it/s, est. speed input: 910.06 toks/s, output: 82.72 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 19.11it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.64it/s, est. speed input: 1180.01 toks/s, output: 79.03 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.64it/s, est. speed input: 1180.01 toks/s, output: 79.03 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.64it/s, est. speed input: 1180.01 toks/s, output: 79.03 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 1/1 [00:00<00:00, 15.71it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.01it/s, est. speed input: 1047.63 toks/s, output: 80.20 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.01it/s, est. speed input: 1047.63 toks/s, output: 80.20 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  5.01it/s, est. speed input: 1047.63 toks/s, output: 80.20 toks/s]
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:03:08
[rank0]:[W114 23:42:52.386181728 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Wrote: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl
Wrote: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/summary.json
Wrote: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 1.3760242396492393,
  "cer_mean": 1.0522220838587564,
  "wer_p50": 0.1111111111111111,
  "wer_p90": 0.375,
  "wer_p95": 0.5333333333333333,
  "cer_p50": 0.0425531914893617,
  "cer_p90": 0.16666666666666666,
  "cer_p95": 0.22727272727272727,
  "segment_error_rate": 0.72,
  "pct_wer_gt_5pct": 0.72,
  "pct_wer_gt_10pct": 0.525,
  "wer_by_language_mean": {
    "hin": 1.3760242396492393
  },
  "wer_by_decision_mean": {
    "accept": 1.651370648815905,
    "human": 0.777255064159826
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.176010101010101,
    "10-30s": 0.2831959706959707,
    "3-10s": 1.4480597859587216
  },
  "cer_by_language_mean": {
    "hin": 1.0522220838587564
  },
  "cer_by_decision_mean": {
    "accept": 1.3648066814614563,
    "human": 0.3724746255798699
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.10565015479876161,
    "10-30s": 0.0722371483484663,
    "3-10s": 1.1140633987540878
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 1.3760242396492393,
    "selection_gap_wer_mean": 0.0,
    "selection_gap_wer_p90": 0.0
  }
}


**Noticing abnormally high WER in Gemma3n results**

Gemma3n output shows an unusually high mean WER of 137%, suggesting potential issues with transcription or system setup. Planning to run Voxtral-Small 24B next and will check GPU usage via nvidia-smi to ensure resources are freed.


Output:
Exit code: 0
Wall time: 0.4 seconds
Output:
Wed Jan 14 23:43:03 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.05              Driver Version: 560.35.05      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A100-SXM4-80GB          On  |   00000000:01:00.0 Off |                    0 |
| N/A   34C    P0             52W /  400W |       1MiB /  81920MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+


# AGENTS.md instructions for /home/ubuntu/maya-transcribe

<INSTRUCTIONS>
## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.
</INSTRUCTIONS>

<environment_context>
  <cwd>/home/ubuntu/maya-transcribe</cwd>
  <shell>bash</shell>
</environment_context>

plan was [master.md](master.md)

Full Benchmark Analysis Summary

  Run Comparison
  ┌─────────────┬────────────────────┬──────────────────┬───────────────────┐
  │   Metric    │ Baseline (FW only) │ + Vaani (strict) │ + Vaani (relaxed) │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ WER (final) │ 30.6%              │ 23.7%            │ 23.7%             │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Accept rate │ 0/200              │ 11/200           │ 81/200            │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Human rate  │ 200/200            │ 189/200          │ 119/200           │
  ├─────────────┼────────────────────┼──────────────────┼───────────────────┤
  │ Oracle best │ 29.7%              │ 9.8%             │ 9.8%              │
  └─────────────┴────────────────────┴──────────────────┴───────────────────┘
  Decision Quality with Relaxed Threshold
  ┌──────────┬───────┬──────────┬───────────────┐
  │ Decision │ Count │ WER Mean │ Disagree Rate │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Accept   │ 81    │ 18.6%    │ 8.5%          │
  ├──────────┼───────┼──────────┼───────────────┤
  │ Human    │ 119   │ 27.1%    │ 13.2%         │
  └──────────┴───────┴──────────┴───────────────┘
  WER by Disagreement Rate Bucket

  - 0-5%: n=35, WER=6.0% (excellent)
  - 5-10%: n=47, WER=19.6%
  - 10-15%: n=68, WER=24.6%
  - 15-25%: n=44, WER=36.9%
  - 25-100%: n=6, WER=50.9%

  Remaining Human Decision Reasons

  - lid_uncertain: 63 (LID confused between Hindi/Urdu/related languages)
  - asr_disagreement: 50 (still above 15% threshold)
  - ctc_ambiguous: 49 (CTC margin too low)

  LID Analysis

  - 82% correctly identified as Hindi
  - 31.5% flagged uncertain due to low margin (Hindi vs Urdu/Gujarati/Bhojpuri)
  - This is expected: MMS-LID struggles with closely related Indic languages

  Final Selection Distribution

  - ROVER: 64.5% (129/200)
  - Vaani Hindi: 34.0% (68/200)
  - FW large-v3: 1.5% (3/200)

  Key Insights

  1. Oracle potential is real: 9.8% WER is achievable with perfect selection
  2. Current selection gap: 13.9pp left on the table due to suboptimal hypothesis selection
  3. Gating works directionally: Accept segments (18.6% WER) are better than Human segments (27.1% WER)
  4. Disagreement correlates with quality: Strong correlation between disagreement rate and WER
  5. LID is the new bottleneck: 31.5% flagged uncertain due to Hindi/Urdu confusion

  Recommendations

  1. Consider LID-Urdu equivalence for Hindi: Treat Hindi and Urdu as equivalent for routing since they share script (when using Devanagari transcription)
  2. Better CTC selection: Current CTC selection isn't capturing the oracle improvement - the selection_gap of 13.9pp suggests the CTC model isn't reliably picking the best hypothesis
  3. Two-stage approach: Accept high-confidence segments (low disagreement), use LLM-based selection for ambiguous cases

✻ Sautéed for 32m 59s

● Background command "Run benchmark with relaxed disagreement threshold" completed (exit code 0).

● I've already analyzed those results above. The benchmark with relaxed threshold (15% disagreement rate) completed successfully:

  Key result: Accept rate improved from 11 → 81 segments (40.5%) while maintaining the same 23.7% WER.

  The analysis shows the pipeline is working - accepted segments have lower WER (18.6%) than segments routed to human review (27.1%). The main remaining bottlenecks are:
  - LID uncertainty (Hindi/Urdu confusion) - 63 segments
  - Disagreement still above threshold - 50 segments
  - CTC ambiguity - 49 segments

  The oracle potential (9.8% WER) is significantly better than what we're achieving (23.7%), indicating room for improved hypothesis selection.

But these WER scores are not what i'm looking for. i need <5% WER scores for all languages consistently. For that to happen, i can run any LLMs like Gemma3n, voxtral.

Can we try these LLMs for audio transcription. can you test and give the performance scores or WER scores when compared to our recent ones above. download these models from huggingface and integrate them and do inference on transcription tasks and get full scores. both models wont fit at once, onload one, then another. you should basically not run transformers based infernece. use vLLM or any other inference engine for this to run the gemma3n models or voxtral models.


**Preparing to run high-GPU benchmark**

Planning to run the Voxtral-Small benchmark using vLLM with increased GPU utilization as requested, confirming no skill or unexpected changes block proceeding.


**Running benchmark with vLLM on GPU**

Proceeding to execute the benchmark command for Voxtral-Small using vLLM with high GPU utilization and configured max model length, directing output to the expected path.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096


Output:
Exit code: 124
Wall time: 120.4 seconds
Output:
command timed out after 120368 milliseconds
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:43:43.007070: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-14 23:43:50 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'disable_log_stats': True, 'model': 'mistralai/Voxtral-Small-24B-2507'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-14 23:43:50 [model.py:514] Resolved architecture: VoxtralForConditionalGeneration
INFO 01-14 23:43:50 [model.py:1661] Using max model len 4096
INFO 01-14 23:43:50 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
[2026-01-14 23:43:51] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
[2026-01-14 23:43:51] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
WARNING 01-14 23:43:52 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:43:57.633505: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:01 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='mistralai/Voxtral-Small-24B-2507', speculative_config=None, tokenizer='mistralai/Voxtral-Small-24B-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=mistralai/Voxtral-Small-24B-2507, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1881506) [2026-01-14 23:44:03] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(EngineCore_DP0 pid=1881506) [2026-01-14 23:44:03] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:04 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:60599 backend=nccl
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:04 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:05 [gpu_model_runner.py:3562] Starting to load model mistralai/Voxtral-Small-24B-2507...
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:06 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=1881506) INFO 01-14 23:44:06 [layer.py:477] Using AttentionBackendEnum.FLASH_ATTN for MultiHeadAttention in multimodal encoder.
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:06 [weight_utils.py:487] Time spent downloading weights for mistralai/Voxtral-Small-24B-2507: 60.188148 seconds
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:06 [weight_utils.py:527] No consolidated.safetensors.index.json found in remote.
(EngineCore_DP0 pid=1881506)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=1881506)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:20<00:00, 20.03s/it]
(EngineCore_DP0 pid=1881506)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:20<00:00, 20.03s/it]
(EngineCore_DP0 pid=1881506)
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:28 [default_loader.py:308] Loading weights took 21.53 seconds
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:29 [gpu_model_runner.py:3659] Model loading took 45.2032 GiB memory and 82.287891 seconds
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:29 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 2 audio items of the maximum feature size.
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:36 [backends.py:643] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/936cd8900f/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=1881506) INFO 01-14 23:45:36 [backends.py:703] Dynamo bytecode transform time: 6.23 s


**Checking for lingering processes**

Planning to run a process search using `ps` and ripgrep to detect any still-running instances of vLLM, maya_transcribe, or related Python commands after the timeout.


ps -ef | rg -i "vllm|voxtral|maya_transcribe"


Output:
Exit code: 0
Wall time: 0 seconds
Output:
ubuntu    274345       1  0 Jan03 ?        01:23:57 /home/ubuntu/qwen_image/.venv/bin/python3 /home/ubuntu/qwen_image/.venv/bin/vllm serve Qwen/Qwen-Image-Edit-2511 --omni --port 8092 --cache-backend tea_cache --cache-config {"rel_l1_thresh": 0.2}
root      738461  738440  2 Jan06 ?        04:06:28 /usr/bin/python3.12 /usr/local/bin/vllm serve Qwen/Qwen-Image-Edit-2511 --omni --port 8000 --trust-remote-code
ubuntu   1882218 1828144  0 23:45 ?        00:00:00 /bin/bash -lc ps -ef | rg -i "vllm|voxtral|maya_transcribe"
ubuntu   1882223 1882218  0 23:45 ?        00:00:00 rg -i vllm|voxtral|maya_transcribe


**Adjusting timeout for heavy tasks**

Planning to rerun the command with a longer timeout around 1 hour to accommodate heavy GPU use and compilation delays, ensuring the transcription run doesn't get killed prematurely.


**Confirming GPU memory utilization setting**

Deciding to use 90% GPU memory utilization to balance efficiency and safety for the 24B model, while rerunning with the longer timeout and possibly reusing cached weights.