Output:
Exit code: 1
Wall time: 74.3 seconds
Total output lines: 32127
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:25:23.595362: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-14 23:25:30 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'gpu_memory_utilization': 0.15, 'disable_log_stats': True, 'model': 'google/gemma-3n-E2B-it'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-14 23:25:41 [model.py:514] Resolved architecture: Gemma3nForConditionalGeneration
INFO 01-14 23:25:41 [model.py:1661] Using max model len 4096
INFO 01-14 23:25:42 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 01-14 23:25:46 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:25:51.114662: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=1859265) INFO 01-14 23:25:54 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='google/gemma-3n-E2B-it', speculative_config=None, tokenizer='google/gemma-3n-E2B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=google/gemma-3n-E2B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1859265) INFO 01-14 23:25:58 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:54547 backend=nccl
(EngineCore_DP0 pid=1859265) INFO 01-14 23:25:58 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=1859265) INFO 01-14 23:26:12 [gpu_model_runner.py:3562] Starting to load model google/gemma-3n-E2B-it...
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     super().__init__(
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self._init_executor()
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self.driver_worker.load_model()
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 289, in load_model
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3581, in load_model
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self.model = model_loader.load_model(
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     model = initialize_model(
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/utils.py", line 48, in initialize_model
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py", line 504, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     self.vision_tower = AutoModel.from_config(config=config.vision_config)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 456, in from_config
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     return model_class._from_config(config, **kwargs)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2311, in _from_config
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     model = cls(config, **kwargs)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/timm_wrapper/modeling_timm_wrapper.py", line 160, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     super().__init__(config)
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/timm_wrapper/modeling_timm_wrapper.py", line 91, in __init__
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     requires_backends(self, ["vision", "timm"])
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2143, in requires_backends
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]     raise ImportError("".join(failed))
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866] ImportError:
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866] TimmWrapperModel requires the timm library but it was not found in your environment. You can install it with pip:
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866] `pip install timm`. Please note that you may need to restart your runtime after installation.
(EngineCore_DP0 pid=1859265) ERROR 01-14 23:26:13 [core.py:866]
(EngineCore_DP0 pid=1859265) Process EngineCore_DP0:
(EngineCore_DP0 pid=1859265) Traceback (most recent call last):
(EngineCore_DP0 pid=1859265)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=1859265)     self.run()
(EngineCore_DP0 pid=1859265)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=1859265)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=1859265)     raise e
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1859265)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1859265)     super().__init__(
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 102, in __init__
(EngineCore_DP0 pid=1859265)     self.model_executor = executor_class(vllm_config)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 101, in __init__
(EngineCore_DP0 pid=1859265)     self._init_executor()
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 48, in _init_executor
(EngineCore_DP0 pid=1859265)     self.driver_worker.load_model()
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 289, in load_model
(EngineCore_DP0 pid=1859265)     self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3581, in load_model
(EngineCore_DP0 pid=1859265)     self.model = model_loader.load_model(
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(EngineCore_DP0 pid=1859265)     model = initialize_model(
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/model_loader/utils.py", line 48, in initialize_model
(EngineCore_DP0 pid=1859265)     return model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py", line 504, in __init__
(EngineCore_DP0 pid=1859265)     self.vision_tower = AutoModel.from_config(config=config.vision_config)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 456, in from_config
(EngineCore_DP0 pid=1859265)     return model_class._from_config(config, **kwargs)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
(EngineCore_DP0 pid=1859265)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2311, in _from_config
(EngineCore_DP0 pid=1859265)     model = cls(config, **kwargs)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/timm_wrapper/modeling_timm_wrapper.py", line 160, in __init__
(EngineCore_DP0 pid=1859265)     super().__init__(config)
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/timm_wrapper/modeling_timm_wrapper.py", line 91, in __init__
(EngineCore_DP0 pid=1859265)     requires_backends(self, ["vision", "timm"])
(EngineCore_DP0 pid=1859265)   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/utils/import_utils.py", line 2143, in requires_backends
(EngineCore_DP0 pid=1859265)     raise ImportError("".join(failed))
(EngineCore_DP0 pid=1859265) ImportError:
(EngineCore_DP0 pid=1859265) TimmWrapperModel requires the timm library but it was not found in your environment. You can install it with pip:
(EngineCore_DP0 pid=1859265) `pip install timm`. Please note that you may need to restart your runtime after installation.
(EngineCore_DP0 pid=1859265)
[rank0]:[W114 23:26:15.736050108 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py:179 in run_pipeline │
│                                                                              │
│   176 │   vllm_backend: VllmTranscriptionASR | None = None                   │
│   177 │   if vllm_asr_model:                                                 │
│   178 │   │   try:                                                           │
│ ❱ 179 │   │   │   vllm_backend = VllmTranscriptionASR(                       │
│   180 │   │   │   │   model_id=vllm_asr_model,                               │
│   181 │   │   │   │   gpu_memory_utilization=vllm_gpu_memory_utilization,    │
│   182 │   │   │   │   max_model_len=vllm_max_model_len,                      │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                       audio_files = [                                    │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hi…717226 tokens truncated…      │ │
│ │              eng_identity = b'\x00\x00'                                  │ │
│ │                 eng_index = 0                                            │ │
│ │                    engine = <vllm.v1.engine.utils.CoreEngine object at   │ │
│ │                             0x7f446e4ae7d0>                              │ │
│ │                    events = [(62, 4)]                                    │ │
│ │                  finished = {}                                           │ │
│ │          handshake_socket = <zmq.Socket(zmq.ROUTER) at 0x7f446e4b42e0    │ │
│ │                             closed>                                      │ │
│ │                  headless = False                                        │ │
│ │              init_message = b'\x83\xa9addresses\x85\xa6inputs\x91\xd9/i… │ │
│ │                     local = True                                         │ │
│ │               local_count = 1                                            │ │
│ │                       msg = {                                            │ │
│ │                             │   'status': 'HELLO',                       │ │
│ │                             │   'local': True,                           │ │
│ │                             │   'headless': False                        │ │
│ │                             }                                            │ │
│ │           parallel_config = ParallelConfig(                              │ │
│ │                             │   pipeline_parallel_size=1,                │ │
│ │                             │   tensor_parallel_size=1,                  │ │
│ │                             │   prefill_context_parallel_size=1,         │ │
│ │                             │   data_parallel_size=1,                    │ │
│ │                             │   data_parallel_size_local=1,              │ │
│ │                             │   data_parallel_rank=0,                    │ │
│ │                             │   data_parallel_rank_local=0,              │ │
│ │                             │   data_parallel_master_ip='127.0.0.1',     │ │
│ │                             │   data_parallel_rpc_port=29550,            │ │
│ │                             │   data_parallel_master_port=0,             │ │
│ │                             │   data_parallel_backend='mp',              │ │
│ │                             │   data_parallel_external_lb=False,         │ │
│ │                             │   data_parallel_hybrid_lb=False,           │ │
│ │                             │   enable_expert_parallel=False,            │ │
│ │                             │   enable_eplb=False,                       │ │
│ │                             │   eplb_config=EPLBConfig(                  │ │
│ │                             │   │   window_size=1000,                    │ │
│ │                             │   │   step_interval=3000,                  │ │
│ │                             │   │   num_redundant_experts=0,             │ │
│ │                             │   │   log_balancedness=False,              │ │
│ │                             │   │   use_async=False,                     │ │
│ │                             │   │   policy='default'                     │ │
│ │                             │   ),                                       │ │
│ │                             │   expert_placement_strategy='linear',      │ │
│ │                             │                                            │ │
│ │                             all2all_backend='allgather_reducescatter',   │ │
│ │                             │   max_parallel_loading_workers=None,       │ │
│ │                             │   disable_custom_all_reduce=False,         │ │
│ │                             │   enable_dbo=False,                        │ │
│ │                             │   dbo_decode_token_threshold=32,           │ │
│ │                             │   dbo_prefill_token_threshold=512,         │ │
│ │                             │                                            │ │
│ │                             disable_nccl_for_dp_synchronization=False,   │ │
│ │                             │   ray_workers_use_nsight=False,            │ │
│ │                             │   ray_runtime_env=None,                    │ │
│ │                             │   placement_group=None,                    │ │
│ │                             │   distributed_executor_backend='uni',      │ │
│ │                             │                                            │ │
│ │                             worker_cls='vllm.v1.worker.gpu_worker.Worke… │ │
│ │                             │   sd_worker_cls='auto',                    │ │
│ │                             │   worker_extension_cls='',                 │ │
│ │                             │   master_addr='127.0.0.1',                 │ │
│ │                             │   master_port=29501,                       │ │
│ │                             │   node_rank=0,                             │ │
│ │                             │   nnodes=1,                                │ │
│ │                             │   world_size=1,                            │ │
│ │                             │   rank=0,                                  │ │
│ │                             │   _data_parallel_master_port_list=[],      │ │
│ │                             │   decode_context_parallel_size=1,          │ │
│ │                             │   dcp_kv_cache_interleave_size=1,          │ │
│ │                             │   cp_kv_cache_interleave_size=1,           │ │
│ │                             │   _api_process_count=1,                    │ │
│ │                             │   _api_process_rank=0                      │ │
│ │                             )                                            │ │
│ │                    poller = <zmq.sugar.poll.Poller object at             │ │
│ │                             0x7f446e4af250>                              │ │
│ │              proc_manager = <vllm.v1.engine.utils.CoreEngineProcManager  │ │
│ │                             object at 0x7f446e4af370>                    │ │
│ │           ready_msg_bytes = b'\x83\xa6status\xa5HELLO\xa5local\xc3\xa8h… │ │
│ │              remote_count = 0                                            │ │
│ │ remote_should_be_headless = True                                         │ │
│ │                  sentinel = 62                                           │ │
│ │             start_pending = [1, 0]                                       │ │
│ │                    status = 'HELLO'                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Engine core initialization failed. See root cause above. Failed
core proc(s): {}

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/maya-transcribe/maya_transcribe/cli.py:463 in bench_run         │
│                                                                              │
│   460 │   │   thresholds["require_ctc"] = require_ctc                        │
│   461 │   │   thresholds["min_ctc_margin"] = float(min_ctc_margin)           │
│   462 │                                                                      │
│ ❱ 463 │   results_path = run_pipeline(                                       │
│   464 │   │   segments_dir=audio_dir,                                        │
│   465 │   │   out_dir=out,                                                   │
│   466 │   │   lid_model_id=lid_model_id,                                     │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                      audio_dir = PosixPath('data/bench/maya_hf_hin_v2/a… │ │
│ │                      bench_dir = PosixPath('data/bench/maya_hf_hin_v2')  │ │
│ │                        ctc_map = None                                    │ │
│ │                      ctc_model = None                                    │ │
│ │                  ctc_model_map = None                                    │ │
│ │                     hypothesis = 'final'                                 │ │
│ │                         ic_map = None                                    │ │
│ │   indicconformer_default_model = None                                    │ │
│ │       indicconformer_model_map = None                                    │ │
│ │                   lid_model_id = 'facebook/mms-lid-256'                  │ │
│ │                          limit = None                                    │ │
│ │                 min_ctc_margin = 0.05                                    │ │
│ │                min_silence_sec = 0.15                                    │ │
│ │                  mms_asr_model = None                                    │ │
│ │                            out = PosixPath('outputs/bench/maya_hf_hin_v… │ │
│ │                      refs_path = PosixPath('data/bench/maya_hf_hin_v2/r… │ │
│ │                    require_ctc = True                                    │ │
│ │            require_lid_backend = True                                    │ │
│ │           silence_threshold_db = -40.0                                   │ │
│ │                   skip_whisper = True                                    │ │
│ │                         tf_map = None                                    │ │
│ │                     thresholds = {                                       │ │
│ │                                  │   'route_lang_from_path': True,       │ │
│ │                                  │   'route_lang_prefer_path': True      │ │
│ │                                  }                                       │ │
│ │                thresholds_json = None                                    │ │
│ │     transformers_whisper_model = []                                      │ │
│ │ transformers_whisper_model_map = None                                    │ │
│ │                   trim_silence = False                                   │ │
│ │                 vllm_asr_model = 'google/gemma-3n-E2B-it'                │ │
│ │    vllm_gpu_memory_utilization = 0.15                                    │ │
│ │             vllm_max_model_len = 4096                                    │ │
│ │                  whisper_model = 'large-v3'                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py:185 in run_pipeline │
│                                                                              │
│   182 │   │   │   │   max_model_len=vllm_max_model_len,                      │
│   183 │   │   │   )                                                          │
│   184 │   │   except Exception as e:                                         │
│ ❱ 185 │   │   │   raise RuntimeError("Failed to initialize vLLM ASR backend. │
│   186 │                                                                      │
│   187 │   default_whisper_run_sampling = bool(whisper_backend is not None) a │
│   188 │   │   transformers_backends or transformers_router or gemini_backend │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                       audio_files = [                                    │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │   ... +190                         │ │
│ │                                     ]                                    │ │
│ │                         ctc_model = None                                 │ │
│ │                     ctc_model_map = None                                 │ │
│ │                        ctc_scorer = None                                 │ │
│ │ extra_transformers_whisper_models = []                                   │ │
│ │                    gemini_backend = None                                 │ │
│ │                  gemini_hard_only = True                                 │ │
│ │          gemini_max_output_tokens = 512                                  │ │
│ │                      gemini_model = None                                 │ │
│ │                gemini_prompt_path = None                                 │ │
│ │      indicconformer_default_model = None                                 │ │
│ │          indicconformer_model_map = None                                 │ │
│ │             indicconformer_router = None                                 │ │
│ │                       lid_backend = <maya_transcribe.lid.MMSLID object   │ │
│ │                                     at 0x7f4690ae82b0>                   │ │
│ │                      lid_model_id = 'facebook/mms-lid-256'               │ │
│ │                             limit = None                                 │ │
│ │                   min_silence_sec = 0.15                                 │ │
│ │                     mms_asr_model = None                                 │ │
│ │                       mms_backend = None                                 │ │
│ │                           out_dir = PosixPath('outputs/bench/maya_hf_hi… │ │
│ │                       rank_by_ctc = None                                 │ │
│ │               require_lid_backend = True                                 │ │
│ │                      segments_dir = PosixPath('data/bench/maya_hf_hin_v… │ │
│ │              silence_threshold_db = -40.0                                │ │
│ │                      skip_whisper = True                                 │ │
│ │                        thresholds = {                                    │ │
│ │                                     │   'route_lang_from_path': True,    │ │
│ │                                     │   'route_lang_prefer_path': True   │ │
│ │                                     }                                    │ │
│ │             transformers_backends = []                                   │ │
│ │               transformers_router = None                                 │ │
│ │  transformers_whisper_default_model None                                 │ │
│ │                                   =                                      │ │
│ │    transformers_whisper_model_map = None                                 │ │
│ │                      trim_silence = False                                │ │
│ │                    vllm_asr_model = 'google/gemma-3n-E2B-it'             │ │
│ │                      vllm_backend = None                                 │ │
│ │       vllm_gpu_memory_utilization = 0.15                                 │ │
│ │                vllm_max_model_len = 4096                                 │ │
│ │                   whisper_backend = None                                 │ │
│ │                     whisper_model = 'large-v3'                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to initialize vLLM ASR backend.


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Installing missing vision dependency**

I'll install the `timm` package to satisfy the vision tower dependency needed by the model, then re-run the process to check if it loads and fits in memory.


Output:
Exit code: 0
Wall time: 3.3 seconds
Output:
Looking in indexes: https://mcache-dsm.massedcompute.com/simple, https://pypi.org/simple
Collecting timm
  Downloading timm-1.0.24-py3-none-any.whl.metadata (38 kB)
Requirement already satisfied: torch in /home/ubuntu/.local/lib/python3.10/site-packages (from timm) (2.9.0)
Requirement already satisfied: torchvision in /home/ubuntu/.local/lib/python3.10/site-packages (from timm) (0.24.0)
Requirement already satisfied: pyyaml in /usr/lib/python3/dist-packages (from timm) (5.4.1)
Requirement already satisfied: huggingface_hub in /home/ubuntu/.local/lib/python3.10/site-packages (from timm) (0.36.0)
Requirement already satisfied: safetensors in /home/ubuntu/.local/lib/python3.10/site-packages (from timm) (0.7.0)
Requirement already satisfied: filelock in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (3.20.0)
Requirement already satisfied: fsspec>=2023.5.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (2023.10.0)
Requirement already satisfied: packaging>=20.9 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (25.0)
Requirement already satisfied: requests in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (2.32.5)
Requirement already satisfied: tqdm>=4.42.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (4.67.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (4.15.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.1.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from huggingface_hub->timm) (1.2.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->huggingface_hub->timm) (3.3)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (2.6.2)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->huggingface_hub->timm) (2020.6.20)
Requirement already satisfied: sympy>=1.13.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (3.4.2)
Requirement already satisfied: jinja2 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (3.1.6)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.93)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.90)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.90)
Requirement already satisfied: nvidia-cudnn-cu12==9.10.2.21 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (9.10.2.21)
Requirement already satisfied: nvidia-cublas-cu12==12.8.4.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.4.1)
Requirement already satisfied: nvidia-cufft-cu12==11.3.3.83 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (11.3.3.83)
Requirement already satisfied: nvidia-curand-cu12==10.3.9.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (10.3.9.90)
Requirement already satisfied: nvidia-cusolver-cu12==11.7.3.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (11.7.3.90)
Requirement already satisfied: nvidia-cusparse-cu12==12.5.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.5.8.93)
Requirement already satisfied: nvidia-cusparselt-cu12==0.7.1 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (0.7.1)
Requirement already satisfied: nvidia-nccl-cu12==2.27.5 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (2.27.5)
Requirement already satisfied: nvidia-nvshmem-cu12==3.3.20 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (3.3.20)
Requirement already satisfied: nvidia-nvtx-cu12==12.8.90 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.90)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.8.93 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (12.8.93)
Requirement already satisfied: nvidia-cufile-cu12==1.13.1.3 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (1.13.1.3)
Requirement already satisfied: triton==3.5.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from torch->timm) (3.5.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from sympy>=1.13.3->torch->timm) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from jinja2->torch->timm) (3.0.3)
Requirement already satisfied: numpy in /home/ubuntu/.local/lib/python3.10/site-packages (from torchvision->timm) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /home/ubuntu/.local/lib/python3.10/site-packages (from torchvision->timm) (10.4.0)
Downloading timm-1.0.24-py3-none-any.whl (2.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 14.9 MB/s  0:00:00
Installing collected packages: timm
Successfully installed timm-1.0.24


Summary: auto


**Preparing to rerun pipeline with timm**

I'm setting up to rerun the gemma E2B pipeline using the same output directory, aware it might contain partial results from the previous failure, but the pipeline will overwrite or create results anew. I'll run it with a 30-minute timeout to monitor progress.


python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.15 --vllm-max-model-len 4096


Output:
Exit code: 1
Wall time: 83.2 seconds
Total output lines: 32172
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:26:55.897822: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-14 23:27:02 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'gpu_memory_utilization': 0.15, 'disable_log_stats': True, 'model': 'google/gemma-3n-E2B-it'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-14 23:27:03 [model.py:514] Resolved architecture: Gemma3nForConditionalGeneration
INFO 01-14 23:27:03 [model.py:1661] Using max model len 4096
INFO 01-14 23:27:03 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
WARNING 01-14 23:27:05 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:27:10.558083: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:14 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='google/gemma-3n-E2B-it', speculative_config=None, tokenizer='google/gemma-3n-E2B-it', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=google/gemma-3n-E2B-it, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:18 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:54773 backend=nccl
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:18 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:32 [gpu_model_runner.py:3562] Starting to load model google/gemma-3n-E2B-it...
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:32 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:49 [weight_utils.py:487] Time spent downloading weights for google/gemma-3n-E2B-it: 16.493541 seconds
(EngineCore_DP0 pid=1860120)
Loading safetensors checkpoint shards:   0% Completed | 0/3 [00:00<?, ?it/s]
(EngineCore_DP0 pid=1860120)
Loading safetensors checkpoint shards:  33% Completed | 1/3 [00:00<00:00,  2.05it/s]
(EngineCore_DP0 pid=1860120)
Loading safetensors checkpoint shards:  67% Completed | 2/3 [00:01<00:00,  1.34it/s]
(EngineCore_DP0 pid=1860120)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.35it/s]
(EngineCore_DP0 pid=1860120)
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00,  1.40it/s]
(EngineCore_DP0 pid=1860120)
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:52 [default_loader.py:308] Loading weights took 2.27 seconds
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:53 [gpu_model_runner.py:3659] Model loading took 10.2839 GiB memory and 19.867864 seconds
(EngineCore_DP0 pid=1860120) INFO 01-14 23:27:53 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 32 image items of the maximum feature size.
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866] EngineCore failed to start.
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866] Traceback (most recent call last):
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     super().__init__(
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 75, in collective_rpc
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/serial_utils.py", line 461, in run_method
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return func(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 340, in determine_available_memory
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     self.model_runner.profile_run()
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 4462, in profile_run
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     dummy_encoder_outputs = self.model.embed_multimodal(
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py", line 659, in embed_multimodal
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     vision_embeddings = self._process_image_input(multimodal_input)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py", line 587, in _process_image_input
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     vision_outputs = self.vision_tower(
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/models/timm_wrapper/modeling_timm_wrapper.py", line 248, in forward
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     last_hidden_state = self.timm_model.forward_features(pixel_values, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/mobilenetv5.py", line 580, in forward_features
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     x = blk(x)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     input = module(input)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/timm/models/_efficientnet_blocks.py", line 754, in forward
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     x = self.bn1(x)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return self._call_impl(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/norm_act.py", line 656, in forward
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     x = rms_norm2d(x, self.normalized_shape, self.weight, self.eps)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]   File "/home/ubuntu/.local/lib/python3.10/site-packages/timm/layers/fast_norm.py", line 184, in rms_norm2d
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866]     x = x * weight.reshape(1, -1, 1, 1)
(EngineCore_DP0 pid=1860120) ERROR 01-14 23:27:54 [core.py:866] torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.12 GiB. GPU 0 has a total capacity of 79.25 GiB of which 217.62 MiB is free. Process 738802 has 60.16 GiB memory in use. Process 1859697 has 4.19 GiB memory in use. Including non-PyTorch memory, this process has 14.67 GiB memory in use. Of the allocated memory 13.52 GiB is allocated by PyTorch, and 658.61 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(EngineCore_DP0 pid=1860120) Process EngineCore_DP0:
(EngineCore_DP0 pid=1860120) Traceback (most recent call last):
(EngineCore_DP0 pid=1860120)   File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=1860120)     self.run()
(EngineCore_DP0 pid=1860120)   File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=1860120)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 870, in run_engine_core
(EngineCore_DP0 pid=1860120)     raise e
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 857, in run_engine_core
(EngineCore_DP0 pid=1860120)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 637, in __init__
(EngineCore_DP0 pid=1860120)     super().__init__(
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=1860120)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 240, in _initialize_kv_caches
(EngineCore_DP0 pid=1860120)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=1860120)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/executor/uniproc_executor.py", line 75, in collective_rpc
(EngineCore_DP0 pid=1860120)     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/serial_utils.py", line 461, in run_method
(EngineCore_DP0 pid=1860120)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=1860120)     return func(*args, **kwargs)
(EngineCore_DP0 pid=1860120)   File "/home/ubuntu/.local/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 340, in determine_…719149 tokens truncated…      │ │
│ │              eng_identity = b'\x00\x00'                                  │ │
│ │                 eng_index = 0                                            │ │
│ │                    engine = <vllm.v1.engine.utils.CoreEngine object at   │ │
│ │                             0x7704fb36ffa0>                              │ │
│ │                    events = [(60, 4)]                                    │ │
│ │                  finished = {}                                           │ │
│ │          handshake_socket = <zmq.Socket(zmq.ROUTER) at 0x7704fb364b30    │ │
│ │                             closed>                                      │ │
│ │                  headless = False                                        │ │
│ │              init_message = b'\x83\xa9addresses\x85\xa6inputs\x91\xd9/i… │ │
│ │                     local = True                                         │ │
│ │               local_count = 1                                            │ │
│ │                       msg = {                                            │ │
│ │                             │   'status': 'HELLO',                       │ │
│ │                             │   'local': True,                           │ │
│ │                             │   'headless': False                        │ │
│ │                             }                                            │ │
│ │           parallel_config = ParallelConfig(                              │ │
│ │                             │   pipeline_parallel_size=1,                │ │
│ │                             │   tensor_parallel_size=1,                  │ │
│ │                             │   prefill_context_parallel_size=1,         │ │
│ │                             │   data_parallel_size=1,                    │ │
│ │                             │   data_parallel_size_local=1,              │ │
│ │                             │   data_parallel_rank=0,                    │ │
│ │                             │   data_parallel_rank_local=0,              │ │
│ │                             │   data_parallel_master_ip='127.0.0.1',     │ │
│ │                             │   data_parallel_rpc_port=29550,            │ │
│ │                             │   data_parallel_master_port=0,             │ │
│ │                             │   data_parallel_backend='mp',              │ │
│ │                             │   data_parallel_external_lb=False,         │ │
│ │                             │   data_parallel_hybrid_lb=False,           │ │
│ │                             │   enable_expert_parallel=False,            │ │
│ │                             │   enable_eplb=False,                       │ │
│ │                             │   eplb_config=EPLBConfig(                  │ │
│ │                             │   │   window_size=1000,                    │ │
│ │                             │   │   step_interval=3000,                  │ │
│ │                             │   │   num_redundant_experts=0,             │ │
│ │                             │   │   log_balancedness=False,              │ │
│ │                             │   │   use_async=False,                     │ │
│ │                             │   │   policy='default'                     │ │
│ │                             │   ),                                       │ │
│ │                             │   expert_placement_strategy='linear',      │ │
│ │                             │                                            │ │
│ │                             all2all_backend='allgather_reducescatter',   │ │
│ │                             │   max_parallel_loading_workers=None,       │ │
│ │                             │   disable_custom_all_reduce=False,         │ │
│ │                             │   enable_dbo=False,                        │ │
│ │                             │   dbo_decode_token_threshold=32,           │ │
│ │                             │   dbo_prefill_token_threshold=512,         │ │
│ │                             │                                            │ │
│ │                             disable_nccl_for_dp_synchronization=False,   │ │
│ │                             │   ray_workers_use_nsight=False,            │ │
│ │                             │   ray_runtime_env=None,                    │ │
│ │                             │   placement_group=None,                    │ │
│ │                             │   distributed_executor_backend='uni',      │ │
│ │                             │                                            │ │
│ │                             worker_cls='vllm.v1.worker.gpu_worker.Worke… │ │
│ │                             │   sd_worker_cls='auto',                    │ │
│ │                             │   worker_extension_cls='',                 │ │
│ │                             │   master_addr='127.0.0.1',                 │ │
│ │                             │   master_port=29501,                       │ │
│ │                             │   node_rank=0,                             │ │
│ │                             │   nnodes=1,                                │ │
│ │                             │   world_size=1,                            │ │
│ │                             │   rank=0,                                  │ │
│ │                             │   _data_parallel_master_port_list=[],      │ │
│ │                             │   decode_context_parallel_size=1,          │ │
│ │                             │   dcp_kv_cache_interleave_size=1,          │ │
│ │                             │   cp_kv_cache_interleave_size=1,           │ │
│ │                             │   _api_process_count=1,                    │ │
│ │                             │   _api_process_rank=0                      │ │
│ │                             )                                            │ │
│ │                    poller = <zmq.sugar.poll.Poller object at             │ │
│ │                             0x7704fb394a90>                              │ │
│ │              proc_manager = <vllm.v1.engine.utils.CoreEngineProcManager  │ │
│ │                             object at 0x7704fb394bb0>                    │ │
│ │           ready_msg_bytes = b'\x83\xa6status\xa5HELLO\xa5local\xc3\xa8h… │ │
│ │              remote_count = 0                                            │ │
│ │ remote_should_be_headless = True                                         │ │
│ │                  sentinel = 60                                           │ │
│ │             start_pending = [1, 0]                                       │ │
│ │                    status = 'HELLO'                                      │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Engine core initialization failed. See root cause above. Failed
core proc(s): {}

The above exception was the direct cause of the following exception:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ubuntu/maya-transcribe/maya_transcribe/cli.py:463 in bench_run         │
│                                                                              │
│   460 │   │   thresholds["require_ctc"] = require_ctc                        │
│   461 │   │   thresholds["min_ctc_margin"] = float(min_ctc_margin)           │
│   462 │                                                                      │
│ ❱ 463 │   results_path = run_pipeline(                                       │
│   464 │   │   segments_dir=audio_dir,                                        │
│   465 │   │   out_dir=out,                                                   │
│   466 │   │   lid_model_id=lid_model_id,                                     │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                      audio_dir = PosixPath('data/bench/maya_hf_hin_v2/a… │ │
│ │                      bench_dir = PosixPath('data/bench/maya_hf_hin_v2')  │ │
│ │                        ctc_map = None                                    │ │
│ │                      ctc_model = None                                    │ │
│ │                  ctc_model_map = None                                    │ │
│ │                     hypothesis = 'final'                                 │ │
│ │                         ic_map = None                                    │ │
│ │   indicconformer_default_model = None                                    │ │
│ │       indicconformer_model_map = None                                    │ │
│ │                   lid_model_id = 'facebook/mms-lid-256'                  │ │
│ │                          limit = None                                    │ │
│ │                 min_ctc_margin = 0.05                                    │ │
│ │                min_silence_sec = 0.15                                    │ │
│ │                  mms_asr_model = None                                    │ │
│ │                            out = PosixPath('outputs/bench/maya_hf_hin_v… │ │
│ │                      refs_path = PosixPath('data/bench/maya_hf_hin_v2/r… │ │
│ │                    require_ctc = True                                    │ │
│ │            require_lid_backend = True                                    │ │
│ │           silence_threshold_db = -40.0                                   │ │
│ │                   skip_whisper = True                                    │ │
│ │                         tf_map = None                                    │ │
│ │                     thresholds = {                                       │ │
│ │                                  │   'route_lang_from_path': True,       │ │
│ │                                  │   'route_lang_prefer_path': True      │ │
│ │                                  }                                       │ │
│ │                thresholds_json = None                                    │ │
│ │     transformers_whisper_model = []                                      │ │
│ │ transformers_whisper_model_map = None                                    │ │
│ │                   trim_silence = False                                   │ │
│ │                 vllm_asr_model = 'google/gemma-3n-E2B-it'                │ │
│ │    vllm_gpu_memory_utilization = 0.15                                    │ │
│ │             vllm_max_model_len = 4096                                    │ │
│ │                  whisper_model = 'large-v3'                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
│                                                                              │
│ /home/ubuntu/maya-transcribe/maya_transcribe/pipeline.py:185 in run_pipeline │
│                                                                              │
│   182 │   │   │   │   max_model_len=vllm_max_model_len,                      │
│   183 │   │   │   )                                                          │
│   184 │   │   except Exception as e:                                         │
│ ❱ 185 │   │   │   raise RuntimeError("Failed to initialize vLLM ASR backend. │
│   186 │                                                                      │
│   187 │   default_whisper_run_sampling = bool(whisper_backend is not None) a │
│   188 │   │   transformers_backends or transformers_router or gemini_backend │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │                       audio_files = [                                    │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │                                    │ │
│ │                                     PosixPath('data/bench/maya_hf_hin_v… │ │
│ │                                     │   ... +190                         │ │
│ │                                     ]                                    │ │
│ │                         ctc_model = None                                 │ │
│ │                     ctc_model_map = None                                 │ │
│ │                        ctc_scorer = None                                 │ │
│ │ extra_transformers_whisper_models = []                                   │ │
│ │                    gemini_backend = None                                 │ │
│ │                  gemini_hard_only = True                                 │ │
│ │          gemini_max_output_tokens = 512                                  │ │
│ │                      gemini_model = None                                 │ │
│ │                gemini_prompt_path = None                                 │ │
│ │      indicconformer_default_model = None                                 │ │
│ │          indicconformer_model_map = None                                 │ │
│ │             indicconformer_router = None                                 │ │
│ │                       lid_backend = <maya_transcribe.lid.MMSLID object   │ │
│ │                                     at 0x770716a64a90>                   │ │
│ │                      lid_model_id = 'facebook/mms-lid-256'               │ │
│ │                             limit = None                                 │ │
│ │                   min_silence_sec = 0.15                                 │ │
│ │                     mms_asr_model = None                                 │ │
│ │                       mms_backend = None                                 │ │
│ │                           out_dir = PosixPath('outputs/bench/maya_hf_hi… │ │
│ │                       rank_by_ctc = None                                 │ │
│ │               require_lid_backend = True                                 │ │
│ │                      segments_dir = PosixPath('data/bench/maya_hf_hin_v… │ │
│ │              silence_threshold_db = -40.0                                │ │
│ │                      skip_whisper = True                                 │ │
│ │                        thresholds = {                                    │ │
│ │                                     │   'route_lang_from_path': True,    │ │
│ │                                     │   'route_lang_prefer_path': True   │ │
│ │                                     }                                    │ │
│ │             transformers_backends = []                                   │ │
│ │               transformers_router = None                                 │ │
│ │  transformers_whisper_default_model None                                 │ │
│ │                                   =                                      │ │
│ │    transformers_whisper_model_map = None                                 │ │
│ │                      trim_silence = False                                │ │
│ │                    vllm_asr_model = 'google/gemma-3n-E2B-it'             │ │
│ │                      vllm_backend = None                                 │ │
│ │       vllm_gpu_memory_utilization = 0.15                                 │ │
│ │                vllm_max_model_len = 4096                                 │ │
│ │                   whisper_backend = None                                 │ │
│ │                     whisper_model = 'large-v3'                           │ │
│ ╰──────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Failed to initialize vLLM ASR backend.


**Investigating GPU OOM from vision tower in gemma3n**

The Gemma3n E2B failure stems from GPU out-of-memory caused by vision tower processing dummy images during profiling, despite the model being audio-focused. Next, I'll explore vLLM config and environment variables to disable or reduce vision modality and dummy image inputs to lower memory usage.


**Exploring dummy input config in vLLM**

I'll inspect vLLM’s dummy input builders and environment variables like `MM_DUMMY` to find ways to override or disable dummy image generation, aiming to reduce GPU memory usage during gemma3n profiling.


rg -n "Dummy" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal -S


Output:
Exit code: 0
Wall time: 0 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:7:from vllm.config.multimodal import BaseDummyOptions
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:18:    BaseDummyInputsBuilder,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:19:    DummyDecoderData,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:20:    DummyEncoderData,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:48:class DummyInputsBuilderFactory(Protocol[_I]):  # type: ignore[misc]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:51:    [`BaseDummyInputsBuilder`][vllm.multimodal.profiling.BaseDummyInputsBuilder]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:55:    def __call__(self, info: _I) -> BaseDummyInputsBuilder[_I]: ...
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:68:        dummy_inputs: BaseDummyInputsBuilder[_I],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:78:    dummy_inputs: DummyInputsBuilderFactory[_I]
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:99:    ) -> Mapping[str, BaseDummyOptions] | None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:194:        dummy_inputs: DummyInputsBuilderFactory[_I],
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:279:    ) -> DummyDecoderData:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/registry.py:312:    ) -> DummyEncoderData:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:13:    AudioDummyOptions,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:14:    BaseDummyOptions,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:15:    ImageDummyOptions,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:16:    VideoDummyOptions,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:49:class DummyEncoderData(NamedTuple):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:50:    """Dummy data used for profiling."""
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:55:class DummyDecoderData(NamedTuple):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:56:    """Dummy data used for profiling."""
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:66:class BaseDummyInputsBuilder(ABC, Generic[_I]):
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:89:        mm_options: Mapping[str, BaseDummyOptions] | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:109:        mm_options: Mapping[str, BaseDummyOptions] | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:138:        overrides: AudioDummyOptions | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:160:        overrides: ImageDummyOptions | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:193:        overrides: VideoDummyOptions | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:247:    def dummy_inputs(self) -> BaseDummyInputsBuilder[_I]:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:257:        mm_options: Mapping[str, BaseDummyOptions] | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:289:        mm_options: Mapping[str, BaseDummyOptions] | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:290:    ) -> DummyEncoderData:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:305:        return DummyEncoderData(encoder_prompt_token_ids)
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:311:        mm_options: Mapping[str, BaseDummyOptions] | None = None,
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:312:    ) -> DummyDecoderData:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/profiling.py:321:        return DummyDecoderData(
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py:59:    from .profiling import BaseDummyInputsBuilder
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py:1289:        dummy_inputs: "BaseDummyInputsBuilder[_I]",
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/multimodal/processing.py:1605:        [`DummyInputsBuilder`][vllm.multimodal.profiling.BaseDummyInputsBuilder]


**Searching for dummy-related env variables**


rg -n "DUMMY|dummy" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/envs.py -S


Output:
Exit code: 0
Wall time: 0 seconds
Output:
151:    VLLM_RANDOMIZE_DP_DUMMY_INPUTS: bool = False
834:        "VLLM_TEST_FORCE_LOAD_FORMAT", "dummy"
1100:    # Randomize inputs during dummy runs when using Data Parallel
1101:    "VLLM_RANDOMIZE_DP_DUMMY_INPUTS": lambda: os.environ.get(
1102:        "VLLM_RANDOMIZE_DP_DUMMY_INPUTS", "0"
1653:        "VLLM_RANDOMIZE_DP_DUMMY_INPUTS",


**Investigating dummy count config in vLLM**

I'm checking for `mm_dummy_options` or related config in vLLM, focusing on `entrypoints/llm.py` around `mm_processor_kwargs` to understand dummy counts handling.


rg -n "dummy" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints -S


Output:
Exit code: 0
Wall time: 0 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py:224:        # Don't keep the dummy data in memory
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/launcher.py:91:    async def dummy_shutdown() -> None:
/home/ubuntu/.local/lib/python3.10/site-packages/vllm/entrypoints/launcher.py:99:        return dummy_shutdown()


**Inspecting LLM initialization parameters**

I'm checking the LLM constructor for any parameters related to `mm_processor_kwargs` or multi-modal processing to understand how to pass such arguments during initialization.


python3 - <<'PY'
import inspect
from vllm import LLM
sig = inspect.signature(LLM)
print(sig)
PY


Output:
Exit code: 0
Wall time: 11.2 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:29:12.312207: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(model: str, *, runner: Literal['auto', 'generate', 'pooling', 'draft'] = 'auto', convert: Literal['auto', 'none', 'embed', 'classify', 'reward'] = 'auto', tokenizer: str | None = None, tokenizer_mode: Union[Literal['auto', 'hf', 'slow', 'mistral', 'deepseek_v32'], str] = 'auto', skip_tokenizer_init: bool = False, trust_remote_code: bool = False, allowed_local_media_path: str = '', allowed_media_domains: list[str] | None = None, tensor_parallel_size: int = 1, dtype: Literal['auto', 'half', 'float16', 'bfloat16', 'float', 'float32'] = 'auto', quantization: Optional[Literal['awq', 'deepspeedfp', 'tpu_int8', 'fp8', 'ptpc_fp8', 'fbgemm_fp8', 'fp_quant', 'modelopt', 'modelopt_fp4', 'bitblas', 'gguf', 'gptq_marlin_24', 'gptq_marlin', 'gptq_bitblas', 'awq_marlin', 'gptq', 'compressed-tensors', 'bitsandbytes', 'hqq', 'experts_int8', 'ipex', 'quark', 'moe_wna16', 'torchao', 'auto-round', 'rtn', 'inc', 'mxfp4', 'petit_nvfp4', 'cpu_gptq', 'cpu_awq']] = None, revision: str | None = None, tokenizer_revision: str | None = None, seed: int = 0, gpu_memory_utilization: float = 0.9, swap_space: float = 4, cpu_offload_gb: float = 0, enforce_eager: bool = False, disable_custom_all_reduce: bool = False, hf_token: bool | str | None = None, hf_overrides: dict[str, typing.Any] | collections.abc.Callable[[typing.Any], typing.Any] | None = None, mm_processor_kwargs: dict[str, typing.Any] | None = None, pooler_config: vllm.config.pooler.PoolerConfig | None = None, structured_outputs_config: dict[str, typing.Any] | vllm.config.structured_outputs.StructuredOutputsConfig | None = None, profiler_config: dict[str, typing.Any] | vllm.config.profiler.ProfilerConfig | None = None, kv_cache_memory_bytes: int | None = None, compilation_config: int | dict[str, typing.Any] | vllm.config.compilation.CompilationConfig | None = None, logits_processors: list[str | type[vllm.v1.sample.logits_processor.interface.LogitsProcessor]] | None = None, **kwargs: Any) -> None


**Investigating dummy input customization**

I'm examining whether `mm_processor_kwargs` or related config can limit dummy input generation to audio-only by inspecting `BaseDummyInputsBuilder` and dummy options in multimodal processing and profiling modules.


python3 - <<'PY'
import inspect
from vllm.multimodal import profiling
src = inspect.getsource(profiling.BaseDummyInputsBuilder)
print(src[:1200])
PY


Output:
Exit code: 0
Wall time: 10.1 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:29:43.319064: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
class BaseDummyInputsBuilder(ABC, Generic[_I]):
    """
    Abstract base class that constructs the dummy data to profile
    multi-modal models.
    """

    def __init__(self, info: _I) -> None:
        super().__init__()

        self.info = info

    @abstractmethod
    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
        """
        Build the text input corresponding to `mm_counts`.
        """
        raise NotImplementedError

    @abstractmethod
    def get_dummy_mm_data(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int],
        mm_options: Mapping[str, BaseDummyOptions] | None = None,
    ) -> MultiModalDataDict:
        """
        Build the multimodal input which, after processing, results in
        the maximum possible number of placeholder tokens.

        Args:
            seq_len: Sequence length
            mm_counts: Count of items per modality
            mm_options: Configurable options per modality (optional).
                       If None, use model defaults for backward compatibility.
                       If provided, models can use these to customize dummy
                       data generation.
        """


**Searching variable initialization**

I'm looking up how the `mm_counts` variable is set in the profiling module to understand its origin and usage.


python3 - <<'PY'
import inspect
from vllm.multimodal import profiling
src = inspect.getsource(profiling.BaseDummyInputsBuilder)
# print method that uses mm_counts
for name in ['get_dummy_inputs', 'get_encoder_dummy_data', 'get_decoder_dummy_data']:
    if hasattr(profiling.BaseDummyInputsBuilder, name):
        print('\n', name)
        print(inspect.getsource(getattr(profiling.BaseDummyInputsBuilder, name))[:800])
PY


Output:
Exit code: 0
Wall time: 10.1 seconds
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:30:08.902331: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)


**Inspecting dummy input builders in gemma3n_mm.py**


rg -n "Dummy" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py -n


Output:
Exit code: 0
Wall time: 0 seconds
Output:
22:from vllm.config.multimodal import BaseDummyOptions
53:from vllm.multimodal.profiling import BaseDummyInputsBuilder
156:class Gemma3nDummyInputsBuilder(BaseDummyInputsBuilder[Gemma3nProcessingInfo]):
171:        mm_options: Optional[Mapping[str, BaseDummyOptions]] = None,
461:    dummy_inputs=Gemma3nDummyInputsBuilder,


Summary: auto


sed -n '140,220p' /home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models/gemma3n_mm.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    ) -> str:
        """
        Get the replacement text for audio tokens.

        For Gemma3n, this should return the full_audio_sequence which includes
        BOA token, repeated audio tokens, and EOA token.
        """
        if processor is None:
            processor = self.get_hf_processor()

        # Return the full audio sequence as defined by the processor
        return PromptUpdateDetails.select_token_id(
            processor.full_audio_sequence, processor.audio_token_id
        )

class Gemma3nDummyInputsBuilder(BaseDummyInputsBuilder[Gemma3nProcessingInfo]):
    def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str:
        num_images = mm_counts.get("image", 0)
        num_audios = mm_counts.get("audio", 0)

        processor = self.info.get_hf_processor()
        image_token = processor.image_token
        audio_token = processor.audio_token

        return image_token * num_images + audio_token * num_audios

    def get_dummy_mm_data(
        self,
        seq_len: int,
        mm_counts: Mapping[str, int],
        mm_options: Optional[Mapping[str, BaseDummyOptions]] = None,
    ) -> MultiModalDataDict:
        num_images = mm_counts.get("image", 0)
        num_audios = mm_counts.get("audio", 0)
        processor = self.info.get_hf_processor()
        audio_feature_extractor: Gemma3nAudioFeatureExtractor = (
            processor.feature_extractor
        )
        audio_len = audio_feature_extractor.fft_length
        image_processor: SiglipImageProcessorFast = processor.image_processor
        img_width = image_processor.size.get("width", 224)
        img_height = image_processor.size.get("height", 224)

        image_overrides = mm_options.get("image") if mm_options else None
        audio_overrides = mm_options.get("audio") if mm_options else None

        return {
            "image": self._get_dummy_images(
                width=img_width,
                height=img_height,
                num_images=num_images,
                overrides=image_overrides,
            ),
            "audio": self._get_dummy_audios(
                length=audio_len, num_audios=num_audios, overrides=audio_overrides
            ),
        }

class Gemma3nMultiModalProcessor(BaseMultiModalProcessor[Gemma3nProcessingInfo]):
    def _get_data_parser(self) -> MultiModalDataParser:
        feature_extractor = self.info.get_hf_processor().feature_extractor
        return MultiModalDataParser(target_sr=feature_extractor.sampling_rate)

    def _call_hf_processor(
        self,
        prompt: str,
        mm_data: Mapping[str, object],
        mm_kwargs: Mapping[str, object],
        tok_kwargs: Mapping[str, object],
    ) -> BatchFeature:
        # HF Transformers audio processor no longer accepts `audios` key.
        # We pop `audios` and replace it with `audio` key to suppress
        # the warning.
        if "audios" in mm_data:
            mm_data["audio"] = mm_data.pop("audios")
        processed_outputs = super()._call_hf_processor(
            prompt,
            mm_data,
            mm_kwargs,


**Tracing mm_counts origin**

The current guess is that default mm_counts come from MULTIMODAL_REGISTRY defaults causing 32 image slots and large memory use; next is to inspect profiling.py to understand how mm_counts are derived and find how to override them for dummy builder.


python3 - <<'PY'
import inspect
from vllm.multimodal import profiling
src = inspect.getsource(profiling)
for line in src.split('\n'):
    if 'mm_counts' in line and 'def' not in line and 'mm_counts' in line:
        pass
# Print section around get_decoder_dummy_data
idx = src.find('def get_decoder_dummy_data')
print(src[idx:idx+1200])
PY