Output:
Exit code: 0
Wall time: 325.6 seconds
Total output lines: 899
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:46:02.128761: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-14 23:46:13 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'disable_log_stats': True, 'model': 'mistralai/Voxtral-Small-24B-2507'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-14 23:46:14 [model.py:514] Resolved architecture: VoxtralForConditionalGeneration
INFO 01-14 23:46:14 [model.py:1661] Using max model len 4096
INFO 01-14 23:46:14 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
[2026-01-14 23:46:14] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
[2026-01-14 23:46:14] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
WARNING 01-14 23:46:16 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-14 23:46:20.924590: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:25 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='mistralai/Voxtral-Small-24B-2507', speculative_config=None, tokenizer='mistralai/Voxtral-Small-24B-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=mistralai/Voxtral-Small-24B-2507, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1882420) [2026-01-14 23:46:26] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(EngineCore_DP0 pid=1882420) [2026-01-14 23:46:26] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:27 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:59849 backend=nccl
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:27 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:28 [gpu_model_runner.py:3562] Starting to load model mistralai/Voxtral-Small-24B-2507...
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:29 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:29 [layer.py:477] Using AttentionBackendEnum.FLASH_ATTN for MultiHeadAttention in multimodal encoder.
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:30 [weight_utils.py:527] No consolidated.safetensors.index.json found in remote.
(EngineCore_DP0 pid=1882420)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=1882420)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00,  7.26s/it]
(EngineCore_DP0 pid=1882420)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:07<00:00,  7.26s/it]
(EngineCore_DP0 pid=1882420)
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:38 [default_loader.py:308] Loading weights took 8.86 seconds
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:40 [gpu_model_runner.py:3659] Model loading took 45.2032 GiB memory and 9.564605 seconds
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:40 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 2 audio items of the maximum feature size.
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:47 [backends.py:643] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/936cd8900f/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:47 [backends.py:703] Dynamo bytecode transform time: 6.07 s
(EngineCore_DP0 pid=1882420) INFO 01-14 23:46:53 [backends.py:261] Cache the graph of compile range (1, 8192) for later use
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:06 [backends.py:278] Compiling a graph for compile range (1, 8192) takes 15.57 s
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:06 [monitor.py:34] torch.compile takes 21.64 s in total
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:08 [gpu_worker.py:375] Available KV cache memory: 24.13 GiB
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:09 [kv_cache_utils.py:1291] GPU KV cache size: 158,144 tokens
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:09 [kv_cache_utils.py:1296] Maximum concurrency for 4,096 tokens per request: 38.61x
(EngineCore_DP0 pid=1882420)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   2%|▏         | 1/51 [00:00<00:06,  8.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   4%|▍         | 2/51 [00:00<00:05,  8.51it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   6%|▌         | 3/51 [00:00<00:05,  8.61it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   8%|▊         | 4/51 [00:00<00:05,  8.69it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  10%|▉         | 5/51 [00:00<00:05,  8.73it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  12%|█▏        | 6/51 [00:00<00:05,  8.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  14%|█▎        | 7/51 [00:00<00:05,  8.80it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 8/51 [00:00<00:04,  8.82it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|█▉        | 10/51 [00:01<00:04,  9.37it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  24%|██▎       | 12/51 [00:01<00:04,  9.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 14/51 [00:01<00:03,  9.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 16/51 [00:01<00:03, 10.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  35%|███▌      | 18/51 [00:01<00:03, 10.96it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  39%|███▉      | 20/51 [00:01<00:02, 11.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 22/51 [00:02<00:02, 12.31it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  47%|████▋     | 24/51 [00:02<00:02, 12.75it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  51%|█████     | 26/51 [00:02<00:01, 13.09it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 28/51 [00:02<00:01, 13.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  59%|█████▉    | 30/51 [00:02<00:01, 13.49it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  63%|██████▎   | 32/51 [00:02<00:01, 13.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:01, 14.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████   | 36/51 [00:03<00:01, 14.63it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  75%|███████▍  | 38/51 [00:03<00:00, 14.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  78%|███████▊  | 40/51 [00:03<00:00, 15.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 42/51 [00:03<00:00, 15.76it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  86%|████████▋ | 44/51 [00:03<00:00, 16.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  90%|█████████ | 46/51 [00:03<00:00, 16.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  94%|█████████▍| 48/51 [00:03<00:00, 16.77it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  98%|█████████▊| 50/51 [00:03<00:00, 17.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:04<00:00, 12.73it/s]
(EngineCore_DP0 pid=1882420)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   6%|▌         | 2/35 [00:00<00:02, 12.03it/s]
Capturing CUDA graphs (decode, FULL):  11%|█▏        | 4/35 [00:00<00:02, 12.86it/s]
Capturing CUDA graphs (decode, FULL):  17%|█▋        | 6/35 [00:00<00:02, 13.26it/s]
Capturing CUDA graphs (decode, FULL):  23%|██▎       | 8/35 [00:00<00:02, 13.46it/s]
Capturing CUDA graphs (decode, FULL):  29%|██▊       | 10/35 [00:00<00:01, 13.67it/s]
Capturing CUDA graphs (decode, FULL):  34%|███▍      | 12/35 [00:00<00:01, 13.76it/s]
Capturing CUDA graphs (decode, FULL):  40%|████      | 14/35 [00:01<00:01, 13.85it/s]
Capturing CUDA graphs (decode, FULL):  46%|████▌     | 16/35 [00:01<00:01, 13.90it/s]
Capturing CUDA graphs (decode, FULL):  51%|█████▏    | 18/35 [00:01<00:01, 14.63it/s]
Capturing CUDA graphs (decode, FULL):  57%|█████▋    | 20/35 [00:01<00:00, 15.18it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 22/35 [00:01<00:00, 15.65it/s]
Capturing CUDA graphs (decode, FULL):  69%|██████▊   | 24/35 [00:01<00:00, 16.01it/s]
Capturing CUDA graphs (decode, FULL):  74%|███████▍  | 26/35 [00:01<00:00, 16.43it/s]
Capturing CUDA graphs (decode, FULL):  80%|████████  | 28/35 [00:01<00:00, 17.03it/s]
Capturing CUDA graphs (decode, FULL):  86%|████████▌ | 30/35 [00:01<00:00, 17.55it/s]
Capturing CUDA graphs (decode, FULL):  91%|█████████▏| 32/35 [00:02<00:00, 17.92it/s]
Capturing CUDA graphs (decode, FULL):  97%|█████████▋| 34/35 [00:02<00:00, 18.24it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 35/35 [00:02<00:00, 15.59it/s]
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:16 [gpu_model_runner.py:4587] Graph capturing finished in 8 secs, took 0.71 GiB
(EngineCore_DP0 pid=1882420) INFO 01-14 23:47:16 [core.py:259] init engine (profile, create kv cache, warmup model) took 36.62 seconds
INFO 01-14 23:47:17 [llm.py:360] Supported tasks: ['generate', 'transcription']

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:01<00:00,  1.36s/it]
Adding requests: 100%|██████████| 1/1 [00:01<00:00,  1.36s/it]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 521.13 toks/s, output: 28.57 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 521.13 toks/s, output: 28.57 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 521.13 toks/s, output: 28.57 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 329.12it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.01it/s, est. speed input: 388.63 toks/s, output: 30.44 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.01it/s, est. speed input: 388.63 toks/s, output: 30.44 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.01it/s, est. speed input: 388.63 toks/s, output: 30.44 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 142.70it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.44s/it, est. speed input: 266.23 toks/s, output: 31.28 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.44s/it, est. speed input: 266.23 toks/s, output: 31.28 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.44s/it, est. speed input: 266.23 toks/s, output: 31.28 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 60.69it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.01s/it, est. speed input: 379.49 toks/s, output: 30.72 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.01s/it, est. speed input: 379.49 toks/s, output: 30.72 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.01s/it, est. speed input: 379.49 toks/s, output: 30.72 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 145.84it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.69it/s, est. speed input: 648.88 toks/s, output: 28.80 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.69it/s, est. speed input: 648.88 toks/s, output: 28.80 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.69it/s, est. speed input: 648.88 toks/s, output: 28.80 toks…41852 tokens truncated…pts: 100%|██████████| 1/1 [00:00<00:00,  1.41it/s, est. speed input: 540.39 toks/s, output: 29.63 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 350.46it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.35it/s, est. speed input: 518.70 toks/s, output: 29.79 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.35it/s, est. speed input: 518.70 toks/s, output: 29.79 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.35it/s, est. speed input: 518.70 toks/s, output: 29.79 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 373.62it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.37s/it, est. speed input: 280.43 toks/s, output: 31.48 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.37s/it, est. speed input: 280.43 toks/s, output: 31.48 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.37s/it, est. speed input: 280.43 toks/s, output: 31.48 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 317.17it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 498.85 toks/s, output: 29.96 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 498.85 toks/s, output: 29.96 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 498.85 toks/s, output: 29.96 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 342.70it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 593.78 toks/s, output: 29.46 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 593.78 toks/s, output: 29.46 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 593.78 toks/s, output: 29.46 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 132.27it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.00s/it, est. speed input: 381.68 toks/s, output: 30.89 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.00s/it, est. speed input: 381.68 toks/s, output: 30.89 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.00s/it, est. speed input: 381.68 toks/s, output: 30.89 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 145.41it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.70it/s, est. speed input: 649.54 toks/s, output: 28.83 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.70it/s, est. speed input: 649.54 toks/s, output: 28.83 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.69it/s, est. speed input: 649.54 toks/s, output: 28.83 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 206.27it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.81it/s, est. speed input: 691.80 toks/s, output: 28.90 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.81it/s, est. speed input: 691.80 toks/s, output: 28.90 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.81it/s, est. speed input: 691.80 toks/s, output: 28.90 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 318.47it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 499.45 toks/s, output: 29.99 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 499.45 toks/s, output: 29.99 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 499.45 toks/s, output: 29.99 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 282.33it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.06it/s, est. speed input: 406.19 toks/s, output: 30.75 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.06it/s, est. speed input: 406.19 toks/s, output: 30.75 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.06it/s, est. speed input: 406.19 toks/s, output: 30.75 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 78.22it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.71it/s, est. speed input: 654.64 toks/s, output: 29.06 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.71it/s, est. speed input: 654.64 toks/s, output: 29.06 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.71it/s, est. speed input: 654.64 toks/s, output: 29.06 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 69.91it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.47it/s, est. speed input: 562.77 toks/s, output: 29.39 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.47it/s, est. speed input: 562.77 toks/s, output: 29.39 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.47it/s, est. speed input: 562.77 toks/s, output: 29.39 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 130.47it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 619.71 toks/s, output: 29.12 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 619.71 toks/s, output: 29.12 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 619.71 toks/s, output: 29.12 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 193.05it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 594.52 toks/s, output: 29.49 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 594.52 toks/s, output: 29.49 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.55it/s, est. speed input: 594.52 toks/s, output: 29.49 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 193.74it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.16it/s, est. speed input: 445.64 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.16it/s, est. speed input: 445.64 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.16it/s, est. speed input: 445.64 toks/s, output: 30.25 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 302.01it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.31it/s, est. speed input: 499.92 toks/s, output: 30.02 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.31it/s, est. speed input: 499.92 toks/s, output: 30.02 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.30it/s, est. speed input: 499.92 toks/s, output: 30.02 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 287.30it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.94 toks/s, output: 29.18 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.94 toks/s, output: 29.18 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.94 toks/s, output: 29.18 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 291.60it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 519.55 toks/s, output: 29.84 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 519.55 toks/s, output: 29.84 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.36it/s, est. speed input: 519.55 toks/s, output: 29.84 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 254.46it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.13 toks/s, output: 29.14 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.13 toks/s, output: 29.14 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.62it/s, est. speed input: 620.13 toks/s, output: 29.14 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 167.04it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.21it/s, est. speed input: 464.18 toks/s, output: 30.30 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.21it/s, est. speed input: 464.18 toks/s, output: 30.30 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.21it/s, est. speed input: 464.18 toks/s, output: 30.30 toks/s]
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:03:59
[rank0]:[W114 23:51:17.889673259 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/results.jsonl
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/summary.json
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 0.14866798479298474,
  "cer_mean": 0.05488169663638188,
  "wer_p50": 0.125,
  "wer_p90": 0.3333333333333333,
  "wer_p95": 0.42857142857142855,
  "cer_p50": 0.034482758620689655,
  "cer_p90": 0.13793103448275862,
  "cer_p95": 0.1891891891891892,
  "segment_error_rate": 0.725,
  "pct_wer_gt_5pct": 0.725,
  "pct_wer_gt_10pct": 0.57,
  "wer_by_language_mean": {
    "hin": 0.14866798479298474
  },
  "wer_by_decision_mean": {
    "accept": 0.14720970432649266,
    "human": 0.1518391661248804
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.07777777777777778,
    "10-30s": 0.2039377289377289,
    "3-10s": 0.1478243830637447
  },
  "cer_by_language_mean": {
    "hin": 0.05488169663638188
  },
  "cer_by_decision_mean": {
    "accept": 0.055736333845422516,
    "human": 0.053023199848468025
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.022058823529411763,
    "10-30s": 0.06514125952230941,
    "3-10s": 0.05514347849457581
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.14866798479298474,
    "selection_gap_wer_mean": 0.0,
    "selection_gap_wer_p90": 0.0
  }
}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Summarizing benchmark results**

The Voxtral-Small model achieved a WER of 14.87% on the Hindi dataset, improving significantly over Voxtral-Mini's 19.3% and Gemma3n E2B's 137.6%. This run used vLLM with high GPU utilization. Key metrics include segment error rate of 0.725 and pct_wer_gt_5pct at 0.725. Results are stored at `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`. Next, I'll suggest further optimization leveraging more GPU resources as requested.


**Providing detailed result context and next steps**

The 14.87% WER for Voxtral-Small improves over Voxtral-Mini (19.3%) and baseline gating (23.7%) but remains above the desired <5% target. To reach <5%, fine-tuning, domain adaptation, or specialized ASR models like Whisper large-v3 may be needed. Suggested next steps: 1) run Voxtral-Small on other languages/datasets; 2) try larger models if available; 3) explore LLM alternatives or specialized ASR models outside generative LLMs. The pipeline used only vLLM with skip-whisper enabled and high GPU utilization (0.9). Results are at `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`.

Summary comparison (WER mean):
```
Model             | WER mean | Notes
------------------|----------|------------------------------
Voxtral-Small 24B | 14.87%   | vLLM run with high GPU usage
Voxtral-Mini 3B   | 19.3%    | Earlier vLLM run
Gemma-3n E2B      | 137.6%   | Poor performance
Baseline (Vaani)  | 23.7%    | Gating baseline, relaxed criteria
```


rg -n "skip_whisper|vllm_asr_model" -S maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
67:    skip_whisper: bool = False,
68:    vllm_asr_model: str | None = None,
122:    if not skip_whisper and whisper_model:
177:    if vllm_asr_model:
180:                model_id=vllm_asr_model,
529:                    "vllm_asr_model": vllm_asr_model,
532:                    "skip_whisper": skip_whisper,


Summary: auto


**Previewing code snippet**


sed -n '110,220p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
    lid_backend: MMSLID | None
    try:
        lid_backend = MMSLID(model_id=lid_model_id)
    except Exception as e:
        if require_lid_backend:
            raise RuntimeError(
                "Failed to initialize MMS-LID. Install torch/transformers deps (pip install -e '.[torch]') "
                "or run with require_lid_backend=False."
            ) from e
        lid_backend = None

    whisper_backend: FasterWhisperASR | None = None
    if not skip_whisper and whisper_model:
        whisper_backend = FasterWhisperASR(model_size_or_path=whisper_model)

    extra_transformers_whisper_models = extra_transformers_whisper_models or []
    transformers_backends: list[TransformersWhisperASR] = []
    for model_id in extra_transformers_whisper_models:
        transformers_backends.append(TransformersWhisperASR(model_id=model_id))

    transformers_router = None
    if transformers_whisper_model_map:
        transformers_router = TransformersWhisperASRRouter(
            model_map=transformers_whisper_model_map, default_model=transformers_whisper_default_model
        )

    # MMS ASR backend for CTC-based diversity (per master.md: "Optional diversity: MMS ASR").
    mms_backend: MMSASR | None = None
    if mms_asr_model:
        try:
            mms_backend = MMSASR(model_id=mms_asr_model)
        except Exception:
            pass  # MMS optional; continue if unavailable.

    # IndicConformer router for architecture diversity (per master.md: "IndicConformer-600M multilingual").
    indicconformer_router: IndicConformerASRRouter | None = None
    if indicconformer_model_map or indicconformer_default_model:
        try:
            indicconformer_router = IndicConformerASRRouter(
                model_map=indicconformer_model_map,
                default_model=indicconformer_default_model,
            )
        except Exception:
            pass  # IndicConformer optional; continue if NeMo unavailable.

    ctc_scorer = None
    rank_by_ctc = None
    if ctc_model or ctc_model_map:
        from maya_transcribe.ctc_fit import CTCFitScorer, rank_candidates_by_ctc

        ctc_scorer = CTCFitScorer(model_map=ctc_model_map, default_model=ctc_model)
        rank_by_ctc = rank_candidates_by_ctc

    gemini_backend = None
    if gemini_model:
        try:
            from maya_transcribe.gemini_asr import GeminiASR, GeminiConfig, load_prompt_template

            prompt = load_prompt_template(gemini_prompt_path or Path("Prompt.md"))
            gemini_backend = GeminiASR(
                config=GeminiConfig(model=gemini_model, max_output_tokens=gemini_max_output_tokens),
                prompt_template=prompt,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize Gemini backend (check GEMINI_KEY and google-genai).") from e

    vllm_backend: VllmTranscriptionASR | None = None
    if vllm_asr_model:
        try:
            vllm_backend = VllmTranscriptionASR(
                model_id=vllm_asr_model,
                gpu_memory_utilization=vllm_gpu_memory_utilization,
                max_model_len=vllm_max_model_len,
            )
        except Exception as e:
            raise RuntimeError("Failed to initialize vLLM ASR backend.") from e

    default_whisper_run_sampling = bool(whisper_backend is not None) and not bool(
        transformers_backends or transformers_router or gemini_backend is not None
    )

    results_path = out_dir / "results.jsonl"
    summary_path = out_dir / "summary.json"

    counts = {"accept": 0, "human": 0, "reject": 0, "error": 0}
    timings_total: list[float] = []

    with results_path.open("w", encoding="utf-8") as f:
        for src in track(audio_files, description="Transcribing"):
            ident = segment_identity(src, root=segments_dir)
            segment_id = ident.segment_id
            t0 = time.perf_counter()

            try:
                # 1) Preprocess
                t_pre0 = time.perf_counter()
                src_info = ffprobe(src)
                wav_path = out_dir / "audio" / f"{segment_id}.wav"
                normalize_audio(
                    src,
                    wav_path,
                    trim_silence=trim_silence,
                    silence_threshold_db=silence_threshold_db,
                    min_silence_sec=min_silence_sec,
                )
                info = ffprobe(wav_path)
                qc = compute_qc(wav_path)
                t_pre1 = time.perf_counter()

                # 2) LID
                t_lid0 = time.perf_counter()


**Planning search for vLLM backend usage**


rg -n "vllm_backend" -S maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
176:    vllm_backend: VllmTranscriptionASR | None = None
179:            vllm_backend = VllmTranscriptionASR(
301:                if vllm_backend is not None:
303:                        vllm_res = vllm_backend.transcribe(wav_path, language=route_lang)
306:                        asr_errors[vllm_backend.name] = f"{type(e).__name__}:{e}"


**Reviewing code context around target lines**


sed -n '260,340p' maya_transcribe/pipeline.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
                if whisper_backend is not None:
                    try:
                        det = whisper_backend.transcribe(
                            wav_path, language=route_lang, beam_size=5, best_of=5, temperature=0.0
                        )
                        primary_asr_key = f"{det.model}|det"
                        asr_results[primary_asr_key] = det
                    except Exception as e:
                        asr_errors[f"{whisper_backend.name}|det"] = f"{type(e).__name__}:{e}"

                run_sampling = bool(thresholds.get("whisper_run_sampling", default_whisper_run_sampling))
                if run_sampling and whisper_backend is not None:
                    try:
                        samp = whisper_backend.transcribe(
                            wav_path, language=route_lang, beam_size=5, best_of=10, temperature=0.2
                        )
                        asr_results[f"{samp.model}|samp_t0.2"] = samp
                    except Exception as e:
                        asr_errors[f"{whisper_backend.name}|samp_t0.2"] = f"{type(e).__name__}:{e}"

                for tf_backend in transformers_backends:
                    try:
                        res = tf_backend.transcribe(wav_path, language=route_lang, num_beams=5, max_new_tokens=256)
                        asr_results[res.model] = res
                    except Exception as e:
                        asr_errors[tf_backend.name] = f"{type(e).__name__}:{e}"

                if transformers_router is not None:
                    try:
                        routed = transformers_router.transcribe(
                            wav_path,
                            language=route_lang,
                            num_beams=5,
                            max_new_tokens=256,
                        )
                        if routed is not None:
                            asr_results[routed.model] = routed
                            router_key = routed.model
                    except Exception as e:
                        asr_errors[f"transformers_router:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if vllm_backend is not None:
                    try:
                        vllm_res = vllm_backend.transcribe(wav_path, language=route_lang)
                        asr_results[vllm_res.model] = vllm_res
                    except Exception as e:
                        asr_errors[vllm_backend.name] = f"{type(e).__name__}:{e}"

                # MMS ASR for CTC-based diversity (per master.md plan).
                if mms_backend is not None:
                    try:
                        mms_res = mms_backend.transcribe(wav_path, language=route_lang)
                        asr_results[mms_res.model] = mms_res
                    except Exception as e:
                        asr_errors[f"mms-ctc:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                # IndicConformer for Conformer architecture diversity (per master.md plan).
                if indicconformer_router is not None:
                    try:
                        conformer_res = indicconformer_router.transcribe(wav_path, language=route_lang)
                        if conformer_res is not None:
                            asr_results[conformer_res.model] = conformer_res
                    except Exception as e:
                        asr_errors[f"indicconformer:{route_lang or 'none'}"] = f"{type(e).__name__}:{e}"

                if primary_asr_key is None and asr_results:
                    primary_asr_key = sorted(asr_results.keys())[0]
                if not asr_results:
                    raise RuntimeError(f"no_asr_hypotheses:{asr_errors or 'unknown'}")

                # Gemini: optional extra hypothesis (prefer running only on "hard" segments).
                if gemini_backend is not None:
                    run_gemini = True
                    if gemini_hard_only:
                        base_for_heur = det or (next(iter(asr_results.values())) if asr_results else None)
                        det_sum = _asr_summary(base_for_heur) if base_for_heur is not None else {}
                        avg_lp = det_sum.get("avg_logprob_mean")
                        comp = det_sum.get("compression_ratio_mean")
                        no_speech = det_sum.get("no_speech_prob_mean")
                        run_gemini = bool(
                            route_lang is None


**Summarizing vLLM evaluation results**

The vLLM model run alone with skip-whisper yields a mean WER of 14.87% and CER of 5.49%, consistent with final "best" hypotheses after gating decisions. GPU utilization was about 0.9 with the 45.2 GiB model loaded. I'll present these concise key findings to the user, omitting large logs.


**Comparing Voxtral-Small results with prior runs**

Ran Voxtral-Small 24B with vLLM at 0.9 GPU utilization on `data/bench/maya_hf_hin_v2`, achieving mean WER 14.87% (CER 5.49%), with 72.5% of segments having WER > 5%, still far from the <5% target.

```
Model (vLLM)           | WER mean | CER mean | Notes
----------------------|----------|----------|-------------------------
Voxtral-Small 24B     | 14.87%   | 5.49%    | 45.2 GiB GPU memory, 0.9 utilization
Voxtral-Mini 3B       | 19.32%   | 7.85%    | Prior run
Gemma-3n E2B          | 137.6%   | 105.2%   | Prior run, poor performance
Baseline + Vaani (relaxed gating) | 23.7%   | -        | From earlier pipeline
```

Evaluation outputs are stored under:

- `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`
- `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm/eval.json`
- `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json`

Next steps to consider:
1) Run Voxtral-Small on other language benchmarks you specify.
2) Try prompt tuning or forced script outputs to improve transcription quality.
3) Evaluate larger or alternative LLM models like Voxtral-Large or Gemma-3n E4B if available.
4) For <5% WER, consider domain adaptation or specialized ASR models alongside LLMs.

Which other language benchmark datasets should I run next?


Ran Voxtral‑Small 24B with vLLM at high GPU utilization (0.9) on the Hindi bench to take advantage of the free 80GB. It finished cleanly and is better than Voxtral‑Mini, but still far from the <5% WER target.

**Voxtral‑Small Results**
- WER mean: 14.87% (CER mean: 5.49%)
- Accept WER: 14.72%, Human WER: 15.18%
- % segments >5% WER: 72.5%
- Eval file: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`

**Comparison (Hindi bench, vLLM)**
- Voxtral‑Small‑24B: 14.87% WER
- Voxtral‑Mini‑3B: 19.32% WER (`outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm/eval.json`)
- Gemma‑3n‑E2B: 137.6% WER (`outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json`)
- Prior Vaani (relaxed): 23.7% WER (from your earlier summary)

Which other language bench dirs should I run next to check multi‑language consistency?

If you want me to continue, pick one:
1) Run Voxtral‑Small on additional language benches you specify.
2) Try another Voxtral size/variant (if you have one in mind).
3) Explore prompt tweaks or domain‑specific fine‑tuning to chase <5% WER.


# Context from my IDE setup:

## Open tabs:
- eval.json: outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json

## My request for Codex:
Can you now check why this gemma 3n model is at such high WER rate ? take it up personally and compare the actual results the model is providing. if after everything the performance is from model's capability, then i agree we can't do much here. but if there is something wrong in our setup or prompt or whatever, we can fix it. dig deeper. Voxtral-small's performance is actually really good. can you note it down in the readme or somewhere for us to note the results. keep this in a file called results.md and explain what the experiemnt and it's results are, so we know which experiement gave best WER results. once you are done tweaking gemma model, let's note those results as well.


**Investigating gemma-3n transcription issues**

I'm exploring gemma-3n's high word error rates, suspecting a prompt mismatch or unsupported ASR mode in vLLM since gemma-3n is multimodal. Next, I'll inspect vLLM's asr.py and sample outputs to understand expected input formats and tweak prompts accordingly while documenting findings in results.md.


**Inspecting gemma output samples**

I'm examining the first few lines of gemma's results.jsonl to verify if outputs explain the high error rates, focusing on transcript content, length, and language to confirm hypotheses about mismatch or repetition.


head -n 5 outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__100__844424930753064-252-f.m4a-50ae5279bfb29b25", "base_id": "Kathbath__hindi__valid__100__844424930753064-252-f.m4a", "relpath": "hin/Kathbath__hindi__valid__100__844424930753064-252-f.m4a.wav", "uid": "50ae5279bfb29b25", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__100__844424930753064-252-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__100__844424930753064-252-f.m4a-50ae5279bfb29b25.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__100__844424930753064-252-f.m4a.wav", "duration_sec": 2.34525, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__100__844424930753064-252-f.m4a-50ae5279bfb29b25.wav", "duration_sec": 2.34525, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 2.34525, "rms_dbfs": -22.665620056300554, "peak_dbfs": -5.093265326094833, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "urd", "prob": 0.9261175394058228}, {"code": "hin", "prob": 0.07227672636508942}, {"code": "snd", "prob": 0.0007639587856829166}, {"code": "pus", "prob": 0.00010607260628603399}, {"code": "mar", "prob": 8.750744018470868e-05}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"vllm:google/gemma-3n-E2B-it": {"model": "vllm:google/gemma-3n-E2B-it", "text": "इस बकाया रकम में ब्याज की राशि भी चल।", "language": "hi", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:google/gemma-3n-E2B-it": {"language": "hi", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:google/gemma-3n-E2B-it"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:google/gemma-3n-E2B-it": "इस बकाया रकम में ब्याज की राशि भी चल"}, "rover": {"text": "इस बकाया रकम में ब्याज की राशि भी चल", "skeleton_key": "vllm:google/gemma-3n-E2B-it", "disagreement_rate": 0.0, "inputs": ["vllm:google/gemma-3n-E2B-it"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "इस बकाया रकम में ब्याज की राशि भी चल", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 2.34525, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "urd", "top1_prob": 0.9261175394058228, "margin": 0.8538408130407333, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:google/gemma-3n-E2B-it", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "vllm_asr_model": "google/gemma-3n-E2B-it", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "skip_whisper": true}, "timing": {"preprocess_sec": 0.19584312895312905, "lid_sec": 0.4332964778877795, "asr_sec": 16.279006659053266, "normalize_sec": 4.507601261138916e-05, "rover_sec": 6.587384268641472e-05, "ctc_sec": 9.82079654932022e-07, "verify_sec": 7.611420005559921e-05, "total_sec": 16.908434791024774}}
{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__101__844424931153739-164-f.m4a-2ae6a38a9e2e94ec", "base_id": "Kathbath__hindi__valid__101__844424931153739-164-f.m4a", "relpath": "hin/Kathbath__hindi__valid__101__844424931153739-164-f.m4a.wav", "uid": "2ae6a38a9e2e94ec", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__101__844424931153739-164-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__101__844424931153739-164-f.m4a-2ae6a38a9e2e94ec.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__101__844424931153739-164-f.m4a.wav", "duration_sec": 5.479938, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__101__844424931153739-164-f.m4a-2ae6a38a9e2e94ec.wav", "duration_sec": 5.479938, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 5.4799375, "rms_dbfs": -14.065149758268166, "peak_dbfs": -2.183646770994282, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "hin", "prob": 0.9907591342926025}, {"code": "pan", "prob": 0.00712191266939044}, {"code": "urd", "prob": 0.000616937642917037}, {"code": "dcc", "prob": 0.00018611203995533288}, {"code": "guj", "prob": 0.0001566595892654732}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"vllm:google/gemma-3n-E2B-it": {"model": "vllm:google/gemma-3n-E2B-it", "text": "जिला परियोजना समन्वयक सुमन कुमार मिणहास ने बैठक का संचालन किया।", "language": "hi", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:google/gemma-3n-E2B-it": {"language": "hi", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:google/gemma-3n-E2B-it"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:google/gemma-3n-E2B-it": "जिला परियोजना समन्वयक सुमन कुमार मिणहास ने बैठक का संचालन किया"}, "rover": {"text": "जिला परियोजना समन्वयक सुमन कुमार मिणहास ने बैठक का संचालन किया", "skeleton_key": "vllm:google/gemma-3n-E2B-it", "disagreement_rate": 0.0, "inputs": ["vllm:google/gemma-3n-E2B-it"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "जिला परियोजना समन्वयक सुमन कुमार मिणहास ने बैठक का संचालन किया", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 5.4799375, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "hin", "top1_prob": 0.9907591342926025, "margin": 0.9836372216232121, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:google/gemma-3n-E2B-it", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "vllm_asr_model": "google/gemma-3n-E2B-it", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "skip_whisper": true}, "timing": {"preprocess_sec": 0.21456579444929957, "lid_sec": 0.12537376722320914, "asr_sec": 0.3425252712331712, "normalize_sec": 8.640298619866371e-05, "rover_sec": 9.189406409859657e-05, "ctc_sec": 1.2922100722789764e-06, "verify_sec": 6.0004182159900665e-05, "total_sec": 0.6827881741337478}}
{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__1020__844424932917156-727-f.m4a-6ae74ae565503623", "base_id": "Kathbath__hindi__valid__1020__844424932917156-727-f.m4a", "relpath": "hin/Kathbath__hindi__valid__1020__844424932917156-727-f.m4a.wav", "uid": "6ae74ae565503623", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__1020__844424932917156-727-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__1020__844424932917156-727-f.m4a-6ae74ae565503623.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__1020__844424932917156-727-f.m4a.wav", "duration_sec": 7.662625, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__1020__844424932917156-727-f.m4a-6ae74ae565503623.wav", "duration_sec": 7.662625, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 7.662625, "rms_dbfs": -25.62665708011137, "peak_dbfs": -4.307964056793518, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "hin", "prob": 0.968579888343811}, {"code": "pan", "prob": 0.01585952751338482}, {"code": "urd", "prob": 0.011577876284718513}, {"code": "phr", "prob": 0.0007108681602403522}, {"code": "pnb", "prob": 0.0004971842863596976}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"vllm:google/gemma-3n-E2B-it": {"model": "vllm:google/gemma-3n-E2B-it", "text": "कुरान-ए-मजीद के प्रख्यात सुन्नी व्याख्याकार फक्र रे राजी अपनी किताब तफसीरे कबीर में कहते हैं, \"अहं", "language": "hi", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:google/gemma-3n-E2B-it": {"language": "hi", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:google/gemma-3n-E2B-it"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:google/gemma-3n-E2B-it": "कुरानएमजीद के प्रख्यात सुन्नी व्याख्याकार फक्र रे राजी अपनी किताब तफसीरे कबीर में कहते हैं अहं"}, "rover": {"text": "कुरानएमजीद के प्रख्यात सुन्नी व्याख्याकार फक्र रे राजी अपनी किताब तफसीरे कबीर में कहते हैं अहं", "skeleton_key": "vllm:google/gemma-3n-E2B-it", "disagreement_rate": 0.0, "inputs": ["vllm:google/gemma-3n-E2B-it"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "कुरानएमजीद के प्रख्यात सुन्नी व्याख्याकार फक्र रे राजी अपनी किताब तफसीरे कबीर में कहते हैं अहं", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 7.662625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "hin", "top1_prob": 0.968579888343811, "margin": 0.9527203608304262, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:google/gemma-3n-E2B-it", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "vllm_asr_model": "google/gemma-3n-E2B-it", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "skip_whisper": true}, "timing": {"preprocess_sec": 0.21965591935440898, "lid_sec": 0.13269539829343557, "asr_sec": 0.4707098212093115, "normalize_sec": 7.484294474124908e-05, "rover_sec": 8.869683369994164e-05, "ctc_sec": 9.420327842235565e-07, "verify_sec": 8.762581273913383e-05, "total_sec": 0.8233999079093337}}
{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__105__844424930613516-252-f.m4a-3509534021eadaf1", "base_id": "Kathbath__hindi__valid__105__844424930613516-252-f.m4a", "relpath": "hin/Kathbath__hindi__valid__105__844424930613516-252-f.m4a.wav", "uid": "3509534021eadaf1", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__105__844424930613516-252-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__105__844424930613516-252-f.m4a-3509534021eadaf1.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__105__844424930613516-252-f.m4a.wav", "duration_sec": 4.226063, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__105__844424930613516-252-f.m4a-3509534021eadaf1.wav", "duration_sec": 4.226063, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 4.2260625, "rms_dbfs": -28.209297799014543, "peak_dbfs": -4.397216919197965, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "urd", "prob": 0.9626322984695435}, {"code": "hin", "prob": 0.036564990878105164}, {"code": "pan", "prob": 0.00021295466285664588}, {"code": "snd", "prob": 0.00012353995407465845}, {"code": "pus", "prob": 8.410733425989747e-05}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"vllm:google/gemma-3n-E2B-it": {"model": "vllm:google/gemma-3n-E2B-it", "text": "पीएम ने दिग्गज खिलाड़ियों से कि कोरोना को लेकर जागरूकता फैलाने की अपील।", "language": "hi", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:google/gemma-3n-E2B-it": {"language": "hi", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:google/gemma-3n-E2B-it"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:google/gemma-3n-E2B-it": "पीएम ने दिग्गज खिलाड़ियों से कि कोरोना को लेकर जागरूकता फैलाने की अपील"}, "rover": {"text": "पीएम ने दिग्गज खिलाड़ियों से कि कोरोना को लेकर जागरूकता फैलाने की अपील", "skeleton_key": "vllm:google/gemma-3n-E2B-it", "disagreement_rate": 0.0, "inputs": ["vllm:google/gemma-3n-E2B-it"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "पीएम ने दिग्गज खिलाड़ियों से कि कोरोना को लेकर जागरूकता फैलाने की अपील", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 4.2260625, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "urd", "top1_prob": 0.9626322984695435, "margin": 0.9260673075914383, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:google/gemma-3n-E2B-it", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "vllm_asr_model": "google/gemma-3n-E2B-it", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "skip_whisper": true}, "timing": {"preprocess_sec": 0.2211446720175445, "lid_sec": 0.11973227327689528, "asr_sec": 0.3413203926756978, "normalize_sec": 8.151493966579437e-05, "rover_sec": 8.26059840619564e-05, "ctc_sec": 1.3830140233039856e-06, "verify_sec": 8.263532072305679e-05, "total_sec": 0.6825489923357964}}
{"schema_version": 1, "segment_id": "Kathbath__hindi__valid__1066__844424930913967-326-f.m4a-3a27eccf077e9e6b", "base_id": "Kathbath__hindi__valid__1066__844424930913967-326-f.m4a", "relpath": "hin/Kathbath__hindi__valid__1066__844424930913967-326-f.m4a.wav", "uid": "3a27eccf077e9e6b", "src_path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__1066__844424930913967-326-f.m4a.wav", "wav_path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__1066__844424930913967-326-f.m4a-3a27eccf077e9e6b.wav", "src_ffprobe": {"path": "data/bench/maya_hf_hin_v2/audio/hin/Kathbath__hindi__valid__1066__844424930913967-326-f.m4a.wav", "duration_sec": 3.390125, "sample_rate": 16000, "channels": 1}, "ffprobe": {"path": "outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/audio/Kathbath__hindi__valid__1066__844424930913967-326-f.m4a-3a27eccf077e9e6b.wav", "duration_sec": 3.390125, "sample_rate": 16000, "channels": 1}, "qc": {"duration_sec": 3.390125, "rms_dbfs": -23.72270893079498, "peak_dbfs": -4.134296498619131, "clipping_fraction": 0.0}, "lid": {"model": "facebook/mms-lid-256", "predictions": [{"code": "urd", "prob": 0.9611856937408447}, {"code": "hin", "prob": 0.0375131256878376}, {"code": "snd", "prob": 0.00045181543100625277}, {"code": "pan", "prob": 0.0001251194189535454}, {"code": "pus", "prob": 0.00010615516657708213}]}, "route": {"language_hint": "hin", "language_hint_source": "path"}, "asr": {"vllm:google/gemma-3n-E2B-it": {"model": "vllm:google/gemma-3n-E2B-it", "text": "इसके बाद उनकी गला रेत कर हत्या कर दी।", "language": "hi", "language_probability": null, "segments": [], "meta": {"dtype": null, "tensor_parallel_size": 1}}}, "asr_summary": {"vllm:google/gemma-3n-E2B-it": {"language": "hi", "language_probability": null, "avg_logprob_mean": null, "no_speech_prob_mean": null, "compression_ratio_mean": null}}, "asr_errors": {}, "ensemble": {"unique_models": ["vllm:google/gemma-3n-E2B-it"], "unique_backends": ["vllm"], "warnings": ["low_model_diversity"]}, "normalized": {"vllm:google/gemma-3n-E2B-it": "इसके बाद उनकी गला रेत कर हत्या कर दी"}, "rover": {"text": "इसके बाद उनकी गला रेत कर हत्या कर दी", "skeleton_key": "vllm:google/gemma-3n-E2B-it", "disagreement_rate": 0.0, "inputs": ["vllm:google/gemma-3n-E2B-it"], "filtered_out": [], "disagreements": []}, "ctc": null, "final": {"text": "इसके बाद उनकी गला रेत कर हत्या कर दी", "source": "rover"}, "verification": {"decision": "accept", "reasons": [], "scores": {"duration_sec": 3.390125, "clipping_fraction": 0.0, "rover_disagreement_rate": 0.0, "lid": {"available": true, "top1": "urd", "top1_prob": 0.9611856937408447, "margin": 0.9236725680530071, "confident": true}, "no_speech_prob_max": null, "heuristics_asr_key": "vllm:google/gemma-3n-E2B-it", "avg_logprob_mean": null, "compression_ratio_mean": null, "ctc": null}}, "config": {"trim_silence": false, "silence_threshold_db": -40.0, "min_silence_sec": 0.15, "gemini_model": null, "gemini_hard_only": true, "ctc_model": null, "mms_asr_model": null, "indicconformer_default_model": null, "vllm_asr_model": "google/gemma-3n-E2B-it", "vllm_gpu_memory_utilization": 0.9, "vllm_max_model_len": 4096, "skip_whisper": true}, "timing": {"preprocess_sec": 0.2214044346474111, "lid_sec": 0.11021916894242167, "asr_sec": 0.2434529452584684, "normalize_sec": 5.758926272392273e-05, "rover_sec": 7.910886779427528e-05, "ctc_sec": 1.2316741049289703e-06, "verify_sec": 7.707485929131508e-05, "total_sec": 0.5753815551288426}}


**Inspecting result fields and WER samples**

I'll load results.jsonl with Python to identify fields like reference and hypothesis, then examine entries with high WER to verify contents and look for text normalization or alignment issues.


python3 - <<'PY'
import json
path='outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/results.jsonl'
with open(path,'r',encoding='utf-8') as f:
    first=json.loads(next(f))
print(first.keys())
print('ref keys', [k for k in first.keys() if 'ref' in k.lower() or 'target' in k.lower()])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
dict_keys(['schema_version', 'segment_id', 'base_id', 'relpath', 'uid', 'src_path', 'wav_path', 'src_ffprobe', 'ffprobe', 'qc', 'lid', 'route', 'asr', 'asr_summary', 'asr_errors', 'ensemble', 'normalized', 'rover', 'ctc', 'final', 'verification', 'config', 'timing'])
ref keys []


**Exploring benchmark metadata**

I noted potential benchmark metadata files like `refs.jsonl` or `bench.jsonl` in the bench directory and plan to check evaluation details in outputs.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
total 120
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 14 03:05 .
drwxrwxr-x 7 ubuntu ubuntu   4096 Jan 14 03:05 ..
drwxrwxr-x 3 ubuntu ubuntu   4096 Jan 14 03:05 audio
-rw-rw-r-- 1 ubuntu ubuntu    631 Jan 14 03:05 manifest.json
-rw-rw-r-- 1 ubuntu ubuntu 106198 Jan 14 03:05 refs.jsonl


Summary: auto


**Planning WER inspection script**

I'm focusing on reading refs.jsonl to map reference texts by segment identifier or relative path, enabling a comparison of prediction and reference pairs for manual WER checks using jiwer.


head -n 3 data/bench/maya_hf_hin_v2/refs.jsonl


Output:
Exit code: 0
Wall time: 0 seconds
Output:
{"segment_id": "Kathbath__hindi__valid__2580__844424933477501-662-m.m4a-c3cd2c9d05d0764b", "text": "इसका निर्माण मुख्यतः दिल्ली और राजस्थान में हुआ है", "lang": "hin", "audio_relpath": "hin/Kathbath__hindi__valid__2580__844424933477501-662-m.m4a.wav", "duration_sec": 5.99075, "source_dataset": "ai4bharat/Kathbath", "source_config": "hindi", "source_split": "valid", "source_id": "844424933477501-662-m.m4a", "source_index": 2580}
{"segment_id": "Kathbath__hindi__valid__2273__844424933477266-934-m.m4a-f344a60c2f8b0933", "text": "आज इस पर चढ़ना मना है लेकिन इसके अंदर जा सकते हैं", "lang": "hin", "audio_relpath": "hin/Kathbath__hindi__valid__2273__844424933477266-934-m.m4a.wav", "duration_sec": 5.1548125, "source_dataset": "ai4bharat/Kathbath", "source_config": "hindi", "source_split": "valid", "source_id": "844424933477266-934-m.m4a", "source_index": 2273}
{"segment_id": "Kathbath__hindi__valid__1726__844424932912747-888-f.m4a-144700198aed0322", "text": "खपत में कमी नहीं होगी और गांवों से कारोबार में इजाफे की उम्मीद है", "lang": "hin", "audio_relpath": "hin/Kathbath__hindi__valid__1726__844424932912747-888-f.m4a.wav", "duration_sec": 3.761625, "source_dataset": "ai4bharat/Kathbath", "source_config": "hindi", "source_split": "valid", "source_id": "844424932912747-888-f.m4a", "source_index": 1726}


**Assessing transcription output quality**

I'm verifying that output IDs align with references and planning to compute word error rates (WER) for sampled segments; preliminary high WER suggests outputs may be hallucinated rather than true transcriptions, so I'll check whether the model and prompt are appropriate for audio transcription.


**Inspecting ASR prompt configuration**

I'm checking the ASR prompt setup in asr.py to confirm if the transcription prompt properly guides the model for audio input, as current prompts may not support audio with vLLM in gemma 3n.


rg -n "VllmTranscriptionASR|transcribe" -S maya_transcribe/asr.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
9:from maya_transcribe.language import ISO3_TO_WHISPER_LANG, to_whisper_lang
69:    def transcribe(
94:        segments_iter, info = self._model.transcribe(
202:    def transcribe(
209:        from maya_transcribe.audio import decode_pcm16
237:                    forced_decoder_ids = get_ids(language=whisper_lang, task="transcribe")
319:    def transcribe(
346:        return backend.transcribe(
354:class VllmTranscriptionASR:
422:        self._stt_config = self._model_cls.get_speech_to_text_config(self._model_config, "transcribe")
442:    def transcribe(
447:        from maya_transcribe.audio import decode_pcm16
473:            task_type="transcribe",
591:    def transcribe(
596:        from maya_transcribe.audio import decode_pcm16
653:    def transcribe(
661:        return self._backend.transcribe(audio_path=audio_path, language=language)
719:    def transcribe(
724:        # NeMo models transcribe directly from file path.
726:            texts = self._model.transcribe([str(audio_path)])
794:    def transcribe(
818:        return backend.transcribe(audio_path=audio_path, language=language)


sed -n '340,520p' maya_transcribe/asr.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
            _, evicted = self._cache.popitem(last=False)
            try:
                evicted.close()
            except Exception:
                pass

        return backend.transcribe(
            audio_path=audio_path,
            language=language,
            max_new_tokens=max_new_tokens,
            num_beams=num_beams,
        )

class VllmTranscriptionASR:
    """
    vLLM-powered speech-to-text backend for multimodal audio models (e.g., Voxtral, Gemma3n).
    """

    def __init__(
        self,
        model_id: str,
        *,
        dtype: str | None = None,
        tensor_parallel_size: int = 1,
        trust_remote_code: bool = True,
        max_tokens: int | None = None,
        gpu_memory_utilization: float | None = None,
        max_model_len: int | None = None,
        limit_mm_per_prompt: dict[str, int | dict[str, int]] | None = None,
    ) -> None:
        try:
            import torch
            from vllm import LLM, SamplingParams
            from vllm.model_executor.model_loader import get_model_cls
        except Exception as e:  # pragma: no cover
            raise RuntimeError(
                "vLLM transcription requires vllm[audio]. Install with: uv pip install -U \"vllm[audio]\" --system"
            ) from e

        self.model_id = model_id
        self.dtype = dtype
        self.tensor_parallel_size = max(1, int(tensor_parallel_size))
        self.trust_remote_code = bool(trust_remote_code)

        try:
            from mistral_common.tokens.tokenizers.audio import AudioEncoder  # type: ignore
            import inspect

            sig = inspect.signature(AudioEncoder.pad)
            needs_default = "is_online_streaming" in sig.parameters and sig.parameters["is_online_streaming"].default is inspect._empty
            if needs_default:
                orig_pad = AudioEncoder.pad

                def _pad(self, audio_array, sampling_rate, is_online_streaming=False):
                    return orig_pad(self, audio_array, sampling_rate, is_online_streaming)

                AudioEncoder.pad = _pad  # type: ignore[assignment]
        except Exception:
            pass

        if limit_mm_per_prompt is None and "gemma-3n" in model_id.lower():
            limit_mm_per_prompt = {"image": 0, "audio": 1}

        llm_kwargs = {
            "model": model_id,
            "tensor_parallel_size": self.tensor_parallel_size,
            "trust_remote_code": self.trust_remote_code,
        }
        llm_kwargs["dtype"] = dtype if dtype is not None else "auto"
        if gpu_memory_utilization is not None:
            llm_kwargs["gpu_memory_utilization"] = float(gpu_memory_utilization)
        if max_model_len is not None:
            llm_kwargs["max_model_len"] = int(max_model_len)
        if limit_mm_per_prompt is not None:
            llm_kwargs["limit_mm_per_prompt"] = limit_mm_per_prompt
        self._llm = LLM(**llm_kwargs)
        self._model_config = self._llm.model_config
        self._model_cls = get_model_cls(self._model_config)
        if not bool(getattr(self._model_cls, "supports_transcription", False)):
            raise RuntimeError(f"Model does not support transcription: {model_id}")

        self._stt_config = self._model_cls.get_speech_to_text_config(self._model_config, "transcribe")

        max_model_len = getattr(self._model_config, "max_model_len", None)
        if max_model_len is None:
            max_model_len = 1024
        if max_tokens is None:
            max_tokens = min(1024, int(max_model_len))
        else:
            max_tokens = min(int(max_tokens), int(max_model_len))
        self._sampling_params = SamplingParams(temperature=0.0, top_p=1.0, max_tokens=max_tokens)

        try:
            self._torch = torch
        except Exception:
            self._torch = None

    @property
    def name(self) -> str:
        return f"vllm:{self.model_id}"

    def transcribe(
        self,
        audio_path: Path,
        language: str | None = None,
    ) -> ASRResult:
        from maya_transcribe.audio import decode_pcm16

        sr = int(getattr(self._stt_config, "sample_rate", 16000))
        wav, _ = decode_pcm16(audio_path, sample_rate=sr, mono=True)
        if wav.size == 0:
            return ASRResult(
                model=self.name,
                text="",
                language=language,
                language_probability=None,
                segments=[],
                meta={"dtype": self.dtype, "tensor_parallel_size": self.tensor_parallel_size},
            )

        lang = to_whisper_lang(language)
        if lang:
            try:
                lang = self._model_cls.validate_language(lang)
            except Exception:
                lang = None

        prompt = self._model_cls.get_generation_prompt(
            audio=wav,
            stt_config=self._stt_config,
            model_config=self._model_config,
            language=lang,
            task_type="transcribe",
            request_prompt="",
            to_language=None,
        )

        outputs = self._llm.generate([prompt], self._sampling_params)
        text = ""
        if outputs and outputs[0].outputs:
            text = outputs[0].outputs[0].text

        return ASRResult(
            model=self.name,
            text=str(text).strip(),
            language=lang,
            language_probability=None,
            segments=[],
            meta={"dtype": self.dtype, "tensor_parallel_size": self.tensor_parallel_size},
        )

def _to_iso3(lang: str | None) -> str | None:
    """Convert language code to ISO-639-3 format."""
    if not lang:
        return None
    code = lang.strip().lower()
    if not code:
        return None
    code = code.replace("-", "_")
    if "_" in code:
        code = code.split("_", 1)[0]
    if len(code) == 3:
        return code
    if len(code) == 2:
        return _WHISPER_TO_ISO3.get(code, None)
    return code

class MMSASR:
    """
    MMS ASR backend using CTC decoding (facebook/mms-1b-all).
    Provides architectural diversity from Whisper's encoder-decoder approach.
    """

    def __init__(
        self,
        model_id: str = "facebook/mms-1b-all",
        device: str | None = None,
    ) -> None:


**Investigating prompt generation for transcription**

I'm considering that gemma-3n may claim transcription support but actually produce generic or hallucinated outputs due to prompt issues; next, I'll inspect the prompt returned by get_generation_prompt for gemma to verify if it properly instructs transcription, possibly by running a light script or introspecting the prompt structure without fully loading the model.


**Checking vLLM gemma3n model code**

I'll search the vLLM site-packages for the gemma3n model implementation to examine its get_generation_prompt method directly for potential issues.


rg -n "Gemma.*3n|gemma-3n|Gemma3" $(python3 - <<'PY'
import site
print(' '.join(site.getsitepackages()))
PY
)


Output:
Exit code: 2
Wall time: 0.2 seconds
Output:
rg: /usr/lib/python3.10/dist-packages: No such file or directory (os error 2)


**Checking actual site-packages paths**

I'll run Python to print its site-packages locations to confirm the correct directories instead of guessing paths like /home/ubuntu/.local/lib/python3.10/site-packages or /usr/local/lib.


python3 - <<'PY'
import site, sys
print('\n'.join(site.getsitepackages()))
print('user:', site.getusersitepackages())
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
/usr/local/lib/python3.10/dist-packages
/usr/lib/python3/dist-packages
/usr/lib/python3.10/dist-packages
user: /home/ubuntu/.local/lib/python3.10/site-packages


**Locating vllm installation path**

Determined that vllm is installed in the user site directory at /home/ubuntu/.local/lib/python3.10/site-packages/vllm, so the search will target that location.


rg -n "Gemma" /home/ubuntu/.local/lib/python3.10/site-packages/vllm/model_executor/models