Output:
Exit code: 0
Wall time: 507.1 seconds
Total output lines: 1699
Output:
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-15 05:03:15.911023: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
INFO 01-15 05:03:22 [utils.py:253] non-default args: {'trust_remote_code': True, 'max_model_len': 4096, 'disable_log_stats': True, 'model': 'mistralai/Voxtral-Small-24B-2507'}
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
INFO 01-15 05:03:23 [model.py:514] Resolved architecture: VoxtralForConditionalGeneration
INFO 01-15 05:03:23 [model.py:1661] Using max model len 4096
INFO 01-15 05:03:23 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=8192.
[2026-01-15 05:03:24] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
[2026-01-15 05:03:24] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
WARNING 01-15 05:03:25 [system_utils.py:136] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
/home/ubuntu/.local/lib/python3.10/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
2026-01-15 05:03:30.566827: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/google/api_core/_python_version_support.py:266: FutureWarning: You are using a Python version (3.10.12) which Google will stop supporting in new releases of google.api_core once it reaches its end of life (2026-10-04). Please upgrade to the latest Python version, or at least Python 3.11, to continue receiving updates for google.api_core past that date.
  warnings.warn(message, FutureWarning)
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:34 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='mistralai/Voxtral-Small-24B-2507', speculative_config=None, tokenizer='mistralai/Voxtral-Small-24B-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=mistralai/Voxtral-Small-24B-2507, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None}
(EngineCore_DP0 pid=2071299) [2026-01-15 05:03:35] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(EngineCore_DP0 pid=2071299) [2026-01-15 05:03:35] INFO tekken.py:567: Cutting non special vocabulary to first 130072 tokens.
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:36 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://216.81.248.184:54819 backend=nccl
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:36 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:37 [gpu_model_runner.py:3562] Starting to load model mistralai/Voxtral-Small-24B-2507...
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:38 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:38 [layer.py:477] Using AttentionBackendEnum.FLASH_ATTN for MultiHeadAttention in multimodal encoder.
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:38 [weight_utils.py:527] No consolidated.safetensors.index.json found in remote.
(EngineCore_DP0 pid=2071299)
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
(EngineCore_DP0 pid=2071299)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:06<00:00,  6.98s/it]
(EngineCore_DP0 pid=2071299)
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:06<00:00,  6.98s/it]
(EngineCore_DP0 pid=2071299)
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:47 [default_loader.py:308] Loading weights took 8.54 seconds
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:48 [gpu_model_runner.py:3659] Model loading took 45.2032 GiB memory and 9.240441 seconds
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:48 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 2 audio items of the maximum feature size.
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:55 [backends.py:643] Using cache directory: /home/ubuntu/.cache/vllm/torch_compile_cache/936cd8900f/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=2071299) INFO 01-15 05:03:55 [backends.py:703] Dynamo bytecode transform time: 6.08 s
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:08 [backends.py:226] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 8.717 s
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:08 [monitor.py:34] torch.compile takes 14.80 s in total
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:09 [gpu_worker.py:375] Available KV cache memory: 24.25 GiB
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:09 [kv_cache_utils.py:1291] GPU KV cache size: 158,912 tokens
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:09 [kv_cache_utils.py:1296] Maximum concurrency for 4,096 tokens per request: 38.80x
(EngineCore_DP0 pid=2071299)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   0%|          | 0/51 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   2%|▏         | 1/51 [00:00<00:05,  8.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   4%|▍         | 2/51 [00:00<00:05,  8.64it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   6%|▌         | 3/51 [00:00<00:05,  8.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):   8%|▊         | 4/51 [00:00<00:05,  8.74it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  10%|▉         | 5/51 [00:00<00:05,  8.78it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  12%|█▏        | 6/51 [00:00<00:05,  8.81it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  14%|█▎        | 7/51 [00:00<00:04,  8.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  16%|█▌        | 8/51 [00:00<00:04,  8.86it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  20%|█▉        | 10/51 [00:01<00:04,  9.42it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  24%|██▎       | 12/51 [00:01<00:04,  9.70it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  27%|██▋       | 14/51 [00:01<00:03,  9.91it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  31%|███▏      | 16/51 [00:01<00:03, 10.05it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  35%|███▌      | 18/51 [00:01<00:03, 10.99it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  39%|███▉      | 20/51 [00:01<00:02, 11.79it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  43%|████▎     | 22/51 [00:02<00:02, 12.39it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  47%|████▋     | 24/51 [00:02<00:02, 12.85it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  51%|█████     | 26/51 [00:02<00:01, 13.20it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  55%|█████▍    | 28/51 [00:02<00:01, 13.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  59%|█████▉    | 30/51 [00:02<00:01, 13.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  63%|██████▎   | 32/51 [00:02<00:01, 13.72it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  67%|██████▋   | 34/51 [00:02<00:01, 14.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  71%|███████   | 36/51 [00:03<00:01, 14.94it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  75%|███████▍  | 38/51 [00:03<00:00, 15.13it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  78%|███████▊  | 40/51 [00:03<00:00, 15.58it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  82%|████████▏ | 42/51 [00:03<00:00, 16.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  86%|████████▋ | 44/51 [00:03<00:00, 16.65it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  90%|█████████ | 46/51 [00:03<00:00, 17.01it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  94%|█████████▍| 48/51 [00:03<00:00, 17.26it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  98%|█████████▊| 50/51 [00:03<00:00, 17.55it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 51/51 [00:03<00:00, 12.91it/s]
(EngineCore_DP0 pid=2071299)
Capturing CUDA graphs (decode, FULL):   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL):   6%|▌         | 2/35 [00:00<00:02, 12.43it/s]
Capturing CUDA graphs (decode, FULL):  11%|█▏        | 4/35 [00:00<00:02, 13.27it/s]
Capturing CUDA graphs (decode, FULL):  17%|█▋        | 6/35 [00:00<00:02, 13.59it/s]
Capturing CUDA graphs (decode, FULL):  23%|██▎       | 8/35 [00:00<00:01, 13.80it/s]
Capturing CUDA graphs (decode, FULL):  29%|██▊       | 10/35 [00:00<00:01, 13.93it/s]
Capturing CUDA graphs (decode, FULL):  34%|███▍      | 12/35 [00:00<00:01, 14.02it/s]
Capturing CUDA graphs (decode, FULL):  40%|████      | 14/35 [00:01<00:01, 14.10it/s]
Capturing CUDA graphs (decode, FULL):  46%|████▌     | 16/35 [00:01<00:01, 14.17it/s]
Capturing CUDA graphs (decode, FULL):  51%|█████▏    | 18/35 [00:01<00:01, 14.92it/s]
Capturing CUDA graphs (decode, FULL):  57%|█████▋    | 20/35 [00:01<00:00, 15.48it/s]
Capturing CUDA graphs (decode, FULL):  63%|██████▎   | 22/35 [00:01<00:00, 15.86it/s]
Capturing CUDA graphs (decode, FULL):  69%|██████▊   | 24/35 [00:01<00:00, 16.28it/s]
Capturing CUDA graphs (decode, FULL):  74%|███████▍  | 26/35 [00:01<00:00, 16.95it/s]
Capturing CUDA graphs (decode, FULL):  80%|████████  | 28/35 [00:01<00:00, 17.51it/s]
Capturing CUDA graphs (decode, FULL):  86%|████████▌ | 30/35 [00:01<00:00, 17.99it/s]
Capturing CUDA graphs (decode, FULL):  91%|█████████▏| 32/35 [00:02<00:00, 18.34it/s]
Capturing CUDA graphs (decode, FULL):  97%|█████████▋| 34/35 [00:02<00:00, 18.67it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 35/35 [00:02<00:00, 15.86it/s]
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:17 [gpu_model_runner.py:4587] Graph capturing finished in 7 secs, took 0.71 GiB
(EngineCore_DP0 pid=2071299) INFO 01-15 05:04:17 [core.py:259] init engine (profile, create kv cache, warmup model) took 28.74 seconds
INFO 01-15 05:04:18 [llm.py:360] Supported tasks: ['generate', 'transcription']

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:01<00:00,  1.17s/it]
Adding requests: 100%|██████████| 1/1 [00:01<00:00,  1.17s/it]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.24 toks/s, output: 27.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.24 toks/s, output: 27.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.24 toks/s, output: 27.42 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]WARNING 01-15 05:04:20 [processing.py:1153] VoxtralProcessorAdapter did not return `BatchFeature`. Make sure to match the behaviour of `ProcessorMixin` when implementing custom processors.
/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 419.47it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.51it/s, est. speed input: 642.71 toks/s, output: 30.17 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.51it/s, est. speed input: 642.71 toks/s, output: 30.17 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.51it/s, est. speed input: 642.71 toks/s, output: 30.17 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 288.13it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.11s/it, est. speed input: 562.88 toks/s, output: 30.52 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.11s/it, est. speed input: 562.88 toks/s, output: 30.52 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.11s/it, est. speed input: 562.88 toks/s, output: 30.52 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 504.37it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.04it/s, est. speed input: 451.46 toks/s, output: 31.06 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.04it/s, est. speed input: 451.46 toks/s, output: 31.06 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.04it/s, est. speed input: 451.46 toks/s, output: 31.06 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 280.63it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:02<00:00,  2.82s/it, est. speed input: 222.11 toks/s, output: 32.24 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:02<00:00,  2.82s/it, est. speed input: 222.11 toks/s, output: 32.24 toks/s]
Processed prompts: 100%|████████…89503 tokens truncated…speed input: 771.71 toks/s, output: 29.54 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 484.61it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.62 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.62 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.62 toks/s, output: 30.25 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 301.79it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 799.81 toks/s, output: 29.34 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 799.81 toks/s, output: 29.34 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 799.81 toks/s, output: 29.34 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 500.93it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.71 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.71 toks/s, output: 30.25 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 613.71 toks/s, output: 30.25 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 329.35it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 904.87 toks/s, output: 28.86 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 904.87 toks/s, output: 28.86 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.44it/s, est. speed input: 904.87 toks/s, output: 28.86 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 484.95it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.65it/s, est. speed input: 697.55 toks/s, output: 29.75 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.65it/s, est. speed input: 697.55 toks/s, output: 29.75 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.65it/s, est. speed input: 697.55 toks/s, output: 29.75 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 336.54it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 800.96 toks/s, output: 29.38 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 800.96 toks/s, output: 29.38 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.28it/s, est. speed input: 800.96 toks/s, output: 29.38 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 515.84it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.58it/s, est. speed input: 670.14 toks/s, output: 29.96 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.58it/s, est. speed input: 670.14 toks/s, output: 29.96 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.58it/s, est. speed input: 670.14 toks/s, output: 29.96 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 334.50it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.00it/s, est. speed input: 629.45 toks/s, output: 30.12 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.00it/s, est. speed input: 629.45 toks/s, output: 30.12 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.00it/s, est. speed input: 629.45 toks/s, output: 30.12 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 508.28it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 512.19 toks/s, output: 30.83 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 512.19 toks/s, output: 30.83 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 512.19 toks/s, output: 30.83 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 73.30it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.23 toks/s, output: 29.70 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.23 toks/s, output: 29.70 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 716.23 toks/s, output: 29.70 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 504.18it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.37it/s, est. speed input: 587.89 toks/s, output: 30.22 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.37it/s, est. speed input: 587.89 toks/s, output: 30.22 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.37it/s, est. speed input: 587.89 toks/s, output: 30.22 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 64.42it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.18it/s, est. speed input: 737.77 toks/s, output: 29.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.18it/s, est. speed input: 737.77 toks/s, output: 29.42 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.18it/s, est. speed input: 737.77 toks/s, output: 29.42 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 474.47it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.75it/s, est. speed input: 1166.15 toks/s, output: 27.50 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.75it/s, est. speed input: 1166.15 toks/s, output: 27.50 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.75it/s, est. speed input: 1166.15 toks/s, output: 27.50 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 63.96it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.83s/it, est. speed input: 342.78 toks/s, output: 31.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.83s/it, est. speed input: 342.78 toks/s, output: 31.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.83s/it, est. speed input: 342.78 toks/s, output: 31.71 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 429.04it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.65s/it, est. speed input: 269.75 toks/s, output: 32.13 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.65s/it, est. speed input: 269.75 toks/s, output: 32.13 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.65s/it, est. speed input: 269.75 toks/s, output: 32.13 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 308.20it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 744.95 toks/s, output: 29.70 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 744.95 toks/s, output: 29.70 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.19it/s, est. speed input: 744.95 toks/s, output: 29.70 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 463.51it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.06it/s, est. speed input: 874.77 toks/s, output: 28.88 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.06it/s, est. speed input: 874.77 toks/s, output: 28.88 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.06it/s, est. speed input: 874.77 toks/s, output: 28.88 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 267.58it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 715.95 toks/s, output: 29.69 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 715.95 toks/s, output: 29.69 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.14it/s, est. speed input: 715.95 toks/s, output: 29.69 toks/s]

Adding requests:   0%|          | 0/1 [00:00<?, ?it/s]/home/ubuntu/.local/lib/python3.10/site-packages/mistral_common/tokens/tokenizers/tekken.py:462: FutureWarning: `get_control_token` is deprecated. Use `get_special_token` instead.
  warnings.warn("`get_control_token` is deprecated. Use `get_special_token` instead.", FutureWarning)

Adding requests: 100%|██████████| 1/1 [00:00<00:00, 443.37it/s]

Processed prompts:   0%|          | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.23it/s, est. speed input: 528.21 toks/s, output: 30.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.23it/s, est. speed input: 528.21 toks/s, output: 30.71 toks/s]
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.23it/s, est. speed input: 528.21 toks/s, output: 30.71 toks/s]
Transcribing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:07:15
[rank0]:[W115 05:11:34.639781404 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/results.jsonl
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/summary.json
Wrote: outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 0.2849680458430461,
  "cer_mean": 0.2195450530720729,
  "wer_p50": 0.15384615384615385,
  "wer_p90": 0.5555555555555556,
  "wer_p95": 1.1428571428571428,
  "cer_p50": 0.05660377358490566,
  "cer_p90": 0.3442622950819672,
  "cer_p95": 1.2702702702702702,
  "segment_error_rate": 0.79,
  "pct_wer_gt_5pct": 0.79,
  "pct_wer_gt_10pct": 0.605,
  "wer_by_language_mean": {
    "hin": 0.2849680458430461
  },
  "wer_by_decision_mean": {
    "accept": 0.24114180323857753,
    "human": 0.32306001371421933
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.15726010101010102,
    "10-30s": 0.29143772893772896,
    "3-10s": 0.28740993049503705
  },
  "cer_by_language_mean": {
    "hin": 0.2195450530720729
  },
  "cer_by_decision_mean": {
    "accept": 0.2064104556366455,
    "human": 0.2309611050486592
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.062306501547987614,
    "10-30s": 0.11758138114057698,
    "3-10s": 0.22722943382498934
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.21911767399267412,
    "selection_gap_wer_mean": 0.06585037185037182,
    "selection_gap_wer_p90": 0.125
  }
}


Summary: auto

## Skills
A skill is a set of local instructions to follow that is stored in a `SKILL.md` file. Below is the list of skills that can be used. Each entry includes a name, description, and file path so you can open the source for full instructions when using a specific skill.
### Available skills
- skill-creator: Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations. (file: /home/ubuntu/.codex/skills/.system/skill-creator/SKILL.md)
- skill-installer: Install Codex skills into $CODEX_HOME/skills from a curated list or a GitHub repo path. Use when a user asks to list installable skills, install a curated skill, or install a skill from another repo (including private repos). (file: /home/ubuntu/.codex/skills/.system/skill-installer/SKILL.md)
### How to use skills
- Discovery: The list above is the skills available in this session (name + description + file path). Skill bodies live on disk at the listed paths.
- Trigger rules: If the user names a skill (with `$SkillName` or plain text) OR the task clearly matches a skill's description shown above, you must use that skill for that turn. Multiple mentions mean use them all. Do not carry skills across turns unless re-mentioned.
- Missing/blocked: If a named skill isn't in the list or the path can't be read, say so briefly and continue with the best fallback.
- How to use a skill (progressive disclosure):
  1) After deciding to use a skill, open its `SKILL.md`. Read only enough to follow the workflow.
  2) If `SKILL.md` points to extra folders such as `references/`, load only the specific files needed for the request; don't bulk-load everything.
  3) If `scripts/` exist, prefer running or patching them instead of retyping large code blocks.
  4) If `assets/` or templates exist, reuse them instead of recreating from scratch.
- Coordination and sequencing:
  - If multiple skills apply, choose the minimal set that covers the request and state the order you'll use them.
  - Announce which skill(s) you're using and why (one short line). If you skip an obvious skill, say why.
- Context hygiene:
  - Keep context small: summarize long sections instead of pasting them; only load extra files when needed.
  - Avoid deep reference-chasing: prefer opening only files directly linked from `SKILL.md` unless you're blocked.
  - When variants exist (frameworks, providers, domains), pick only the relevant reference file(s) and note that choice.
- Safety and fallback: If a skill can't be applied cleanly (missing files, unclear instructions), state the issue, pick the next-best approach, and continue.


**Checking proofread evaluation outputs**

Noted the proofread prompt has a high WER (28.5%) and need to verify if raw and proofread evaluations are saved separately; I'll inspect the output directory to confirm what files are produced.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
audio
eval.json
results.jsonl
summary.json


Summary: auto


**Investigating missing proofread results**

Noticed that proofread outputs may not be stored in the current pipeline, possibly due to handling outputs as lists or code changes; next I'll verify if proofread results are saved in the pipeline or benchmark run by inspecting results.jsonl for relevant metadata.


python3 - <<'PY'
import json
from pathlib import Path
path = Path('outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/results.jsonl')
print('lines', sum(1 for _ in path.open()))
with path.open() as f:
    first = json.loads(next(f))
print(first.keys())
print(first.get('asr'))
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
lines 200
dict_keys(['schema_version', 'segment_id', 'base_id', 'relpath', 'uid', 'src_path', 'wav_path', 'src_ffprobe', 'ffprobe', 'qc', 'lid', 'route', 'asr', 'asr_summary', 'asr_errors', 'ensemble', 'normalized', 'rover', 'ctc', 'final', 'verification', 'config', 'timing'])
{'vllm:mistralai/Voxtral-Small-24B-2507|raw': {'model': 'vllm:mistralai/Voxtral-Small-24B-2507', 'text': 'इस बकाया रकम में ब्याज की राशि भी शामिल है।', 'language': 'hi', 'language_probability': None, 'segments': [], 'meta': {'dtype': None, 'tensor_parallel_size': 1}}, 'vllm:mistralai/Voxtral-Small-24B-2507|proofread': {'model': 'vllm:mistralai/Voxtral-Small-24B-2507|proofread', 'text': 'इस बकाया रकम में ब्याज की राशि भी शामिल है।', 'language': 'hi', 'language_probability': None, 'segments': [], 'meta': {'dtype': None, 'tensor_parallel_size': 1}}}


**Planning separate WER evaluation for raw and proofread**

The results.jsonl mixes raw and proofread outputs, so I'll search for an existing evaluation CLI or script to compute WER separately for each and compare them before focusing on the final WER summary.


rg -n "bench eval|eval.json|eval" maya_transcribe/cli.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
331:def eval(
335:    hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
343:    from maya_transcribe.eval import evaluate
345:    summary = evaluate(
433:    hypothesis: str = typer.Option("final", help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
436:    Run the full pipeline on a local benchmark corpus and write eval.json.
440:    from maya_transcribe.eval import evaluate
513:    eval_path = out / "eval.json"
514:    summary = evaluate(
517:        out_path=eval_path,
524:    typer.echo(f"Wrote: {eval_path}")


**Inspecting eval command usage**

I'm checking the eval function in the CLI to understand how to run evaluation commands with benchmarks, results, and hypotheses for specific ASR models.


sed -n '300,380p' maya_transcribe/cli.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        lid_model_id=lid_model_id,
        require_lid_backend=require_lid_backend,
        whisper_model=whisper_model,
        skip_whisper=skip_whisper,
        vllm_asr_model=vllm_asr_model,
        vllm_gpu_memory_utilization=vllm_gpu_memory_utilization,
        vllm_max_model_len=vllm_max_model_len,
        vllm_transcribe_prompt=vllm_transcribe_prompt_text,
        vllm_proofread=vllm_proofread,
        vllm_proofread_prompt=vllm_proofread_prompt,
        extra_transformers_whisper_models=transformers_whisper_model,
        transformers_whisper_model_map=tf_map,
        mms_asr_model=mms_asr_model,
        indicconformer_model_map=ic_map,
        indicconformer_default_model=indicconformer_default_model,
        gemini_model=gemini_model,
        gemini_prompt_path=gemini_prompt_file,
        gemini_hard_only=gemini_hard_only,
        ctc_model=ctc_model,
        ctc_model_map=ctc_map,
        trim_silence=trim_silence,
        silence_threshold_db=silence_threshold_db,
        min_silence_sec=min_silence_sec,
        limit=limit,
        thresholds=thresholds,
    )
    typer.echo(f"Wrote: {results_path}")
    typer.echo(f"Wrote: {out / 'summary.json'}")

@app.command()
def eval(
    results: Path = typer.Option(..., exists=True, dir_okay=False, help="Pipeline results JSONL (outputs/run/results.jsonl)."),
    refs: Path = typer.Option(..., exists=True, dir_okay=False, help="Reference transcripts JSONL."),
    out: Path | None = typer.Option(None, help="Write summary JSON to this path."),
    hypothesis: str = typer.Option('final', help='Hypothesis to evaluate: "final", "rover", or "asr:<key>".'),
    oracle: bool = typer.Option(True, help="Compute oracle-best and selection-gap stats across all candidates."),
    language_source: str = typer.Option("route", help="Use language from: 'route' (LID) or 'refs' (refs.jsonl)."),
    decision: list[str] = typer.Option([], help="Filter to verification decision(s): accept/human/reject (repeatable)."),
) -> None:
    """
    Compute WER/CER for a hypothesis against a reference JSONL keyed by `segment_id`.
    """
    from maya_transcribe.eval import evaluate

    summary = evaluate(
        results_path=results,
        refs_path=refs,
        out_path=out,
        hypothesis=hypothesis,
        oracle=oracle,
        language_source=language_source,
        decision_filter=decision or None,
    )
    typer.echo(json.dumps(summary, ensure_ascii=False, indent=2))

@bench_app.command("build")
def bench_build(
    out: Path = typer.Option(Path("data/bench/maya_hf_v0"), help="Output benchmark corpus directory."),
    samples_per_lang: int = typer.Option(200, help="Target samples per language."),
    seed: int = typer.Option(0, help="Shuffle seed."),
    shuffle: bool = typer.Option(True, help="Shuffle within each source before selecting."),
    lang: list[str] = typer.Option([], help="Limit to ISO-639-3 language codes (repeatable)."),
    min_duration_sec: float = typer.Option(1.0, help="Min audio duration."),
    max_duration_sec: float = typer.Option(30.0, help="Max audio duration."),
    force: bool = typer.Option(False, help="Delete and rebuild output directory if it exists."),
) -> None:
    """
    Build a local benchmark corpus from Hugging Face datasets.
    """
    from maya_transcribe.bench import build_benchmark_corpus

    manifest = build_benchmark_corpus(
        out_dir=out,
        samples_per_lang=samples_per_lang,
        seed=seed,
        shuffle=shuffle,
        langs=lang or None,
        min_duration_sec=min_duration_sec,
        max_duration_sec=max_duration_sec,


**Checking bench directory for references path**

I'm verifying if the references file exists at `data/bench/maya_hf_hin_v2/refs.jsonl` to decide on using eval for raw and proofread data for more comprehensive results.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
audio
manifest.json
refs.jsonl


**Running raw evaluation sequentially**

I'm choosing to run the raw evaluation command sequentially to avoid concurrency issues since both evaluations are CPU-bound; the final proofread evaluation will run separately afterward for comparison.


python3 -m maya_transcribe.cli eval --results outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/results.jsonl --refs data/bench/maya_hf_hin_v2/refs.jsonl --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_raw.json --hypothesis "asr:vllm:mistralai/Voxtral-Small-24B-2507|raw"


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "hypothesis": "asr:vllm:mistralai/Voxtral-Small-24B-2507|raw",
  "pairs_evaluated": 200,
  "wer_mean": 0.2565013736263737,
  "cer_mean": 0.181590870576232,
  "wer_p50": 0.14285714285714285,
  "wer_p90": 0.5,
  "wer_p95": 1.0,
  "cer_p50": 0.05128205128205128,
  "cer_p90": 0.2903225806451613,
  "cer_p95": 1.0,
  "segment_error_rate": 0.78,
  "pct_wer_gt_5pct": 0.78,
  "pct_wer_gt_10pct": 0.595,
  "wer_by_language_mean": {
    "hin": 0.2565013736263737
  },
  "wer_by_decision_mean": {
    "accept": 0.2107121970025196,
    "human": 0.2962994430284149
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.15726010101010102,
    "10-30s": 0.29143772893772896,
    "3-10s": 0.25712623664751333
  },
  "cer_by_language_mean": {
    "hin": 0.181590870576232
  },
  "cer_by_decision_mean": {
    "accept": 0.16786188017450598,
    "human": 0.1935235444767976
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.062306501547987614,
    "10-30s": 0.11758138114057698,
    "3-10s": 0.18685264393579695
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.21911767399267412,
    "selection_gap_wer_mean": 0.03738369963369963,
    "selection_gap_wer_p90": 0.07692307692307693
  }
}


**Evaluating proofreading hypothesis**


python3 -m maya_transcribe.cli eval --results outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/results.jsonl --refs data/bench/maya_hf_hin_v2/refs.jsonl --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_proofread.json --hypothesis "asr:vllm:mistralai/Voxtral-Small-24B-2507|proofread"


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "hypothesis": "asr:vllm:mistralai/Voxtral-Small-24B-2507|proofread",
  "pairs_evaluated": 200,
  "wer_mean": 0.3184132534132537,
  "cer_mean": 0.27358345387323096,
  "wer_p50": 0.15384615384615385,
  "wer_p90": 1.0,
  "wer_p95": 1.1111111111111112,
  "cer_p50": 0.06451612903225806,
  "cer_p90": 1.0,
  "cer_p95": 1.3181818181818181,
  "segment_error_rate": 0.775,
  "pct_wer_gt_5pct": 0.775,
  "pct_wer_gt_10pct": 0.605,
  "wer_by_language_mean": {
    "hin": 0.3184132534132537
  },
  "wer_by_decision_mean": {
    "accept": 0.2419689330979654,
    "human": 0.3848555131265413
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.09823232323232324,
    "10-30s": 0.4313186813186813,
    "3-10s": 0.3182934677615531
  },
  "cer_by_language_mean": {
    "hin": 0.27358345387323096
  },
  "cer_by_decision_mean": {
    "accept": 0.2067812379874056,
    "human": 0.331645192914182
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.04760061919504644,
    "10-30s": 0.29591471447391027,
    "3-10s": 0.27744133288337625
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.21911767399267412,
    "selection_gap_wer_mean": 0.09929557942057946,
    "selection_gap_wer_p90": 0.5
  }
}


**Searching for existing ensemble script**

Checking for a script to merge Voxtral and Gemma prompts for offline ensemble, likely in tools or maya_transcribe/rover folders, to reuse the previous approach that reads both results.jsonl files and combines them.


rg -n "ensemble" -S tools maya_transcribe outputs | head


Output:
Exit code: 0
Wall time: 0 seconds
Output:
rg: tools: No such file or directory (os error 2)
maya_transcribe/pipeline.py:102:    End-to-end run: preprocess -> LID -> ASR ensemble -> normalize -> ROVER -> verify.
maya_transcribe/pipeline.py:260:                # 3) ASR ensemble
maya_transcribe/pipeline.py:387:                ensemble_models = sorted({v.model for v in asr_results.values()})
maya_transcribe/pipeline.py:388:                ensemble_backends = sorted({v.model.split(":", 1)[0] for v in asr_results.values()})
maya_transcribe/pipeline.py:389:                ensemble_warnings: list[str] = []
maya_transcribe/pipeline.py:390:                if len(ensemble_models) < 2:
maya_transcribe/pipeline.py:391:                    ensemble_warnings.append("low_model_diversity")
maya_transcribe/pipeline.py:426:                    include_sampling = len(ensemble_backends) < 2
maya_transcribe/pipeline.py:525:                    "ensemble": {
maya_transcribe/pipeline.py:526:                        "unique_models": ensemble_models,


**Planning custom script to merge and evaluate results**

I'll inspect `maya_transcribe/rover.py` to leverage its functions for merging two results.jsonl files and computing rover evaluation, since the previous ad-hoc script isn't available.


sed -n '1,240p' maya_transcribe/rover.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

from dataclasses import dataclass
from typing import Any

from maya_transcribe.align import align_tokens, edit_distance
from maya_transcribe.textnorm import tokenize_words

@dataclass(frozen=True)
class RoverDisagreement:
    kind: str  # "token" | "gap"
    index: int
    chosen: Any
    options: Any

@dataclass(frozen=True)
class RoverResult:
    text: str
    skeleton_key: str
    disagreement_rate: float
    disagreements: list[RoverDisagreement]

def _avg_norm_distance(tokens_a: list[str], tokens_b: list[str]) -> float:
    if not tokens_a and not tokens_b:
        return 0.0
    denom = max(1, (len(tokens_a) + len(tokens_b)) // 2)
    return float(edit_distance(tokens_a, tokens_b)) / float(denom)

def choose_centroid(hyps: dict[str, list[str]]) -> str:
    keys = list(hyps.keys())
    if not keys:
        raise ValueError("No hypotheses provided")
    if len(keys) == 1:
        return keys[0]

    avg: dict[str, float] = {}
    for k in keys:
        dsum = 0.0
        for k2 in keys:
            if k2 == k:
                continue
            dsum += _avg_norm_distance(hyps[k], hyps[k2])
        avg[k] = dsum / float(len(keys) - 1)
    return min(avg.items(), key=lambda kv: kv[1])[0]

def rover_combine(
    hypotheses: dict[str, str],
    weights: dict[str, float] | None = None,
    min_gap_support: float = 0.5,
) -> RoverResult:
    """
    Lightweight ROVER-style combination:
    - pick a centroid hypothesis as skeleton
    - align others to skeleton
    - vote per skeleton token position
    - for gap insertions, keep a centroid insertion sequence if it has >= min_gap_support weight
    """
    if not hypotheses:
        return RoverResult(text="", skeleton_key="", disagreement_rate=0.0, disagreements=[])

    weights = weights or {}
    tokenized: dict[str, list[str]] = {k: tokenize_words(v) for k, v in hypotheses.items()}
    skel_key = choose_centroid(tokenized)
    skel_tokens = tokenized[skel_key]

    keys = list(tokenized.keys())
    total_weight = sum(float(weights.get(k, 1.0)) for k in keys)
    if total_weight <= 0:
        total_weight = float(len(keys))

    aligned_tokens: dict[str, list[str | None]] = {k: [None] * len(skel_tokens) for k in keys}
    gap_inserts: dict[str, list[list[str]]] = {k: [[] for _ in range(len(skel_tokens) + 1)] for k in keys}

    # Skeleton aligns to itself.
    for j, tok in enumerate(skel_tokens):
        aligned_tokens[skel_key][j] = tok

    # Align all others to skeleton.
    for k in keys:
        if k == skel_key:
            continue
        hyp_tokens = tokenized[k]
        alignment = align_tokens(skel_tokens, hyp_tokens).aligned

        skel_pos = 0
        for ref_tok, hyp_tok in alignment:
            if ref_tok is None:
                if hyp_tok is not None:
                    gap_inserts[k][skel_pos].append(hyp_tok)
                continue
            # ref_tok is not None
            if skel_pos >= len(skel_tokens):
                break
            aligned_tokens[k][skel_pos] = hyp_tok
            skel_pos += 1

    disagreements: list[RoverDisagreement] = []

    # Combine gap insertions.
    chosen_gaps: list[list[str]] = [[] for _ in range(len(skel_tokens) + 1)]
    for gap_idx in range(len(skel_tokens) + 1):
        sequences: list[tuple[str, tuple[str, ...], float]] = []
        present_weight = 0.0
        for k in keys:
            w = float(weights.get(k, 1.0))
            seq = tuple(gap_inserts[k][gap_idx])
            if seq:
                present_weight += w
            sequences.append((k, seq, w))

        if present_weight < (min_gap_support * total_weight):
            chosen_gaps[gap_idx] = []
            continue

        # Choose centroid sequence among non-empty candidates (weighted by average distance).
        candidates = [s for s in sequences if s[1]]
        if not candidates:
            chosen_gaps[gap_idx] = []
            continue

        best_seq: tuple[str, ...] | None = None
        best_score: float | None = None
        for _, seq, _w in candidates:
            score = 0.0
            for _, seq2, w2 in candidates:
                score += _avg_norm_distance(list(seq), list(seq2)) * w2
            score /= max(1e-9, present_weight)
            if best_score is None or score < best_score:
                best_score = score
                best_seq = seq

        chosen = list(best_seq or ())
        chosen_gaps[gap_idx] = chosen

        unique = {s[1] for s in candidates}
        if len(unique) > 1:
            disagreements.append(
                RoverDisagreement(
                    kind="gap",
                    index=gap_idx,
                    chosen=chosen,
                    options={k: list(seq) for k, seq, _w in candidates},
                )
            )

    # Combine skeleton positions.
    chosen_tokens: list[str] = []
    for j in range(len(skel_tokens)):
        votes: dict[str, float] = {}
        for k in keys:
            tok = aligned_tokens[k][j]
            if tok is None:
                continue
            tok = tok.strip()
            if not tok:
                continue
            votes[tok] = votes.get(tok, 0.0) + float(weights.get(k, 1.0))

        if not votes:
            continue

        max_vote = max(votes.values())
        top = [t for t, v in votes.items() if v == max_vote]
        if len(top) == 1:
            chosen_tok = top[0]
        else:
            chosen_tok = skel_tokens[j] if skel_tokens[j] in top else sorted(top)[0]

        if len(votes) > 1:
            disagreements.append(
                RoverDisagreement(kind="token", index=j, chosen=chosen_tok, options=dict(sorted(votes.items())))
            )

        chosen_tokens.append(chosen_tok)

    # Assemble final transcript with gap insertions.
    final_tokens: list[str] = []
    for j in range(len(skel_tokens)):
        final_tokens.extend(chosen_gaps[j])
        if j < len(chosen_tokens):
            final_tokens.append(chosen_tokens[j])
    final_tokens.extend(chosen_gaps[len(skel_tokens)])

    text = " ".join(final_tokens).strip()
    total_positions = len(skel_tokens) + (len(skel_tokens) + 1)
    disagreement_rate = float(len(disagreements)) / float(max(1, total_positions))

    return RoverResult(
        text=text,
        skeleton_key=skel_key,
        disagreement_rate=disagreement_rate,
        disagreements=disagreements,
    )


**Planning offline ensemble script**

I'm outlining a Python script to merge two results.jsonl files by segment ID, combining their final texts via ROVER and producing a new results.jsonl with minimal fields compatible with the evaluation script. Next, I'll verify the evaluation script's required fields to ensure output format correctness.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

from jiwer import cer, wer

from maya_transcribe.io import read_jsonl
from maya_transcribe.textnorm import normalize_text

@dataclass(frozen=True)
class EvalRow:
    segment_id: str
    language: str | None
    decision: str | None
    duration_sec: float | None
    duration_bucket: str | None
    ref: str
    hyp: str
    wer: float
    cer: float
    oracle_best_wer: float | None = None
    oracle_best_cer: float | None = None
    oracle_best_key: str | None = None
    selection_gap_wer: float | None = None

def load_refs(refs_path: Path) -> dict[str, dict[str, Any]]:
    """
    Read a JSONL file of references.
    Expected keys (any one): `text`, `transcript`, `reference`.
    Required key: `segment_id` (or `id`).
    """
    refs: dict[str, dict[str, Any]] = {}
    for row in read_jsonl(refs_path):
        segment_id = str(row.get("segment_id") or row.get("id") or "").strip()
        if not segment_id:
            continue
        refs[segment_id] = row
    return refs

def _get_text(row: dict[str, Any], keys: tuple[str, ...]) -> str | None:
    for k in keys:
        v = row.get(k)
        if isinstance(v, str) and v.strip():
            return v
    return None

def evaluate(
    results_path: Path,
    refs_path: Path,
    out_path: Path | None = None,
    hypothesis: str = "final",
    oracle: bool = True,
    language_source: str = "route",
    decision_filter: list[str] | None = None,
) -> dict[str, Any]:
    refs = load_refs(refs_path)
    results = read_jsonl(results_path)

    decision_filter_set = None
    if decision_filter:
        decision_filter_set = {str(d).strip().lower() for d in decision_filter if str(d).strip()}

    rows: list[EvalRow] = []
    for r in results:
        segment_id = str(r.get("segment_id") or "").strip()
        if not segment_id or segment_id not in refs:
            continue

        ref_row = refs[segment_id]
        ref_text = _get_text(ref_row, ("text", "transcript", "reference"))
        if ref_text is None:
            continue

        hyp_text = extract_hypothesis_text(r, hypothesis=hypothesis)
        if hyp_text is None:
            continue

        route_lang = None
        route = r.get("route") or {}
        if isinstance(route, dict):
            route_lang = route.get("language_hint")

        ref_lang = None
        if language_source == "refs":
            v = ref_row.get("lang") or ref_row.get("language")
            if isinstance(v, str) and v.strip():
                ref_lang = v.strip()
        elif language_source != "route":
            raise ValueError("language_source must be 'route' or 'refs'")

        lang = ref_lang or route_lang

        decision = None
        ver = r.get("verification") or {}
        if isinstance(ver, dict):
            decision = ver.get("decision")
        if decision_filter_set is not None:
            dec_norm = str(decision or "").strip().lower()
            if dec_norm not in decision_filter_set:
                continue

        duration_sec = None
        qc = r.get("qc") or {}
        if isinstance(qc, dict):
            try:
                duration_sec = float(qc.get("duration_sec")) if qc.get("duration_sec") is not None else None
            except (TypeError, ValueError):
                duration_sec = None
        bucket = duration_bucket(duration_sec)

        ref_norm = normalize_text(ref_text, lang=lang)
        hyp_norm = normalize_text(hyp_text, lang=lang)
        w = float(wer(ref_norm, hyp_norm))
        c = float(cer(ref_norm.replace(" ", ""), hyp_norm.replace(" ", "")))

        oracle_best_wer = None
        oracle_best_cer = None
        oracle_best_key = None
        selection_gap_wer = None

        if oracle:
            cand_wer: dict[str, float] = {}
            cand_cer: dict[str, float] = {}
            for key, text in extract_all_candidates(result_row=r).items():
                ref_n = ref_norm
                hyp_n = normalize_text(text, lang=lang)
                cand_wer[key] = float(wer(ref_n, hyp_n))
                cand_cer[key] = float(cer(ref_n.replace(" ", ""), hyp_n.replace(" ", "")))

            if cand_wer:
                oracle_best_key, oracle_best_wer = min(cand_wer.items(), key=lambda kv: kv[1])
                oracle_best_cer = cand_cer.get(oracle_best_key)
                selection_gap_wer = float(w) - float(oracle_best_wer)

        rows.append(
            EvalRow(
                segment_id=segment_id,
                language=lang,
                decision=str(decision) if decision is not None else None,
                duration_sec=duration_sec,
                duration_bucket=bucket,
                ref=ref_text,
                hyp=hyp_text,
                wer=w,
                cer=c,
                oracle_best_wer=oracle_best_wer,
                oracle_best_cer=oracle_best_cer,
                oracle_best_key=oracle_best_key,
                selection_gap_wer=selection_gap_wer,
            )
        )

    by_lang: dict[str, list[float]] = {}
    by_dec: dict[str, list[float]] = {}
    by_bucket: dict[str, list[float]] = {}
    by_lang_cer: dict[str, list[float]] = {}
    by_dec_cer: dict[str, list[float]] = {}
    by_bucket_cer: dict[str, list[float]] = {}
    selection_gaps: list[float] = []
    oracle_best_wers: list[float] = []
    for row in rows:
        if row.language:
            by_lang.setdefault(row.language, []).append(row.wer)
            by_lang_cer.setdefault(row.language, []).append(row.cer)
        if row.decision:
            by_dec.setdefault(row.decision, []).append(row.wer)
            by_dec_cer.setdefault(row.decision, []).append(row.cer)
        if row.duration_bucket:
            by_bucket.setdefault(row.duration_bucket, []).append(row.wer)
            by_bucket_cer.setdefault(row.duration_bucket, []).append(row.cer)
        if row.selection_gap_wer is not None:
            selection_gaps.append(float(row.selection_gap_wer))
        if row.oracle_best_wer is not None:
            oracle_best_wers.append(float(row.oracle_best_wer))

    summary = {
        "hypothesis": hypothesis,
        "pairs_evaluated": len(rows),
        "wer_mean": (sum(r.wer for r in rows) / len(rows)) if rows else None,
        "cer_mean": (sum(r.cer for r in rows) / len(rows)) if rows else None,
        "wer_p50": _percentile([r.wer for r in rows], 0.50),
        "wer_p90": _percentile([r.wer for r in rows], 0.90),
        "wer_p95": _percentile([r.wer for r in rows], 0.95),
        "cer_p50": _percentile([r.cer for r in rows], 0.50),
        "cer_p90": _percentile([r.cer for r in rows], 0.90),
        "cer_p95": _percentile([r.cer for r in rows], 0.95),
        "segment_error_rate": (sum(1 for r in rows if r.wer > 0.0) / len(rows)) if rows else None,
        "pct_wer_gt_5pct": (sum(1 for r in rows if r.wer > 0.05) / len(rows)) if rows else None,
        "pct_wer_gt_10pct": (sum(1 for r in rows if r.wer > 0.10) / len(rows)) if rows else None,
        "wer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang.items())},
        "wer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec.items())},
        "wer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket.items())},
        "cer_by_language_mean": {k: sum(v) / len(v) for k, v in sorted(by_lang_cer.items())},
        "cer_by_decision_mean": {k: sum(v) / len(v) for k, v in sorted(by_dec_cer.items())},
        "cer_by_duration_bucket_mean": {k: sum(v) / len(v) for k, v in sorted(by_bucket_cer.items())},
        "oracle": {
            "enabled": bool(oracle),
            "oracle_best_wer_mean": (sum(oracle_best_wers) / len(oracle_best_wers)) if oracle_best_wers else None,
            "selection_gap_wer_mean": (sum(selection_gaps) / len(selection_gaps)) if selection_gaps else None,
            "selection_gap_wer_p90": _percentile(selection_gaps, 0.90),
        },
    }

    if out_path is not None:
        out_path.parent.mkdir(parents=True, exist_ok=True)
        out_path.write_text(json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")

    return summary

def extract_hypothesis_text(result_row: dict[str, Any], hypothesis: str) -> str | None:
    """
    hypothesis:
      - "final": uses result_row["final"]["text"] (falls back to rover if missing)
      - "rover" (default): uses result_row["rover"]["text"]
      - "asr:<key>": uses result_row["asr"][<key>]["text"]
    """
    hypothesis = (hypothesis or "rover").strip()
    if hypothesis == "final":
        final = result_row.get("final") or {}
        if isinstance(final, dict) and isinstance(final.get("text"), str):
            return final["text"]
        hypothesis = "rover"
    if hypothesis == "rover":
        rover = result_row.get("rover") or {}
        if isinstance(rover, dict) and isinstance(rover.get("text"), str):
            return rover["text"]
        return None

    if hypothesis.startswith("asr:"):
        key = hypothesis[len("asr:") :].strip()
        asr = result_row.get("asr") or {}
        if isinstance(asr, dict):


Summary: auto


**Planning combined results script**

I'm outlining a script to merge Voxtral and Gemma results keyed by segment_id, using Voxtral as the base and adding Gemma data under asr, computing combined rover scores, and including route and QC fields. Next, I'll verify write_jsonl support and continue building the script accordingly.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
from __future__ import annotations

import json
from pathlib import Path
from typing import Any, Iterable

def write_jsonl(path: Path, rows: Iterable[dict[str, Any]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8") as f:
        for row in rows:
            f.write(json.dumps(row, ensure_ascii=False) + "\n")

def read_jsonl(path: Path) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    with path.open("r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            rows.append(json.loads(line))
    return rows


**Planning ensemble script approach**

I'm preparing to load both JSONL outputs keyed by segment_id to merge aligned results, starting by inspecting the first entries to identify ASR key names and confirm data structure for combining.


python3 - <<'PY'
import json
from pathlib import Path
for name in ['voxtral_small_vllm_prompt', 'gemma3n_e2b_vllm_prompt']:
    path = Path('outputs/bench/maya_hf_hin_v2')/name/'results.jsonl'
    with path.open() as f:
        row = json.loads(next(f))
    print(name)
    print('segment_id', row.get('segment_id'))
    print('asr keys', list((row.get('asr') or {}).keys()))
    print('final keys', row.get('final', {}).keys())
    print('final text', row.get('final', {}).get('text')[:60])
    print()
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
voxtral_small_vllm_prompt
segment_id Kathbath__hindi__valid__100__844424930753064-252-f.m4a-50ae5279bfb29b25
asr keys ['vllm:mistralai/Voxtral-Small-24B-2507']
final keys dict_keys(['text', 'source'])
final text इस बकाया रकम में ब्याज की राशि भी शामिल है

gemma3n_e2b_vllm_prompt
segment_id Kathbath__hindi__valid__100__844424930753064-252-f.m4a-50ae5279bfb29b25
asr keys ['vllm:google/gemma-3n-E2B-it']
final keys dict_keys(['text', 'source'])
final text इस बकाया रकम में ब्याज की राशि भी चार चार चार चार चार चार चा


**Planning ensemble output and evaluation**

I'm setting up a new output directory with JSONL and eval files, reusing ROVER for combining two final texts, and planning to include per-model ASR info keyed by model names. I'll check how `extract_all_candidates` handles candidate extraction to ensure consistent oracle evaluation.


rg -n "extract_all_candidates" -n maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
131:            for key, text in extract_all_candidates(result_row=r).items():
249:def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:


sed -n '240,320p' maya_transcribe/eval.py


Output:
Exit code: 0
Wall time: 0 seconds
Output:
        if isinstance(asr, dict):
            entry = asr.get(key) or {}
            if isinstance(entry, dict) and isinstance(entry.get("text"), str):
                return entry["text"]
        return None

    return None

def extract_all_candidates(result_row: dict[str, Any]) -> dict[str, str]:
    """
    Collect all text candidates available in a pipeline result row.
    Keys are stable and namespaced (final/rover/asr:<key>).
    """
    candidates: dict[str, str] = {}

    final = extract_hypothesis_text(result_row, "final")
    if final is not None:
        candidates["final"] = final

    rover = extract_hypothesis_text(result_row, "rover")
    if rover is not None:
        candidates["rover"] = rover

    asr = result_row.get("asr") or {}
    if isinstance(asr, dict):
        for key, entry in asr.items():
            if not isinstance(key, str):
                continue
            if not isinstance(entry, dict):
                continue
            text = entry.get("text")
            if isinstance(text, str) and text.strip():
                candidates[f"asr:{key}"] = text

    # Also include normalized hypotheses if present (useful for debugging).
    normalized = result_row.get("normalized") or {}
    if isinstance(normalized, dict):
        for key, text in normalized.items():
            if not isinstance(key, str) or not isinstance(text, str):
                continue
            if text.strip():
                candidates[f"norm:{key}"] = text

    return candidates

def duration_bucket(duration_sec: float | None) -> str | None:
    if duration_sec is None:
        return None
    if duration_sec < 3.0:
        return "0-3s"
    if duration_sec < 10.0:
        return "3-10s"
    if duration_sec < 30.0:
        return "10-30s"
    if duration_sec < 60.0:
        return "30-60s"
    return "60s+"

def _percentile(xs: list[float], q: float) -> float | None:
    if not xs:
        return None
    xs_sorted = sorted(xs)
    idx = int(round((len(xs_sorted) - 1) * q))
    idx = max(0, min(len(xs_sorted) - 1, idx))
    return float(xs_sorted[idx])


**Planning integrated results script**

I'm outlining a script to merge ASR data from both sources, using voxtral's row for QC and route, injecting gemma ASR, and computing rover-based combined fields; it'll produce a new results file with a consolidated hypotheses dictionary and disagreement flags.


**Drafting integrated results script**

I'm creating a script that reads both results, merges ASR entries, combines hypotheses with rover_combine, serializes disagreement data as dicts, and writes the merged output; after that, I'll run evaluation on the combined results with default final text.


python3 - <<'PY'
from pathlib import Path
from maya_transcribe.io import read_jsonl, write_jsonl
from maya_transcribe.rover import rover_combine

vox_path = Path('outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt/results.jsonl')
gemma_path = Path('outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt/results.jsonl')
out_dir = Path('outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt')

vox_rows = read_jsonl(vox_path)
gem_rows = read_jsonl(gemma_path)
gem_by_id = {r.get('segment_id'): r for r in gem_rows}

out_rows = []
missing = 0
for v in vox_rows:
    seg_id = v.get('segment_id')
    if not seg_id:
        continue
    g = gem_by_id.get(seg_id)
    if g is None:
        missing += 1
        continue
    v_final = (v.get('final') or {}).get('text') or ''
    g_final = (g.get('final') or {}).get('text') or ''
    hyps = {
        'voxtral': v_final,
        'gemma': g_final,
    }
    rover = rover_combine(hyps)
    out = {
        'schema_version': v.get('schema_version'),
        'segment_id': seg_id,
        'base_id': v.get('base_id'),
        'relpath': v.get('relpath'),
        'uid': v.get('uid'),
        'src_path': v.get('src_path'),
        'wav_path': v.get('wav_path'),
        'src_ffprobe': v.get('src_ffprobe'),
        'ffprobe': v.get('ffprobe'),
        'qc': v.get('qc'),
        'route': v.get('route'),
        'verification': v.get('verification'),
        'asr': {},
        'rover': {
            'text': rover.text,
            'skeleton_key': rover.skeleton_key,
            'disagreement_rate': rover.disagreement_rate,
            'disagreements': [
                {
                    'kind': d.kind,
                    'index': d.index,
                    'chosen': d.chosen,
                    'options': d.options,
                }
                for d in rover.disagreements
            ],
        },
        'final': {
            'text': rover.text,
            'source': 'rover',
        },
        'ensemble': {
            'unique_models': sorted({
                *((v.get('ensemble') or {}).get('unique_models') or []),
                *((g.get('ensemble') or {}).get('unique_models') or []),
                *((v.get('asr') or {}).keys()),
                *((g.get('asr') or {}).keys()),
            }),
        },
    }
    if isinstance(v.get('asr'), dict):
        out['asr'].update(v['asr'])
    if isinstance(g.get('asr'), dict):
        out['asr'].update(g['asr'])
    out_rows.append(out)

out_dir.mkdir(parents=True, exist_ok=True)
write_jsonl(out_dir / 'results.jsonl', out_rows)
print(f"Wrote {len(out_rows)} rows, missing {missing}")
PY


Output:
Exit code: 0
Wall time: 0.1 seconds
Output:
Wrote 200 rows, missing 0


python3 -m maya_transcribe.cli eval --results outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/results.jsonl --refs data/bench/maya_hf_hin_v2/refs.jsonl --out outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json


Output:
Exit code: 0
Wall time: 0.3 seconds
Output:
{
  "hypothesis": "final",
  "pairs_evaluated": 200,
  "wer_mean": 0.2681708291708293,
  "cer_mean": 0.18124495078831873,
  "wer_p50": 0.15384615384615385,
  "wer_p90": 0.5,
  "wer_p95": 1.0,
  "cer_p50": 0.06557377049180328,
  "cer_p90": 0.27586206896551724,
  "cer_p95": 0.6486486486486487,
  "segment_error_rate": 0.805,
  "pct_wer_gt_5pct": 0.805,
  "pct_wer_gt_10pct": 0.625,
  "wer_by_language_mean": {
    "hin": 0.2681708291708293
  },
  "wer_by_decision_mean": {
    "accept": 0.26752516489358585,
    "human": 0.26945252094505817
  },
  "wer_by_duration_bucket_mean": {
    "0-3s": 0.707260101010101,
    "10-30s": 0.30989010989010984,
    "3-10s": 0.25705321569683276
  },
  "cer_by_language_mean": {
    "hin": 0.18124495078831873
  },
  "cer_by_decision_mean": {
    "accept": 0.17987051360156397,
    "human": 0.18397331117396631
  },
  "cer_by_duration_bucket_mean": {
    "0-3s": 0.5476006191950464,
    "10-30s": 0.10925865542581849,
    "3-10s": 0.17651339594402668
  },
  "oracle": {
    "enabled": true,
    "oracle_best_wer_mean": 0.1360475912975912,
    "selection_gap_wer_mean": 0.13212323787323788,
    "selection_gap_wer_p90": 0.2222222222222222
  }
}


**Updating plan after ensemble WER results**

The ensemble model shows worse overall WER at 26.8%, with very poor performance on the shortest transcript bucket (0-3s) at 0.707, likely due to short transcripts. Step 1 is done, so I'll update the plan and tools accordingly.


Run remaining Prompt.md experiments (Voxtral proofread and Voxtral+Gemma ensemble) and capture eval outputs

Update results.md with a new Prompt.md experiments section and summary of outcomes

Summarize findings and next-step suggestions


Output: Plan updated


**Updating results.md with new prompt experiments**

I'm preparing to add a new "Prompt.md Experiments (custom prompt override)" section after "Prompts/Decoding," summarizing new prompt runs including language replacements with "Hindi," and adding corresponding details in "Voxtral" and "Gemma-3n" sections while preserving ASCII formatting.


**Detailing WER updates for prompt override runs**

I'm specifying exact WER values for Voxtral-Small, Voxtral-Mini, Gemma-3n prompt runs, and the Voxtral+Gemma ensemble with prompt overrides, noting degradation versus baselines and planning to update related sections with prompt override usage and output stripping descriptions.


python3 - <<'PY'
import json
from pathlib import Path
paths = {
    'voxtral_small_prompt': 'outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt/eval.json',
    'voxtral_mini_prompt': 'outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt/eval.json',
    'gemma_prompt': 'outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt/eval.json',
    'voxtral_small_proofread_prompt_final': 'outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json',
    'voxtral_small_proofread_prompt_raw': 'outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_raw.json',
    'voxtral_small_proofread_prompt_proofread': 'outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_proofread.json',
    'voxtral_gemma_ensemble_prompt': 'outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json',
}
for name, p in paths.items():
    data = json.loads(Path(p).read_text())
    print(name, data['wer_mean'], data['cer_mean'])
PY


Output:
Exit code: 0
Wall time: 0 seconds
Output:
voxtral_small_prompt 0.24517020479520482 0.17030685871627074
voxtral_mini_prompt 0.9848948273948275 0.9987700421526944
gemma_prompt 0.2425516844266846 0.13898414203048856
voxtral_small_proofread_prompt_final 0.2849680458430461 0.2195450530720729
voxtral_small_proofread_prompt_raw 0.2565013736263737 0.181590870576232
voxtral_small_proofread_prompt_proofread 0.3184132534132537 0.27358345387323096
voxtral_gemma_ensemble_prompt 0.2681708291708293 0.18124495078831873


**Planning updates to results.md**

I'm preparing to add a new top-level section for prompt experiments with custom overrides for Voxtral, Gemma, and Ensemble, plus updates to the "Prompts / Decoding" section about using supplied prompts and output stripping; I'll also update results with rounded two-decimal percentage numbers.


**Implementing patch for prompt experiments section**

I'm adding a bullet to the "Prompts / Decoding" list explaining custom prompt overrides and output stripping, then creating a detailed new "Prompt.md Experiments" section with rounded WER/CER results, eval paths, commands, and notes on prompt override causing degraded performance across models.


Input:
*** Begin Patch
*** Update File: results.md
@@
 ### Prompts / Decoding
 - Voxtral transcription mode: vLLM’s Voxtral `TranscriptionRequest` path (audio-only, no extra text prompt). Language is set to `hi` from path routing.
+- Prompt.md override: when supplied, Voxtral uses a chat prompt with text + audio; Gemma uses a custom `<start_of_turn>` prompt with `<audio_soft_token>`. `<language>` is replaced with the model-supported language name (Hindi). Output is post-processed to strip the `Output 1: Transcript` header line.
 - Gemma-3n transcription mode: vLLM’s built-in Gemma3n prompt template:
   - `"Transcribe this audio into Hindi: <audio_soft_token><end_of_turn>\n<start_of_turn>model\n"`
 - Voxtral proofread prompt (audio + pass1 transcript, chat mode):
   - `"Correct transcription errors using the audio. Keep the same language/script. Do not add new content. Output only the corrected transcript.\n\nTranscript:\n{transcript}"`
@@
 ## Ensembles
 - Voxtral-Small + Gemma-3n (ROVER on both finals): WER 20.81%, CER 11.04% (worse than Voxtral alone).
   - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble/eval.json`

+## Prompt.md Experiments (custom instruction prompt)
+Prompt source: `Prompt.md` (strict verbatim transcription, `Output 1: Transcript` format). All runs below used the same Hindi bench (`data/bench/maya_hf_hin_v2`).
+
+### Voxtral (Prompt.md)
+- Voxtral-Small-24B-2507: WER 24.52%, CER 17.03% (worse than baseline).
+  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt/eval.json`
+  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`
+- Voxtral-Small-24B-2507 (2-pass proofread + Prompt.md on pass1): final WER 28.50%, CER 21.95% (worse than baseline).
+  - Final eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json`
+  - Raw eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_raw.json` (WER 25.65%, CER 18.16%)
+  - Proofread eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_proofread.json` (WER 31.84%, CER 27.36%)
+  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-proofread --vllm-transcribe-prompt-file Prompt.md`
+- Voxtral-Mini-3B-2507: WER 98.49%, CER 99.88% (prompt caused severe degradation).
+  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt/eval.json`
+  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Mini-3B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`
+
+### Gemma-3n (Prompt.md)
+- Gemma-3n-E2B-it: WER 24.26%, CER 13.90% (worse than tuned Gemma with the built-in prompt).
+  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt/eval.json`
+  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`
+
+### Ensemble (Prompt.md)
+- Voxtral-Small + Gemma-3n (ROVER on both finals): WER 26.82%, CER 18.12% (worse than Voxtral prompt alone).
+  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json`
+  - Results: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/results.jsonl`
+
 ## Notes
 - Gemma-3n now uses a per-audio max_tokens cap (12 tokens/sec, max 1024) and a Gemma stop token (`<end_of_turn>`) to prevent runaway decoding.
 - Best WER on this Hindi bench is currently Voxtral-Small (14.87%).
*** End Patch


Output: {"output":"Success. Updated the following files:\nM results.md\n","metadata":{"exit_code":0,"duration_seconds":0.0}}


Summary: auto


**Completing prompt experiment metrics update**

I've ensured that new prompt experiments report WER/CER with correct decimal precision and updated the plan accordingly.


Output: Plan updated


**Summarizing ASR with LLMs findings**

I'm preparing a summary covering experiments, ensemble selection, and updated results in results.md, which now includes methodology and new prompt outcomes. I'll explain why prompt-based experiments yielded worse evaluations, noting that strict prompt rules (like <unk> tokens and no punctuation) likely increased WER due to evaluation normalization and model mode differences.


**Confirming ensemble and prompt results**

I'll note that prompting Voxtral forces chat mode, which worsens performance, explaining prompt experiment degradation. Ensemble results combining base and prompt outputs don’t improve performance, and results.md reflects these final updated evaluations after error-free runs.


Output:
Exit code: 0
Wall time: 0 seconds
Output:
# Benchmark Results

Dataset: `data/bench/maya_hf_hin_v2` (Hindi, 200 segments)
Metric: mean WER (lower is better); CER included for context.
Inference: vLLM audio transcription, Whisper skipped.

## How We Ran These Experiments
- Pipeline: preprocess -> MMS-LID -> ASR -> ROVER -> verify. Language routing is forced from the path (`/hin/`), so Voxtral receives `language=hi`.
- Final selection in the pipeline: `final` defaults to `rover.text` unless CTC scoring or explicit router preferences are enabled. We did not enable CTC or router overrides in these runs, so `final` == ROVER output.
- Verification (accept/human) only labels segments; it does not change the transcript.

### ROVER / Ensemble Logic (inside pipeline)
- Implemented in `maya_transcribe/rover.py`:
  - Tokenize each hypothesis.
  - Choose a “centroid” hypothesis (lowest average normalized edit distance to others).
  - Align all hypotheses to the centroid.
  - Vote per token position; ties fall back to the centroid token.
  - Gap insertions are kept only if >=50% support.
- With two hypotheses, ROVER tends to pick the centroid transcript and keeps only gaps supported by both.

### Voxtral + Gemma Offline Ensemble
- We merged `results.jsonl` from Voxtral and Gemma, then ran ROVER on their `final` texts only.
- No verification gating or CTC reranking in that offline ensemble; the output is purely ROVER.

### Prompts / Decoding
- Voxtral transcription mode: vLLM’s Voxtral `TranscriptionRequest` path (audio-only, no extra text prompt). Language is set to `hi` from path routing.
- Prompt.md override: when supplied, Voxtral uses a chat prompt with text + audio; Gemma uses a custom `<start_of_turn>` prompt with `<audio_soft_token>`. `<language>` is replaced with the model-supported language name (Hindi). Output is post-processed to strip the `Output 1: Transcript` header line.
- Gemma-3n transcription mode: vLLM’s built-in Gemma3n prompt template:
  - `"Transcribe this audio into Hindi: <audio_soft_token><end_of_turn>\n<start_of_turn>model\n"`
- Voxtral proofread prompt (audio + pass1 transcript, chat mode):
  - `"Correct transcription errors using the audio. Keep the same language/script. Do not add new content. Output only the corrected transcript.\n\nTranscript:\n{transcript}"`
- Decoding: `temperature=0.0`, `top_p=1.0`. For Gemma-3n we also add stop token `<end_of_turn>` and cap output length to 12 tokens/sec (min 32, max 1024).

## Voxtral
- Voxtral-Small-24B-2507: WER 14.87%, CER 5.49% (best so far).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096`
- Voxtral-Small-24B-2507 (2-pass proofread, audio + pass1 transcript via chat): final WER 16.62%, CER 7.32%. Raw pass stayed competitive, proofread pass degraded.
  - Final eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval.json`
  - Raw eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval_raw.json` (WER 15.03%, CER 5.56%)
  - Proofread eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread/eval_proofread.json` (WER 24.58%, CER 17.59%)
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-proofread`
- Voxtral-Mini-3B-2507: WER 19.32%, CER 7.85%.
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm --skip-whisper --vllm-asr-model mistralai/Voxtral-Mini-3B-2507 --vllm-gpu-memory-utilization 0.15 --vllm-max-model-len 4096`

## Gemma-3n
- Gemma-3n-E2B-it (initial run): WER 137.6%, CER 105.2%. This was dominated by runaway repetitions (hundreds/thousands of tokens) when the model did not emit a stop token.
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm/eval.json`
- Gemma-3n-E2B-it (tuned stop token + dynamic max_tokens): WER 20.63%, CER 10.05%. The runaway repetition is capped, but accuracy still lags Voxtral.
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_tuned/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_tuned --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096`

## Ensembles
- Voxtral-Small + Gemma-3n (ROVER on both finals): WER 20.81%, CER 11.04% (worse than Voxtral alone).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble/eval.json`

## Prompt.md Experiments (custom instruction prompt)
Prompt source: `Prompt.md` (strict verbatim transcription, `Output 1: Transcript` format). All runs below used the same Hindi bench (`data/bench/maya_hf_hin_v2`).

### Voxtral (Prompt.md)
- Voxtral-Small-24B-2507: WER 24.52%, CER 17.03% (worse than baseline).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`
- Voxtral-Small-24B-2507 (2-pass proofread + Prompt.md on pass1): final WER 28.50%, CER 21.95% (worse than baseline).
  - Final eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json`
  - Raw eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_raw.json` (WER 25.65%, CER 18.16%)
  - Proofread eval: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval_proofread.json` (WER 31.84%, CER 27.36%)
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Small-24B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-proofread --vllm-transcribe-prompt-file Prompt.md`
- Voxtral-Mini-3B-2507: WER 98.49%, CER 99.88% (prompt caused severe degradation).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/voxtral_mini_vllm_prompt --skip-whisper --vllm-asr-model mistralai/Voxtral-Mini-3B-2507 --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`

### Gemma-3n (Prompt.md)
- Gemma-3n-E2B-it: WER 24.26%, CER 13.90% (worse than tuned Gemma with the built-in prompt).
  - Eval: `outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt/eval.json`
  - Command: `python3 -m maya_transcribe.cli bench run --bench-dir data/bench/maya_hf_hin_v2 --out outputs/bench/maya_hf_hin_v2/gemma3n_e2b_vllm_prompt --skip-whisper --vllm-asr-model google/gemma-3n-E2B-it --vllm-gpu-memory-utilization 0.9 --vllm-max-model-len 4096 --vllm-transcribe-prompt-file Prompt.md`

### Ensemble (Prompt.md)
- Voxtral-Small + Gemma-3n (ROVER on both finals): WER 26.82%, CER 18.12% (worse than Voxtral prompt alone).
  - Eval: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json`
  - Results: `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/results.jsonl`

## Notes
- Gemma-3n now uses a per-audio max_tokens cap (12 tokens/sec, max 1024) and a Gemma stop token (`<end_of_turn>`) to prevent runaway decoding.
- Best WER on this Hindi bench is currently Voxtral-Small (14.87%).


**Summarizing experiment results and prompt impact**

I'm preparing the summary to highlight updated results in results.md, noting prompt override caused Voxtral to degrade by pushing it into chat mode instead of transcription. I'll explain possible causes for Gemma's degrade including strict output format and `<unk>` tokens affecting WER despite normalization. I'll emphasize Voxtral-Small transcription mode remains best so far (14.87% WER) and note specialized ASR or segmentation are needed for <5% WER, respecting user constraints.


**Detailing findings and next steps**

I'm summarizing key WER/CER results confirming prompt overrides degrade performance, with Voxtral-Small transcription mode best at 14.87% WER. I'll explain experiment setup, ensemble method (ROVER), prompt approaches, and error sources like `<unk>`. I'll suggest avoiding prompt overrides for Voxtral to keep transcription mode and note Gemma’s persistently high WER despite fixes. I'll update the plan to completed after delivering this summary.


Output: Plan updated


**Summarizing prompt experiments and results**

Ran remaining Prompt.md experiments (Voxtral proofread and Voxtral+Gemma ensemble), evaluated raw and proofread outputs, and documented findings in `results.md`. Prompt overrides consistently performed worse, with the best WER of 14.87% using Voxtral-Small transcription mode. New output directories with experiment outputs were added for reference. Evaluation results and details can be found in `results.md`.

Next steps could include:
1. Exploring the default transcription mode without prompt overrides for potential improvements.
2. Experimenting with alternative prompt designs to better guide the ASR model.
3. Applying normalization techniques during evaluation to refine WER comparisons.


**Reporting final Prompt.md experiments and results**

Ran the remaining Prompt.md benchmarks (Voxtral proofread and Voxtral+Gemma ensemble), evaluated raw and proofread outputs, and updated `results.md` with full details. New evaluation outputs were saved under respective `outputs/bench/maya_hf_hin_v2` subdirectories.

Key results:
- Voxtral-Small transcription mode (no prompt override) achieved best WER: 14.87%.
- Prompt overrides degraded performance: Voxtral-Small with prompt 24.52%, proofread variant 28.50%, Gemma-3n prompt 24.26%, Voxtral-Mini prompt collapsed (98.49%), ensemble prompt 26.82%.
- Gemma’s initially high WER was caused by runaway repetition; after adding stop tokens and limiting length, WER improved but remained 20–24%, indicating likely model capability limits rather than prompt design.
- Ensemble uses ROVER for centroid voting; final outputs found in `rover.text`.
- Prompt.md runs applied chat-style prompts to Voxtral and a specialized <audio_soft_token> prompt to Gemma, which overall harmed WER compared to vanilla transcription mode.

Next steps to consider:
1) Retain Voxtral in transcription mode without prompt overrides to preserve low WER.
2) If prompt influence is desired, test minimal or less intrusive prompts (e.g., avoid `<unk>` tokens and output headers) to reduce degradation.
3) Optionally perform manual inspection of transcripts from Gemma and Voxtral to identify error patterns driving WER.

Evaluation outputs and detailed metrics are in `results.md` and in subfolders like `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt` and `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt`.


Ran the remaining Prompt.md experiments (Voxtral proofread + Voxtral/Gemma ensemble), evaluated raw/proofread variants, and documented everything in `results.md`.

Key outcomes from the Prompt.md runs
- Voxtral-Small + Prompt.md: WER 24.52%, CER 17.03% (worse than baseline 14.87%).
- Voxtral-Small proofread + Prompt.md: final WER 28.50% (raw 25.65%, proofread 31.84%).
- Gemma-3n + Prompt.md: WER 24.26%, CER 13.90% (worse than tuned Gemma at 20.63%).
- Voxtral-Mini + Prompt.md: WER 98.49% (collapsed).
- Voxtral+Gemma Prompt.md ensemble: WER 26.82% (worse than either prompt run alone).

What this says about Gemma-3n
- The original 137% WER was a decoding bug (runaway repetition); with stop token + max_tokens, it drops to ~20.6%.
- Prompt.md didn’t help; it pushed Gemma to ~24.3% WER, so the remaining gap now looks like model capability/fit rather than setup.

Where I recorded everything
- `results.md` (new “Prompt.md Experiments” section with commands, criteria, and eval paths)
- New evals: `outputs/bench/maya_hf_hin_v2/voxtral_small_vllm_proofread_prompt/eval.json`, `outputs/bench/maya_hf_hin_v2/voxtral_gemma_ensemble_prompt/eval.json`

If you want, I can try one of these next:
1) Keep Voxtral in transcription mode and try a minimal prompt for Gemma only (no `<unk>`, no header) to see if strict formatting was the regression.
2) Sample the worst 10 Gemma segments and categorize error types (repetition vs omissions vs boundaries) to see if a fixable pattern exists.
3) Tweak normalization (digits/diacritics/word-joins) to see if we can reclaim a few WER points without model changes.